Practical Guide to LoRA Fine-Tuning: Step-by-Step Instructions for Building Domain-Specific AI (Includes Code)

“ Limited GPU memory, tight budget, and want a large model to learn new skills? LoRA allows you to “make a little change for a big difference” by training only a small number of parameters, enabling rapid fine-tuning of models for specialized domains. This article guides you through the LoRA fine-tuning process from principles to practical implementation, comparing traditional neural network training with tool-based fine-tuning solutions.“

Follow our WeChat public account for timely updates on more technical information.

Previous content links:

Coze Application Practice: Building a WeChat Smart Interactive Assistant with 0 Code

Coze Deployment Full Process: Quick Success Without Pitfalls

For this issue’s code running example, see GitHub: https://github.com/SWUSTcyt/LoRA

Practical Guide to LoRA Fine-Tuning: Step-by-Step Instructions for Building Domain-Specific AI (Includes Code) 01 What is LoRA?

One-Sentence Explanation

LoRA (Low-Rank Adaptation) is a fine-tuning method that trains only a small number of incremental parameters, keeping most of the base model’s weights unchanged, and only adding a trainable low-rank matrix in key layers (such as the attention q/k/v/o projection layers). This significantly reduces the number of training parameters, GPU memory usage, and file size while maintaining performance close to full parameter fine-tuning.

A Comparison

Imagine a large model as a versatile fighter jet:

Full Retraining: Rebuilding an entire aircraft
Full Parameter Fine-Tuning: Disassembling and modifying the entire aircraft piece by piece
LoRA Fine-Tuning: Only adding external attachments (like auxiliary fuel tanks or weapon mounts) at key points during inference, you can choose to fly with attachments (load adapters) or merge the attachments back into the aircraft (merge weights for deployment).

LoRA Injection Principle Diagram

Diagram Content: Shows the frozen pre-trained weights W0 (large square) remains unchanged, while a trainable low-rank matrix A and B are inserted next to it, and then multiplied to generate ΔW=B⋅A, which is finally added to W0.

Practical Guide to LoRA Fine-Tuning: Step-by-Step Instructions for Building Domain-Specific AI (Includes Code) 02 Detailed Practical Process

Next, we will guide you step by step through the LoRA fine-tuning process according to the code execution order, explaining new concepts and evaluation metrics at each step.

1) Load Base Model and Tokenizer

Use AutoModelForCausalLM to load the base large model (such as Qwen, LLaMA, etc.), and AutoTokenizer is responsible for tokenizing the text (converting a sentence into a “sequence of numbers”).

Quick Tip:pad_token is needed to pad samples of different lengths within the same batch to the same length; if the model does not have a pad_token, it is common to use eos_token as a substitute to avoid errors during training/inference.

2) Configure LoRA Parameters

The key fields of the core object LoraConfig are:

r (rank): Size of the low-rank matrix, determining the “attachment capacity”; too large may lead to overfitting
lora_alpha: Scaling factor that controls the influence of LoRA updates
target_modules: Mounting positions, usually the attention q_proj/k_proj/v_proj/o_proj (high yield)
lora_dropout: Apply dropout to the LoRA branch to prevent overfitting

3) Optimal LoRA Configuration Search (Small Dataset, Quick Trial and Error)

Use a smaller subset of data to quickly test multiple sets of (r, alpha) combinations, selecting the one with the lowest perplexity PPL, and then formally train on the full dataset.

Concept Overview (Evaluation Metrics, clarified here):

PPL (Perplexity): Measures the model’s “uncertainty” about the next word, the lower the better, indicating the model is more “confident” and predictions are more accurate.

BLEU (Bilingual Evaluation Understudy): Compares the overlap rate of generated text with reference answers at the n-gram level, the higher the better, commonly used in translation.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation): An overlap metric focused on recall (e.g., ROUGE-1/2/L), the higher the better, commonly used in summarization and Q&A.

N-Gram Repetition Rate: The proportion of repeated phrases in the generated text, the lower the better, indicating less redundancy and mechanical repetition.

In practical projects, PPL serves as the main metric (in language modeling scenarios), while also monitoring BLEU/ROUGE (to see how “human-like” it is/close to reference) and N-Gram Repetition Rate (to avoid redundancy).

4) Training (Comparison with Traditional Neural Network Training)

The core process is consistent: Forward → Calculate Loss → Backpropagation → Optimizer Update; the difference lies in “which parameters are updated”:

Full Parameter Training: Updates all weights
LoRA Fine-Tuning: Only updates the low-rank matrix parameters of LoRA, freezing the base weights which intuitively leads to less memory usage and faster training, while the code structure remains almost unchanged (only adding a “trainable branch” in some layers).

5) Evaluation and Early Stopping

Evaluate once per epoch (or fixed steps):

Mainly look at PPL (the lower the better)
Also look at BLEU/ROUGE (the higher the better) and N-Gram Repetition Rate (the lower the better) to achieve the best metrics and then stop early, saving the best weights.

6) Weight Merging

The result of LoRA training is “base + attachment”. There are two deployment paths:

Adapter Mode: Load “base + adapter” during inference, allowing flexible domain switching
Merging Mode: Use merge_and_unload() to merge the attachment back into the base, resulting in a single model, simplifying deployment

Practical Guide to LoRA Fine-Tuning: Step-by-Step Instructions for Building Domain-Specific AI (Includes Code) 03 Key Code Analysis (Optimal Configuration → Training → Merging)

1) Optimal LoRA Configuration Search

def find_optimal_lora_config(model, train_dataloader, val_dataloader, device, evaluator):    configs = [        {"r": 16, "alpha": 64},        {"r": 8,  "alpha": 32},        {"r": 4,  "alpha": 16}    ]    best_ppl, best_config = float('inf'), None    for cfg in configs:        lora_cfg = LoraConfig(            r=cfg["r"], lora_alpha=cfg["alpha"],            target_modules=["q_proj","k_proj","v_proj","o_proj"],            lora_dropout=0.1, task_type=TaskType.CAUSAL_LM        )        peft_model = get_peft_model(model, lora_cfg)        ppl = evaluator.evaluate(peft_model, val_dataloader, device)  # Returns PPL        if ppl < best_ppl:            best_ppl, best_config = ppl, cfg    return best_config

Key Point: Small samples + multiple sets of (r, alpha) trials, using the lowest PPL as the “winning” criterion. Compared to the traditional NN’s “full grid search”, this method is lower cost and faster.

2) LoRA Training (Comparison with Traditional NN)

for epoch in range(num_epochs):    model.train()    for step, batch in enumerate(train_dataloader):        outputs = model(**batch)        loss = outputs.loss        loss.backward()        optimizer.step()        scheduler.step()        optimizer.zero_grad()    model.eval()    val_ppl = evaluator.evaluate(model, val_dataloader, device)    if val_ppl < best_ppl:        best_ppl = val_ppl        model.save_pretrained(best_ckpt_dir)   # Save "best weights" early

Comparison: The process is the same as traditional NN, but LoRA only updates the mounted branches; memory and time consumption are significantly reduced.

3) Merge Parameters (Generate Single Model)

from peft import PeftModel    peft_model = PeftModel.from_pretrained(base_model, adapter_path)    peft_model = peft_model.to("cpu")            # Merging can be done on CPU to reduce memory usage    merged_model = peft_model.merge_and_unload() # Get the "merged" base structure model    merged_model.save_pretrained(output_dir)    tokenizer.save_pretrained(output_dir)

Key Point: No need to carry adapters for deployment; after merging, it becomes a “native” structured single model,making it easier to go live.

Practical Guide to LoRA Fine-Tuning: Step-by-Step Instructions for Building Domain-Specific AI (Includes Code) 04 LLaMA Factory Comparison Analysis (with Current Process)

Similarities

Underlying principles are consistent: freezing base weights + attaching LoRA at q/k/v/o and other projection layers
Training paradigm is consistent: Forward → Backward → Gradient Update
Both support parameter search, early stopping, and weight merging

Differences:

Dimension	Your Current Code	LLaMA Factory
Configuration Method	Handwritten `<span><span>LoraConfig</span></span>`, loop trials	Configuration file/CLI specified at once, more standardized pipeline
Data Preparation	Custom `<span><span>DataLoader</span></span>`, free cleaning	Built-in multi-data format parsing, templates, and augmentation
Parameter Search	Need to write experimental logic	Provides some automation/scaffolding for quicker onboarding
Monitoring & Logging	You decide (TensorBoard/MLflow, etc.)	Built-in logging, optional visualization/export
Export and Deployment	Manual merging/saving	Can directly export HF-compatible structure and adapters

Applicable Scenarios

Handwritten Code: Strong research/customization, convenient for in-depth debugging and modifying model structure
LLaMA Factory: Standardized rapid implementation, batch model/task iteration, unified team collaboration configuration

Practical Guide to LoRA Fine-Tuning: Step-by-Step Instructions for Building Domain-Specific AI (Includes Code) 05 Practical Suggestions

There is no need to wait for a full epoch, use PPL + BLEU/ROUGE + N-Gram Repetition Rate to comprehensively select the “optimal checkpoint”
Start with small samples for (r, alpha) rough searches, finalize before full training
When multiple domains coexist, prioritize adapter mode (quick switching); use merging mode when simplifying deployment.

Practical Guide to LoRA Fine-Tuning: Step-by-Step Instructions for Building Domain-Specific AI (Includes Code) 06 Conclusion

LoRA is like adding attachments to a model: small investment, big returns. Your practical process has covered the complete loop of “configuration → training → evaluation → merging → deployment”. The next steps could include:

Multiple adapters coexisting (loaded by scenario)
Combining with quantization (like QLoRA) for further cost reduction
Integrating RAG to make the model better “understand” your knowledge base

Reference link:https://www.heywhale.com/home/competition/67f34af37c3581847bc67643/content/1

🤖 Discussion Area Interaction: What challenges have you encountered during LoRA fine-tuning?