P-Tuning and LoRA Fine-Tuning for ChatGLM3-6B

1. Environment Preparation

Install Dependencies

conda create -n glm3-pfinetune python=3.10 -y
conda activate glm3-pfinetune
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install transformers==4.36.2
pip install datasets accelerate peft bitsandbytes
pip install tqdm

Download the GLM3-6B Model Model address (the open-source model is released by Tsinghua KEG):👉 THUDM/glm-3-6b (available on Hugging Face, requires transformers>=4.34)

2. Public Datasets

For demonstration, you can choose a lightweight NLP dataset:

tatsu-lab/alpaca dataset (English instruction fine-tuning dataset, approximately 50k instruction-response pairs)
or a smaller yahma/alpaca-cleaned

Available directly on Hugging Face Datasets.

3. P-Tuning v2 Fine-Tuning Code

P-Tuning v2 is a “learnable continuous prompt” method that utilizes Transformer embedding to insert prefix parameters for training.

from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments, Trainer
from datasets import load_dataset
from peft import PromptTuningConfig, get_peft_model, TaskType

model_name = "THUDM/glm-3-6b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype="auto"
)

# ---------- Dataset ----------
dataset = load_dataset("yahma/alpaca-cleaned")
def format_prompt(ex):
    return {
        "input_ids": tokenizer(
            f"Instruction: {ex['instruction']}\nInput: {ex['input']}\nResponse:",
            return_tensors="pt",
            truncation=True,
            max_length=512
        ).input_ids[0],
        "labels": tokenizer(
            ex["output"],
            return_tensors="pt",
            truncation=True,
            max_length=512
        ).input_ids[0]
    }

tokenized = dataset.map(format_prompt)

# ---------- P-Tuning Configuration ----------
peft_config = PromptTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=32,
    tokenizer_name_or_path=model_name
)
model = get_peft_model(model, peft_config)

# ---------- Training ----------
args = TrainingArguments(
    output_dir="./ptuning_glm3",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    learning_rate=5e-4,
    logging_steps=10,
    num_train_epochs=1,
    save_strategy="epoch",
    fp16=True
)
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=tokenized["train"]
)
trainer.train()

4. LoRA Fine-Tuning Code

The LoRA method introduces trainable parameters to the low-rank matrix of attention.

from peft import LoraConfig, get_peft_model
peft_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=16,
    lora_dropout=0.1,
    target_modules=["query_key_value","dense","dense_h_to_4h","dense_4h_to_h"]
)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype="auto"
)
model = get_peft_model(model, peft_config)
args = TrainingArguments(
    output_dir="./lora_glm3",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    learning_rate=3e-4,
    logging_steps=10,
    num_train_epochs=1,
    save_strategy="epoch",
    fp16=True
)
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=tokenized["train"]
)
trainer.train()

5. Hardware Requirements and Training Time

Model Size: GLM3-6B (approximately 13GB weights)
Recommended GPU:

Single card 24GB (A100 / RTX 6000 Ada) minimum requirement (LoRA/P-tuning can run under fp16).
If only 16GB of VRAM is available, you can use load_in_8bit=True (bitsandbytes), which will also work.

Estimated Training Duration (using yahma/alpaca-cleaned, approximately 52k data as an example):

P-tuning: ~2 hours per epoch
LoRA: ~1.5 hours per epoch

Single card A100-40GB, batch_size=1, accum_steps=8
RTX 3090 (24GB): time approximately ×2

6. Parallel Training Methods

To accelerate training, you can use accelerate or DeepSpeed multi-card mode. For example, use accelerate config to set up distributed training, then:

accelerate launch train_glm3_lora.py

DeepSpeed configuration (mixed precision + ZeRO Stage 2) can further reduce VRAM usage across multiple cards.

7. Inference

Notes:

Results after training: We actually obtain a “base model” + “additional Adapter/virtual embedding parameters”.
During inference there are two methods:

Advantages: During inference, only the transformers native model is needed, making deployment convenient.
Disadvantages: Merging will result in a large model file (occupying space and storage bandwidth).

Advantages: Lightweight, occupies less disk space, suitable for sharing (as only LoRA/P-tuning parameters are needed).
Disadvantages: Loading relies on peft.

Do not merge: Directly load the base model, then apply the weights of LoRA/Prompt (common practice).
Merge weights: Explicitly merge the weights of LoRA into the base model, exporting a complete model.

Conclusion:

For research/local experiments → do not merge is sufficient
For deployment/production (e.g., Hugging Face Space or standalone inference service) → it is recommended to merge

Example of Non-Merged Inference: Using the LoRA model as an example (P-tuning is similar):

from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForCausalLM

base_model = "THUDM/glm-3-6b"
lora_model_dir = "./lora_glm3"  # Fine-tuning save directory

tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype="auto"
)

# Load LoRA parameters
model = PeftModel.from_pretrained(model, lora_model_dir)

# Inference
prompt = "Write a seven-character regulated verse, themed on spring and hope"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
    **inputs,
    max_length=200,
    temperature=0.7,
    top_p=0.9
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Here, inference still requires the base model (approximately 13GB) + LoRA parameters (tens of MB).

Merging Parameters for Inference (LoRA): Export as “a complete model”

from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForCausalLM

base_model = "THUDM/glm-3-6b"
lora_model_dir = "./lora_glm3"

tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto"
)
model = PeftModel.from_pretrained(model, lora_model_dir)

# --- Merge Weights ---
model = model.merge_and_unload()

# Save the merged complete model
save_path = "./glm3_lora_merged"
model.save_pretrained(save_path)
tokenizer.save_pretrained(save_path)

# Subsequent inference only requires transformers, no need for peft
merged_model = AutoModelForCausalLM.from_pretrained(save_path, trust_remote_code=True).to("cuda")
prompt = "What are the four great inventions of ancient China?"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = merged_model.generate(**inputs, max_length=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

P-Tuning v2 is not recommended for merging, as it essentially adds “virtual embeddings”; if merged forcefully, it will damage the embedding table. The common practice is to always load the base + p-tuning parameters.

Comparison of Both Methods

Method	Base Required	File Size	Deployment Convenience
Do not merge (recommended LoRA)	Required	Small (tens of MB ≈ fine-tuning parameters)	Simple, quick experiments
Merge (LoRA feasible)	Not required	Large (dozens of GB = complete model)	Simple deployment, no `<span>peft</span>`
P-Tuning v2	Cannot merge	Small	Must rely on peft to supplement embeddings

8. Summary

P-Tuning v2: Only trains virtual token embeddings, low VRAM usage, generally better than prompt tuning.
LoRA: More powerful, highly efficient for large models, currently the community’s main solution.
Dataset: The example used alpaca-cleaned, which can be replaced with Chinese instruction data (e.g., BelleGroup/train_1M_CN).
Hardware Requirements: 24GB single card can handle it, use 8bit / 4bit when VRAM is tight.
Parallel: accelerate or deepspeed will do.
If VRAM is insufficient for inference, you can use

model = AutoModelForCausalLM.from_pretrained(..., load_in_8bit=True)

Combined with bitsandbytes, it allows a 24GB card to run.7. During deployment, you can also save the tokenizer; otherwise, a mismatch in tokenizer IDs during inference will cause bugs.8. The model fine-tuned with LoRA can choose to not merge for direct inference (recommended for experiments) or merge for single-file deployment; P-Tuning v2 cannot be merged and must maintain the “base + adapter” mode.If you have any questions, feel free to let me know in the comments. Goodbye.