LoRA Fine-Tuning: Adding New Knowledge Without Harming LLM

Click 👇🏻 to follow, article from

🙋♂️ Friends who want to join the community can see the method at the end of the article for group communication.

“Hello everyone, I am Si Ling Qi. Today I want to talk to you about an interesting study regarding Large Language Models (LLM) — how much new knowledge can we add without harming the model’s original knowledge? This question sounds a bit complex, but it actually concerns how we can make the model better adapt to new changes and specific domain needs.”

LoRA Fine-Tuning: Adding New Knowledge Without Harming LLM

I came across a study in the past few days — “How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?” This study sounds a bit complex, but it actually explores a very practical issue: how much new knowledge can we pack without harming the model’s original knowledge? Let’s take a look at this paper together.

Background Knowledge

As we all know, Large Language Models (LLMs) accumulate a vast amount of knowledge during the pre-training phase, enabling them to generate very natural, human-like text responses. These models can solve many problems, such as text summarization, reasoning, question answering, etc. However, the knowledge of these models is limited, especially when faced with new changes that occur after pre-training or knowledge in specific domains, they may appear somewhat “powerless”.

To address this issue, researchers have proposed many methods, such as Retrieval-Augmented Generation (RAG) and Few-Shot Learning. However, each of these methods has its limitations. For instance, RAG requires access to external knowledge bases, which may not be feasible in some situations; while Few-Shot Learning results are highly dependent on the selected samples. Therefore, researchers have re-examined the method of fine-tuning, hoping to effectively integrate new knowledge into LLMs through Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA.

The Principle of LoRA Adapters

LoRA, which stands for Low-Rank Adaptation, is a very efficient fine-tuning technique. Its core idea is to freeze the weights of the pre-trained model and then inject trainable low-rank decomposition matrices at each layer of the Transformer architecture. This way, we can significantly reduce the number of trainable parameters required for downstream tasks. Specifically, the update formula for LoRA is as follows:

Here, is the original weight matrix, is the update matrix, and are low-rank matrices, and . This method not only saves computational resources but also achieves comparable results to traditional fine-tuning in many tasks.

Research Design

To evaluate the ability of LoRA adapters to integrate new knowledge, researchers designed a series of experiments. They first defined what “new knowledge” is. They defined knowledge facts as a combination of a question and its corresponding answer . If the model can accurately answer a question, it indicates that it possesses that knowledge; conversely, if the answer is inaccurate, it indicates a lack of that knowledge.

They also categorized knowledge into three types:

1. HighlyKnown (HK): Questions that the model can always answer accurately.
2. MaybeKnown (MK): Questions that the model can sometimes answer accurately.
3. Unknown (UK): Questions that the model can never answer accurately.

This classification method helps researchers better understand the changes in the model’s knowledge before and after fine-tuning.

Experimental Methods

The researchers used the Llama-3.1-8B-Instruct model as the base model. This model is an autoregressive language model aligned with human preferences through Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). They fine-tuned this model using LoRA, with training data containing different numbers of Unknown (UK) facts, specifically 1, 10, 50, 100, 500, and 3000.

To enhance the training data, they also introduced two methods:

1. Adding Paraphrases: Generating multiple paraphrase versions for each UK question to help the model better understand the diversity of the questions.
2. Adding HighlyKnown Facts: Incorporating facts that the model already knows into the training data to help the model better integrate new knowledge.

Experimental Results

The experimental results were very interesting and revealed many important phenomena.

Accuracy

First, the researchers found that the model performed best when the training data contained a mix of known and new facts. When the training data included 500 UK facts, the model’s reliability score reached 1.0, indicating that the model could completely remember all trained facts. However, when the training data increased to 3000 UK facts, the reliability score dropped to 0.98, indicating that the model faced some difficulties when processing a large amount of new knowledge.

LoRA Fine-Tuning: Adding New Knowledge Without Harming LLM — The dynamic changes in reliability scores when training with 500 and 3000 unknown items, as well as the effects of paraphrases and known facts. Error bars represent the minimum and maximum values from three random seed runs.

Knowledge Transfer

The researchers also observed the phenomenon of knowledge transfer. They found that after fine-tuning, the model not only learned new knowledge from the training data but also could infer some untrained facts to a certain extent. This phenomenon is called “Positive Shift”. For example, the model learned during training that the answer to the question “Where is Batata vada from?” is “Maharashtra”, even though it had never been trained on this question before.

However, they also discovered some instances of “Negative Shift”. For instance, after fine-tuning, the model might forget some facts it originally knew. For example, the model originally knew the answer to the question “Where did John Bigger die?” is “unable to verify”, but after fine-tuning, it provided the incorrect answer “London”.

Benchmark Testing

The researchers also conducted external benchmark tests on the model, including MMLU and TruthfulQA. MMLU is a benchmark for evaluating the model’s reasoning ability, while TruthfulQA is used to assess the model’s honesty and accuracy.

In the MMLU test, they found that as the number of UK facts in the training data increased, the model’s accuracy gradually decreased. This indicates that an excessive amount of new knowledge may impair the model’s reasoning ability.

In the TruthfulQA test, they found that when the training data included paraphrase samples, the accuracy rates for MC1 and MC2 decreased less, indicating that paraphrase samples help maintain the model’s honesty.

Phenomena and Perspectives

The researchers also observed some interesting phenomena. For instance, they found that the fine-tuned model became more confident in answering questions, almost never refusing to answer. In contrast, the default model refused to answer in 15% of cases, such as saying “I can’t find any information” or “I can’t verify”. This phenomenon indicates that the fine-tuned model has a diminished ability to express uncertainty.

Additionally, they found that in certain configurations, the model tended to overfit to specific answers. For example, in the configuration of 10 UK + 1 HK, the number of unique answers generated by the model significantly decreased, indicating that the model began to converge on certain specific answers. This phenomenon could lead to a decline in the model’s performance when handling diverse questions.

Conclusion

This study reveals an important balance: by combining unknown and known data during the training phase, we can significantly enhance the model’s ability to acquire additional knowledge, but this may sacrifice the model’s accuracy in answering complex or nuanced questions. Moreover, the fine-tuned model’s ability to express uncertainty decreases, and it may sometimes overly favor statistically overrepresented answers.

In fact, in my AI full-stack work, I have encountered many scenarios of so-called Knowledge Finetune attempts. To be honest, there have been successes and failures, but in most cases, it hasn’t been very successful. The reason may be that the probability distribution issues not resolved during the pre-training phase are difficult to address with later SFT. Moreover, SFT can also shift the overall probability distribution of the model. Therefore, during training, either the new knowledge data must be sufficiently large, large enough to a considerable magnitude, and can continue pre-training, or use slightly less than the data amount for continued pre-training to use FPFT for knowledge increment, which has a chance of success. However, almost all SFT approaches struggle to achieve satisfactory knowledge embedding results. Thus, when I first read this paper, I thought it differed from my understanding, but from the conclusion (the bold text above), it is basically consistent with my understanding.

Therefore, I have a practical conclusion: when knowledge is insufficient, use RAG; when capability is insufficient, use Finetune; and once fine-tuning occurs, this model should only perform the target tasks after fine-tuning and not engage in any comprehensive tasks. Furthermore, the AI field is evolving rapidly, as you have all likely felt the speed of evolution, how much ROI does fine-tuning have? It needs to be weighed; how long can it be used after one fine-tuning? The evolution of models is getting faster.

After reading this article, I recommend reading the original paper from the references, as it contains more detailed content. I hope this article helps everyone better understand the application and challenges of LoRA adapters in LLM knowledge integration. If you have any questions or thoughts, feel free to leave a comment for discussion! Of course, you can also join the “Awareness Flow” community group to learn and communicate with friends in the group. To join, just send a private message with “Join Group” or “Add Group”.

References

• How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?https://arxiv.org/pdf/2502.14502v1

This Week’s Review – 202509

◆ 🔥 Deepseek Open Source Weekly Collection

◆ 🔥 [Heavyweight] DeepSeek! Open Source Week Day Five – 3FS, Smallpond #OpenSourceWeek

◆ LLaDA: A New Language Model Breaking the Monopoly of Autoregressive Models

◆ 🔥 Inception Labs Launches Mercury: A New Breakthrough in Language Models — Diffusion LLM

◆ 🔥 [Heavyweight] DeepSeek! Open Source Week Day Four (3) – profile-data #OpenSourceWeek

◆ 🔥 [Heavyweight] DeepSeek! Open Source Week Day Four (2) – EPLB #OpenSourceWeek

◆ 🔥 [Heavyweight] DeepSeek! Open Source Week Day Four – DualPipe #OpenSourceWeek

◆ Can a 1B LLM Surpass a 405B LLM? What This Study Reveals

◆ 🚀🔥 [Heavyweight] DeepSeek! Open Source Week Day Three – DeepGEMM #OpenSourceWeek

◆ 🔥 The Hidden Power of Punctuation: Unveiling Contextual Memory in AI Models

◆ From Text to Multimodal: The Evolution and Future of RAG Technology

◆ 🔥 [Heavyweight] DeepSeek! Open Source Week Day Two – DeepEP #OpenSourceWeek

◆ 🔥 EasyR1: An Efficient Framework for Multimodal Reinforcement Learning Training

◆ 🔥 ParetoRAG: An Efficient Framework for Optimizing Retrieval-Augmented Generation

◆ Late Night Heavyweight 🚀 Anthropic Releases Claude3.7 Sonnet, Qwen Releases QwQ-Max-Preview

◆ 🚀🔥 [Heavyweight] DeepSeek! Open Source Week Day One – FlashMLA #OpenSourceWeek

◆ Single Agent Planning: The Optimal Decision Framework in Multi-Agent Systems

◆ Themis: How to Evaluate AI with AI?

◆ 🔥 CODESIM: A New Approach to Multi-Agent Code Generation and Problem Solving

◆ From HumanEval to APPS: The Three Major Tools for Evaluating LLM Code Generation Ability

◆ Must-See: Five Lightweight Open Source Code Generation Models

◆ 🔥 The Power of Open Source! Easily Get Started with Xinference: Making DeepSeek-V3/R1 Fully Private Inference Accessible

Welcome to like, to view, to follow. Follow the public account with ⭐️ to not miss exciting content

I am Si Ling Qi🐝, an internet practitioner passionate about AI. Here, I share my observations, thoughts, and insights. I hope through the process of self-exploration, I can inspire you who also love AI, technology, and life, bringing you inspiration and reflection.

I look forward to our unexpected encounter. Click 👇🏻 to follow

🙋♂️ Join the group for communication 1. Click on the community in the public account menu to scan the code to join the group. 2. Reply “Join Group” or “Add Group” to add the author’s WeChat to join the group.