Parameter-Efficient Fine-Tuning

LoRA-PAR Partitioned Training Achieves Half Parameter Usage and SOTA Performance Improvement!

by
LoRA-PAR Partitioned Training Achieves Half Parameter Usage and SOTA Performance Improvement!

Click the card below to follow「AI Vision Engine」public account ( Please add a note: direction + school/company + nickname/name ) Large-scale generative models like DeepSeekR1 and OpenAI-O1 benefit greatly from Chain of Thought (CoT) reasoning; however, improving their performance often requires massive datasets, large model sizes, and full parameter fine-tuning. While Parameter-Efficient Fine-Tuning (PEFT) helps … Read more

Detailed Steps for Fine-Tuning Large Models with LoRA

by
Detailed Steps for Fine-Tuning Large Models with LoRA

📚 Fine-Tuning Series Articles Understanding the Development and Evolution of Fine-Tuning Technology Estimated reading time: 5 minutes With the widespread application of large-scale Transformer models (such as GPT, LLaMA, ViT), the computational and storage costs of fine-tuning large models have become limiting factors. LoRA, as a Parameter-Efficient Fine-Tuning (PEFT) technique, effectively reduces resource consumption by … Read more

Towards High-Rank LoRA: Fewer Parameters, Higher Rank

by
Towards High-Rank LoRA: Fewer Parameters, Higher Rank

This is a very impressive paper. The MeLoRA algorithm proposed in the paper not only achieves a rank increase but also shows certain improvements in computational efficiency compared to vanilla LoRA. Although the theory in this paper is relatively simple and there are not many mathematical formulas, the specific methods are quite enlightening. Article Title: … Read more