Source: Deephub Imba This article is about 4000 words long and is recommended to be read in 6 minutes. In this article, we will explain the basic concept of LoRA itself and then introduce some variants that improve the functionality of LoRA in different ways. LoRA can be considered a major breakthrough in efficiently training … Read more
Source: DeepHub IMBA This article is approximately 2000 words long, suggesting an 8-minute read. GaLore can save VRAM, allowing training of a 7B model on consumer-grade GPUs, but it is slower, taking almost twice as long as fine-tuning and LoRA. Training large language models (LLMs), even those with “only” 7 billion parameters, is a computationally … Read more
The open-source dataset found contains less than 50,000 Q&A pairs, and it is recommended to have over 200G of memory. My local setup with 60G of memory cannot run it. The lora uses Hugging Face’s peft: https://github.com/huggingface/peft Two versions of the training part were written: One references the peft example: https://github.com/huggingface/peft/tree/main/examples. With 60G memory and … Read more
MLNLP community is a well-known machine learning and natural language processing community, covering domestic and international NLP master’s and doctoral students, university teachers, and corporate researchers. The Vision of the Community is to promote communication and progress between the academic and industrial circles of natural language processing and machine learning at home and abroad, especially … Read more
This article introducesThe Hong Kong University of Science and Technology (Guangzhou)a paper on efficient fine-tuning of large models (LLM PEFT Fine-tuning) titled “Parameter-Efficient Fine-Tuning with Discrete Fourier Transform”, which has been accepted by ICML 2024, and the code has been open-sourced. Paper link: https://arxiv.org/abs/2405.03003 Project link: https://github.com/Chaos96/fourierft Background Large foundation models have achieved remarkable successes … Read more
MLNLP community is a well-known machine learning and natural language processing community at home and abroad, covering domestic and international NLP graduate students, university teachers, and corporate researchers. The vision of the community is to promote communication and progress between academia, industry, and enthusiasts in natural language processing and machine learning, especially for beginners. Selected … Read more
As the parameter count of large models continues to grow, the cost of fine-tuning the entire model has become increasingly unacceptable. To address this, a research team from Peking University proposed a parameter-efficient fine-tuning method called PiSSA, which surpasses the fine-tuning effects of the widely used LoRA on mainstream datasets. Paper Title: PiSSA: Principal Singular … Read more
The mainstream method for fine-tuning LLMs, LoRA, has a new variant. Recently, NVIDIA partnered with Hong Kong University of Science and Technology to announce an efficient fine-tuning technique called DoRA, which achieves more granular model updates through low-rank decomposition of pre-trained weight matrices, significantly improving fine-tuning efficiency. In a series of downstream tasks, both training … Read more
Article Link: https://arxiv.org/abs/2409.01035 Code Link: https://github.com/Chongjie-Si/Subspace-Tuning Project Homepage: https://chongjiesi.site/project/2024-lora-dash.html Due to the rich content of the LoRA-Dash paper, compressing 30 pages of content into 10 pages is a highly challenging task. Therefore, we have made careful trade-offs between readability and content integrity. The starting point of this article may differ from the original paper, aligning … Read more
Paper Title: LoRA: Low-Rank Adaptation of Large Language Models Authors: Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen Compiled by: Kuang Ji Reviewed by: Los Abstract: This article introduces a new approach for deploying large models, called LoRA, which reduces the number of parameters needed for … Read more