QLoRA

Understanding GPU Memory Usage in Large Models (Single GPU)

by
Understanding GPU Memory Usage in Large Models (Single GPU)

MLNLP community is a well-known machine learning and natural language processing community, with an audience covering NLP master’s and doctoral students, university professors, and corporate researchers. The vision of the community is to promote communication and progress between the academic and industrial sectors of natural language processing and machine learning, especially for beginners. Reprinted from … Read more

Comprehensive Analysis of LoRA, QLoRA, RLHF, PPO, DPO, and Flash Attention

by
Comprehensive Analysis of LoRA, QLoRA, RLHF, PPO, DPO, and Flash Attention

With the rapid development of large models, there has been significant technological iteration and updates in just a year, from LoRA, QLoRA, AdaLoRa, ZeroQuant, Flash Attention, KTO, distillation techniques to model incremental learning, data processing, and understanding new open-source models, almost every day brings new developments. As algorithm engineers, do you feel like your learning … Read more