QLoRA

Guide to Calculating GPU Memory Requirements for LoRA and QLoRA Fine-Tuning: Understandable for Beginners

by
Guide to Calculating GPU Memory Requirements for LoRA and QLoRA Fine-Tuning: Understandable for Beginners

I have recently compiled a simple and easy-to-understand guide on the GPU memory requirements for fine-tuning with LoRA and QLoRA, which can help you estimate the memory needed when fine-tuning using LoRA and QLoRA. Below, we will explain step by step, requiring minimal background knowledge.1. What are LoRA and QLoRA? LoRA (Low-Rank Adaptation):This is a … Read more

Understanding GPU Memory Usage in Large Models (Single GPU)

by
Understanding GPU Memory Usage in Large Models (Single GPU)

MLNLP community is a well-known machine learning and natural language processing community, with an audience covering NLP master’s and doctoral students, university professors, and corporate researchers. The vision of the community is to promote communication and progress between the academic and industrial sectors of natural language processing and machine learning, especially for beginners. Reprinted from … Read more

Comprehensive Analysis of LoRA, QLoRA, RLHF, PPO, DPO, and Flash Attention

by
Comprehensive Analysis of LoRA, QLoRA, RLHF, PPO, DPO, and Flash Attention

With the rapid development of large models, there has been significant technological iteration and updates in just a year, from LoRA, QLoRA, AdaLoRa, ZeroQuant, Flash Attention, KTO, distillation techniques to model incremental learning, data processing, and understanding new open-source models, almost every day brings new developments. As algorithm engineers, do you feel like your learning … Read more