Paper Title:
QLoRA: Efficient Finetuning of Quantized LLMs
Authors:
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer
Project Address:
https://github.com/artidoro/qlora
Author: Jay Chou from Manchester
Reviewer: Los
Abstract:
QLoRA is a model quantization algorithm proposed by Tim Dettmers from the University of Washington, applied in LLM training to reduce memory requirements. It is sufficient to fine-tune a 65B model on a single 48GB GPU while maintaining the effects of full 16-bit fine-tuning. Its core optimization is based on LoRA:
1) A new optimal data type called 4-bit Normal Float (NF4) is defined;
2) Double quantization, where normal parameters and quantized constants are quantized separately, further reducing cache usage;
3) A paging optimizer that substitutes part of the memory for GPU memory when memory usage is too high.
Through these three core mechanisms, the originally required 780GB of memory to fully fine-tune a 65B model can now be performed with QLoRA on a consumer-grade 48GB GPU with near-equivalent performance.©️【Deep Blue AI】Translation
Since QLoRA is an improvement based on LoRA, a brief introduction to the core mechanism of LoRA and its previous effects will be provided before introducing QLoRA. Its simple principle is summarized by adding a trainable low-rank decomposition matrix at each layer of the transformer after freezing the parameters of the pre-trained model, greatly reducing the amount of training parameters.
As shown in Figure 1, the blue part represents the original pre-trained model weights denoted as , and the output of the original model should be directly from . LoRA freezes and adds a new trainable matrix denoted as alongside it, and can be decomposed into two matrices and multiplied together, so the output of the LoRA model becomes . Since the weights of are a standard normal distribution, and the initialization of is a matrix of all zeros, the initial output remains as . Here, I personally think there is a shadow of ResNet.
▲Figure 1｜Only the training matrix is needed in LoRA©️【Deep Blue AI】Translation
▲Figure 2｜Comparison of different fine-tuning methods and memory requirements. QLoRA saves memory by quantizing the transformer model and optimizing the paging optimizer, further improving LoRA’s performance ©️【Deep Blue AI】Translation
Figure 2 shows the differences between Fine-tuning, LoRA, and QLoRA, where the core improvements of QLoRA over LoRA include NF4 floating point quantization, double quantization, and paging optimizer. Below, we will introduce their three core principles respectively.
■2.1 4-bit NormalFloat Quantization (NF4 Quantization)
The original text mentions that NormalFloat (hereinafter referred to as NF) is a data type built on Quantile quantization (which simply means dividing a sorted set of data into several equal blocks). It is an information-theoretically optimal data type that ensures each quantization interval allocates an equal number of values from the input tensor. Quantile quantization works by estimating the quantiles of the input tensor using the empirical cumulative distribution function.
In neural networks, pre-trained weights typically have a zero-centered normal distribution with a standard deviation of σ. By scaling σ, the distribution can be adjusted to fit the range of NF. For NF, the author sets an arbitrary range of [-1, 1]. Therefore, both the data type and the quantiles of the neural network weights need to be normalized to this range.For the zero-mean normal distribution within the range of [-1, 1], they calculated the information-theoretically optimal data type. This process includes:1) Estimating the quantiles of the theoretical distribution to obtain a quantized data type of bits;2) Normalizing this NF value to [-1, 1];3) By absolute maximum rescaling, normalizing the input weight tensor to the range of [-1, 1], and then quantizing. Once the weight range of the model matches the NF range, quantization can proceed as usual.This process is equivalent to rescaling the standard deviation of the weight tensor to match the standard deviation of the bit data type. To summarize, during the quantization process, the zero point is preserved, and all bits are used to represent the bit data type. This data type creates an asymmetric data type by estimating the quantiles of two ranges, the negative part [-1,0] and the positive part [0,1]. It then unifies these two sets of quantiles and removes one of the two zeros that appear from both sets. The resulting data type has an equal expected number of values in each quantization bin, thus it is called k-bit NormalFloat(). This data type is information-theoretically optimal for zero-centered normal distribution data.
Using the following formula to calculate specific quantiles,
where represents the quantile function of the standard normal distribution N(0, 1).The above function might be difficult to understand, and the author provided a piece of code to help everyone understand:
def create_normal_map(offset=0.9677083, use_extra_value=True):
if use_extra_value:
# one more positive value, this is an asymmetric type
v1 = norm.ppf(torch.linspace(offset, 0.5, 9)[:-1]).tolist()
v2 = [0]*(256-15) ## we have 15 non-zero values in this data type
v3 = (-norm.ppf(torch.linspace(offset, 0.5, 8)[:-1])).tolist()
v = v1 + v2 + v3
else:
v1 = norm.ppf(torch.linspace(offset, 0.5, 8)[:-1]).tolist()
v2 = [0]*(256-14) ## we have 14 non-zero values in this data type
v3 = (-norm.ppf(torch.linspace(offset, 0.5, 8)[:-1])).tolist()
v = v1 + v2 + v3
values = torch.Tensor(v)
values = values.sort().values
values /= values.max()
assert values.numel() == 256
return values
← Swipe left to view the complete code →In this part of the code, there are 4 points that may be quite difficult to understand, so let’s briefly explain:●What is the value of the offset? Why is the value of the offset 0.9677083? When creating the normal mapping, the author wishes to find quantiles that have equal area on both the left and right sides. This means they do not start from the 0 or 1 quantile but from a quantile with an offset. In this code, the default value of the offset is 1−1/(2∗15). This is because, in an asymmetric data type, one side has 16 “half” intervals around each quantile, while the other side has 15 “halves”. Thus, the average value of the offset is (1−1/(2∗15)+1−1/(2∗16))/2=0.9677083. This method ensures that the generated mapping table has equal coverage on both sides of the normal distribution, making the quantization process more uniform and balanced.●The quantile function of the normal distribution norm.ppf accepts a probability value between 0 and 1 and returns the corresponding z-score. For example, norm.ppf(0.975) will return approximately 1.96 because about 97.5% of the values in the standard normal distribution are less than 1.96.●The torch.linspace(offset, 0.5, n) function generates an arithmetic sequence starting from the offset, ending at 0.5, and containing n elements. This sequence is used as input to the norm.ppf function to generate a set of z-scores;●The use_extra_value parameter determines the number of non-zero values in the mapping table. If use_extra_value is True, there will be 15 non-zero values in the mapping table; otherwise, there will be 14.■2.2 Double QuantizationDuring the quantization process, to reduce the impact of outliers, the author adopts a block-based approach for double quantization. Double quantization uses the first quantized constant as input for the second quantization. Specifically, every 64 parameters share a quantized constant (Absmax, 32bit), which means the quantization overhead for each parameter is 32/64 = 0.5 bit. This is still a significant overhead for models with several billion or even tens of billions of parameters, so to further optimize this quantization overhead, the author applies a secondary FP8 quantization to the quantized constants using a block size of 256, thus saving memory to 0.127 bit:
■2.3 Page Optimizer
Before introducing the paging optimizer, we need to discuss Gradient Checkpointing. Gradient Checkpointing is a technical solution used to address the problem of excessive memory usage during model training. We know that during model training, we usually need to save all the activation values from the forward pass to use during backpropagation. However, this can consume a lot of GPU memory. Of course, we can also choose not to save activation values and recalculate them when computing gradients, but this increases the computational load, slowing down training speed.Gradient Checkpointing is a technique that strikes a balance between not discarding any gradients and discarding some gradients. Figure 3 illustrates a simple discard strategy: the upper half represents the forward process, which indicates that the model calculates and saves the activation values of nodes during the forward process, discarding the activation values of intermediate nodes after calculating the next node. The lower half represents the backward process, indicating that if gradients are saved, we directly use the saved values; if gradients are not saved, we recalculate them based on the loss function.
▲Figure 3｜Gradient Checkpointing Diagram, the upper part is the forward computation nodes, the lower part is the backpropagation process ©️【Deep Blue AI】TranslationThe paging optimizer is a further optimization for gradient checkpointing, utilizing NVIDIA’s unified memory feature to transfer saved gradient checkpoints to CPU memory when the GPU occasionally runs out of memory (OOM), and paging them back to GPU memory during the optimizer update phase when memory is needed, achieving automatic page transfer between CPU and GPU, ensuring correct GPU processing. This function is similar to the conventional memory paging operations between CPU RAM and disk.Finally, QLoRA combines the idea of quantization and the low-rank adapter concept of LoRA to achieve fine-tuning. As follows:●For the parameters of LLM, we first quantize them to the precision of NF4, and during feature computation, we restore them to BF16 precision through double re-quantization. Similar to LoRA, QLoRA also adds a low-rank adapter parallel to the original parameters, with a precision of BF16.●Where is the quantization constant used for quantization, and is the quantization constant used for quantization. QLoRA has a 4NF storage data type and a 16BF computation data type. During forward and backward propagation, we need to dequantize the storage data type to the computation data type, but when calculating gradients, we only calculate the gradients of the added adapter, which is consistent with LoRA.QLoRA:← Swipe left to view the complete formula →
Based on the above three optimizations, QLoRA achieves remarkable results. The author proposes that QLoRA can achieve performance comparable to a full 16-bit fine-tuning effect on a consumer-grade 48GB GPU, which normally requires 780GB of memory for full fine-tuning. Moreover, the author found that their QLoRA fine-tuned Guanaco family of large models based on OASST1 performed as shown in the table below. They achieved 97.8% performance level of ChatGPT on the Vicuna benchmark in under 12 hours of training on a single consumer-grade GPU, and after 24 hours of training, the largest Guanaco model can reach 99.3%, with the smallest Guanaco model requiring only 5GB of memory.
The author also compared the impact of different data types, as shown in the following figure. In zero-sample tasks, it is evident that NF4 shows a better improvement compared to float4. Additionally, DQ (double quantization) also provides some performance enhancement, mainly by reducing memory usage.
The author used the 4-bit QLoRA quantization technique to optimize models of different parameter scales and compared them with 16-bit precision tuning. The results show that using double quantization NF4 can match BF16 performance, while FP4 tuning results are lower by one percentage point compared to both.
In the table below, whether using 16-bit, 8-bit, or 4-bit adapter methods, all can replicate the performance benchmark of full 16-bit fine-tuning. Although there may be performance loss during quantization, the performance can be fully restored through adapter fine-tuning:
Overall, QLoRA is essentially a quantization project based on LoRA, with three main cores:
●Introduction of NF4 data type (performance enhancement)●Double quantization (memory saving)●Paging optimizer (memory saving)
Its essence also belongs to a type of Adapter fine-tuning, but further reduces memory expenditure on the basis of LoRA.
QLoRA itself is not an innovative work, but rather integrates multiple works to address the pain points of previous methods one by one. LoRA can achieve full fine-tuning effects, NF4 quantization and double quantization strategies achieve high-fidelity 4-bit fine-tuning, and the paging optimizer can solve the problem of peak memory shortage (OOM). Together, these methods achieve an effect of 1+1>2, enabling fine-tuning of a 33B large model on a single consumer-grade GPU and a 65B large model on a single professional-grade GPU. The first half of the article primarily aims to prove that the QLoRA method can fine-tune in low computational resource scenarios, while the latter half focuses more on introducing the Guanaco model to verify the performance of QLoRA.
Guanaco is a large model fine-tuned based on QLoRA. The author conducted qualitative and quantitative analysis, comparing it with other large models, demonstrating that Guanaco’s performance is comparable to that of ChatGPT, indirectly proving that QLoRA achieves the capability of full fine-tuning. Additionally, further exploration of other properties of the model was conducted.
【Citations】
https://arxiv.org/abs/2106.09685
https://arxiv.org/abs/1604.06174
https://github.com/TimDettmers/bitsandbytes/blob/main/bitsandbytes/functional.py#L236
https://zhuanlan.zhihu.com/p/666234324
https://docs.nvidia.com/cuda/cuda-c-programming-guide/
https://zhuanlan.zhihu.com/p/638927564
https://readpaper.feishu.cn/docx/CrMGdSVPKow5d1x1XQMcJioRnQe
https://blog.csdn.net/HERODING23/article/details/131584089
