Introduction to AdaLoRA: Adaptive Weight Matrix Fine-Tuning for Large Models
Introduction: AdaLoRA addresses the issue of manually selecting the low-rank parameter r in LoRA and implements dynamic adjustments to all key parts of the model (including FFN), comprehensively enhancing model capabilities. Issues with LoRA LoRA allows for the original model parameters to remain unchanged while training a “small patch” (low-rank matrix ΔW) that is added … Read more
