LoRA and Weight Decay (2023) (irhum.github.io)

A recent blog post explores the interaction between LoRA (Low Rank Adaptation) and weight decay in the fine-tuning of Large Language Models (LLMs). Traditionally, fine-tuning involves adjusting all the model's weights to improve task-specific performance, but LoRA simplifies this by introducing smaller adapter matrices that modify frozen base model weights. The key finding is that LoRA, while offering significant memory efficiencies and flexibility, approaches a different optimization problem due to its weight decay strategy; it biases the adjusted weights towards the original model instead of moving them towards zero, as seen in full fine-tuning. This means that even with increased resources or larger adapter ranks, LoRA does not converge to the same solutions as full fine-tuning. Understanding this discrepancy has important implications for practitioners in the AI/ML community. It highlights the necessity of explicitly accounting for this behavior when choosing between full fine-tuning and LoRA, depending on the use case. The post also proposes a modified weight decay approach that could align LoRA's objectives closer to those of full fine-tuning, suggesting a viable pathway for developers looking to optimize LLM performance in real-world applications.