Bf16 might be freezing your weight updates (lacuna.tiptreesystems.com)

Recent investigations into the AdamW optimizer's behavior with bf16-encoded model weights have revealed a significant issue that could freeze weight updates during fine-tuning. This effect occurs because bf16's limited precision can only represent weight changes that are substantial enough—approximately 1/512 of the weight itself—meaning that with small learning rates, only weights of very small magnitude are able to update. This leads to a phenomenon where the majority of weights remain unchanged, effectively creating a sparse fine-tuning scenario. The findings, likened to a bug, indicate that this issue has been overlooked in several notable research papers that employ similar fine-tuning methods but do not address the underlying impact of bf16 rounding. The implications of this research are profound for the AI/ML community, particularly for model fine-tuning strategies. It highlights the necessity of understanding the limitations of numerical representations when optimizing models, especially as newer architectures scale in complexity. The study underscores the idea that what might be perceived as an optimization disadvantage can inadvertently lead to stable training conditions, as seen in the Quiet-STaR approach. This revelation may prompt researchers to reconsider their methodologies and potentially develop more robust techniques that account for such weight-update dynamics, enhancing the reliability of fine-tuning processes across various applications.