Writing an LLM from scratch, part 32d – Interventions: adding attention bias (www.gilesthomas.com)

In the latest installment of his series on building a large language model (LLM) from scratch, the author explores the impact of adding attention bias—specifically, a bias to the query-key-value (QKV) weight matrices—using a small GPT-2 model. This intervention follows prior experiments aimed at reducing test loss, with the rationale that while modern LLM architectures often exclude bias, there could be potential benefits for smaller models like the one in this study. The experiments revealed that introducing QKV bias resulted in a slight improvement in test loss, highlighting previously overlooked advantages in model stability, as evidenced by a test loss of 3.669 compared to a baseline of 3.692. The significance of this finding lies in its challenge to the prevailing assumption that bias has negligible effects on LLM performance, particularly in smaller configurations. By adding only 27,648 new parameters—representing less than 0.02% increase in total size—and still achieving a notable improvement in test metrics, it emphasizes the delicate balance between model complexity and performance. As LLM research continues to trend towards larger models, this study underscores the value of iterative experimentation in fine-tuning architectures, with the next phase focusing on adaptations in learning rate and decay parameters to further enhance model performance.