It's Always the Learning Rates (ianbarber.blog)

In a recent exploration of learning rates in large language model (LLM) training, Lilian Weng of Thinky emphasizes the critical role of scaling laws, which illustrate how training loss decreases predictably as model size, dataset size, and compute resources increase. These scaling laws serve as a framework for optimizing resource allocation, allowing researchers to estimate token and compute needs for larger models based on smaller, more affordable training runs. A significant aspect of this work is the importance of selecting the right learning rate, as an inappropriate choice can skew experiment outcomes dramatically. Weng highlights recent findings by Zhou et al., who argue for a more straightforward approach to selecting learning rates for large models by fitting learning rates directly from smaller scales rather than transferring them via established methods like Maximal Update Parametrization. Their method involves training on a reduced dataset and keeping consistent width-to-depth ratios, leading to a single, effective global learning rate suited for models scaling up to ten times larger. This approach challenges conventional wisdom and underscores the variability and unpredictability inherent in LLM training, ultimately suggesting that even minor procedural choices can lead to significant discrepancies in scaling law applications.