How to Scale (howtoscalenn.github.io)

This post distills practical lessons on “how to scale”: not just that more compute (C ≈ 6ND) improves models, but how to set initialization, learning rates and batch sizes as width/depth and dataset size grow. The author highlights Maximal Update Parameterization (muP), from the Tensor Program line of work (TP-V / muTransfer), as a theoretically grounded way to make optimal hyperparameters transfer across model widths. Unlike Standard Parameterization (SP), which focuses only on initialization and yields “left‑shifted” LR curves and uneven per‑layer learning, muP prescribes per-parameter init stds, multipliers and learning rates so that pre‑activations remain Θ(1) during training and every layer learns features maximally even as width → ∞. A practical rule that emerges: scale learning rates with width (e.g., halve LR when doubling hidden size), enabling cheap tuning on small models that transfers to large ones. Technically, muP’s value is that it avoids the NTK/kernel regime where features freeze, and it stabilizes gradient magnitudes across layers so some weights don’t dominate while others stagnate. The framework underpins reported use in large LM training and can improve LM performance and HP transfer, though it doesn’t directly address dataset scaling and may yield diminishing returns at extreme scales. Complementary tools—better normalization schemes and advanced optimizers (e.g., Muon, SCION)—can help, but muP gives a principled foundation for scaling hyperparameters and preserving effective feature learning.