New NanoGPT speedrun record with ANVIL II (hyperstition.cc)

A new record for the NanoGPT speedrun has been set, demonstrating a significant 46% reduction in training time, achieving a validation loss of 3.278 on FineWeb in just 39.914 seconds using 8 H100 GPUs. This breakthrough is attributed to several innovations, primarily the introduction of ANVIL II, an enhanced optimizer that better maintains momentum streams and implements late-stage weight averaging. These techniques not only streamline computations but also improve the model's ability to converge on lower validation losses effectively. Furthermore, the implementation of sampled softmax and an expanded sparse n-gram table—now featuring 84.6 million rows—contributes to this speedup by reducing the computational burden during training. By selectively scoring vocabulary elements and optimizing gradient updates, efficiency is maximized, allowing the model to process batches faster while maintaining performance quality. These advancements have significant implications for the AI/ML community, showcasing how algorithmic improvements can lead to enhanced model training efficiency and speed, which could accelerate research and development in language modeling.