N-gram hinted language modeling: NanoGPT Speedrun in 6.7s (brunborg.com)

A recent breakthrough in the NanoGPT Speedrun has reduced the world record training time from 39.9 seconds to an astonishing 6.7 seconds, leveraging retrieval-augmented n-gram hints. Since its inception in May 2024, the speedrun has focused on training models to achieve GPT-2-level loss rapidly, with nearly 100 records set. The key innovation lies in utilizing idle CPU resources to build a lookup table that identifies which tokens typically follow specific n-grams (sequences of n tokens) from the training data. This allows the model to receive hints about probable next tokens, significantly improving its learning efficiency. This development holds substantial implications for the AI/ML community, as it illustrates the potential of integrating simple yet effective techniques, such as n-gram hinting, to enhance model training speeds dramatically. The method showcases a meticulously designed hint-building algorithm that offers better cache utilization and drastically reduces memory access times. By training on a smaller hint corpus initially and scaling up during validation, the model benefits from improved accuracy without becoming overly reliant on hints. Furthermore, the insights gained could inspire further optimizations in model training processes, potentially leading to even faster training times as initial tests suggest possible reductions to as little as 3.1 seconds when integrating GPU capabilities.