NanoGPT Speedrun Living Worklog (www.tylerromero.com)

A tech enthusiast has embarked on a personal project called the NanoGPT Speedrun, aiming to train the GPT-2 model on two RTX 4090 GPUs in record time. Inspired by previous speedrun results, he plans to achieve a validation loss of 3.28 on the FineWeb dataset, while documenting his progress in a public log. Initial attempts saw a baseline training time of 8.13 hours, but through a series of optimizations—including architectural tweaks, the implementation of a novel Muon optimizer, and data-loading adjustments—he has drastically reduced the training duration. His most recent improvement achieved a training time of just 4.01 hours, showcasing an innovative approach to maximizing performance through better resource utilization. This initiative is significant for the AI/ML community as it demonstrates practical applications of cutting-edge techniques to optimize large language model training. The use of advanced methods like Muon optimization and the exploration of longer sequence lengths provide valuable insights for researchers aiming to enhance throughput and efficiency in deep learning. By sharing his code and findings on GitHub, the author encourages collaboration and experimentation, potentially paving the way for broader advancements in model training speed and efficiency. As the project develops, it offers a roadmap for practitioners looking to push the boundaries of model performance on limited hardware.