Porting nanochat to a TPU: what carries over from PyTorch, and what breaks (github.com)

Recent developments in the AI/ML space have seen a successful port of Karpathy's "nanochat" model from PyTorch to JAX running on TPU v6e, demonstrating both the strengths and challenges of the transition. The aim was to maintain parity with the original architecture while enhancing model quality and training performance. The port achieved a CORE score of 0.2695, slightly exceeding previous benchmarks while training performance trailed Karpathy's original measurements on H100 GPUs, achieving around 24% MFU—less than half of the expected efficiency. This transition is significant for the AI/ML community as it illustrates the ongoing evolution and optimization of large language models (LLMs) and their deployment across different hardware architectures, like TPUs. The findings reveal critical insights about data handling and model efficiency, highlighting the importance of careful data management and the benefits of combining multiple computational steps into a single, optimized process. The successful reproduction of model quality showcases the potential of JAX and TPUs in scaling AI solutions, while also serving as a cautionary tale regarding the hidden complexities involved in ports between different frameworks.