Chasing Speed of Light on TPU v6e (www.sailresearch.com)

Google has introduced HTDYM, a performance modeling tool aimed at optimizing the use of AI chips for deploying machine learning models. The tool helps in evaluating cost-effective accelerators such as the TPU v6e, which, despite its lower memory capacity and bandwidth compared to NVIDIA's H100, can outperform in specific workloads. A recent optimization journey demonstrated improved performance of the Gemma 4 31B model on TPU v6e, boosting the Model FLOPs Utilization (MFU) from 32% to an impressive 63%. The TPU v6e is designed with a unique architecture that includes significant compute power through 256x256 systolic arrays. However, it faces challenges from its limited memory bandwidth and inter-chip connectivity, especially when serving medium-sized models that require efficient sharding. Innovations like 'collective matmuls' were explored to manage inter-chip communication efficiently, while leveraging Google's Pallas framework allowed for more granular control of memory movement and communication, further enhancing performance. This development highlights the potential of specialized models on niche accelerators to achieve cost-effective solutions for AI workloads, reinforcing the importance of tailored optimization in the ever-evolving landscape of AI hardware.