With TPU 8, Google Makes GenAI Systems Better, Not Just Bigger (www.nextplatform.com)

Google has announced the launch of its TPU 8 series, featuring distinct architectures for training and inference, marking a significant evolution in its Tensor Processing Units after a decade. The TPU 8t is optimized for training and employs advanced technologies like a new inter-chip interconnect that doubles bandwidth compared to previous models, while also introducing a new FP4 format that enhances performance significantly. Conversely, the TPU 8i focuses on inference tasks, equipped with a Collectives Acceleration Engine that drastically reduces latency for tasks like auto-regressive decoding, essential for the responsiveness of generative AI systems. This architectural separation signifies a shift in how Google approaches AI model training and inference, acknowledging that each workload has unique computational and networking requirements. By tailoring the TPU 8 chips to handle specific tasks more efficiently, Google aims to improve performance per watt, with reported enhancements of 2.7X for training and 1.8X for inference compared to previous generations. This innovation not only addresses the growing complexities of generative AI but also positions Google’s TPU offerings as more competitive against rivals like Nvidia, paving the way for faster, more efficient AI applications in its vast data center operations.