Beyond FLOP/s: comparing AI hardware for inference by memory and communication (epoch.ai)

Recent findings reveal that improvements in AI hardware inference performance can significantly exceed what raw compute speeds (FLOP/s) indicate. The Nvidia GB300 chip boasts an estimated 25 times the inference capacity compared to the H100 model, despite having only 2.5 times the 8-bit FLOP/s. This disparity highlights the importance of memory and inter-chip communication in determining inference capabilities, a factor often overlooked when measuring pure arithmetic throughput. To quantify these differences, the Inference Performance Index (IPI) has been introduced, which assesses how many requests each chip can handle concurrently, based on diverse output speed demands. Key technical insights from the report reveal that not only does the GB300 NVL72 outperform the B300 HGX by about 1.9 times, but the parameters of system configuration and batch processing also significantly impact overall throughput. The IPI enables users to estimate data center inference capacities in a standardized way using GB300 equivalents, facilitating better comparisons across different hardware configurations. As the AI landscape evolves, this pioneering metric will adapt to changing workloads, ultimately refining our understanding of optimal inference performance and influencing design choices in AI hardware for diverse applications.