Ionattention: Grace Hopper–Native Inference (cumulus.blog)

The recent announcement of ionattention, a C++ inference runtime tailored for NVIDIA's GH200 architecture, is a significant advancement in AI/ML performance optimization. Unlike traditional discrete GPUs, the GH200 integrates a CPU-GPU system with high-bandwidth memory and coherence capabilities that enhance execution efficiency. The ionattention runtime leverages this architecture to eliminate the limitations of static CUDA graphs, allowing for dynamic parameter updates without the need for costly recapturing or patching processes. This innovation has resulted in a remarkable benchmark performance of 588 tokens/second on multimodal pipelines, significantly outperforming competitors. Key technical innovations include utilizing coherent memory for dynamic CUDA graphs, implementing background copying of immutable key-value blocks to minimize latency during evictions, and employing phantom-tile scheduling to maximize GPU utilization during small batch sizes. These architectural optimizations have dramatically reduced processing times, improved throughput, and shifted performance bottlenecks favorably within the computation pipeline. The findings underline the potential of the GH200 to revolutionize inference processes by exploiting unique hardware characteristics, paving the way for more efficient AI workloads across various applications.