TPS on Kimi K3 with TPU Megakernels (inferact.ai)

Inferact has launched a new collection of megakernels for Google's TPU v7, named inferact/tpu-megakernels, which significantly boosts decoding speeds for the Kimi K3 model. Achieving over 700 tokens per second with speculative decoding, this implementation outperforms the previous GB200 GPUs, delivering nearly double the decode throughput at smaller batch sizes. The design leverages TPU's large on-chip memory (VMEM) and a unique programming model that allows for more efficient data prefacing, pushing performance closer to the hardware's peak memory bandwidth. This development is crucial for the AI/ML community as it optimizes inference capabilities on TPUs, which can be pivotal for deploying large models more effectively and efficiently. The megakernel integrates 92 layers of the Mixture-of-Experts architecture while allowing concurrent operations between weights and computations, reducing memory bottlenecks typical in GPU architectures. Future work will focus on scaling the megakernel for diverse workloads, adapting to various chip topologies, and enhancing tools for developers. This innovation exemplifies how tailored computational strategies can enhance model performance in real-world applications.