Serving a 1T open model at 511 tok.s single-stream, lossless, on four B200s (brainsless.com)

A breakthrough in AI model serving has been achieved with the Kimi-K2.6, a trillion-parameter open model, operating at an unprecedented speed of 511.6 tokens per second using just four B200 GPUs. This record was initially set at 505.9 tokens per second and was raised following blind re-runs of the unmodified public release, demonstrating that the performance is repeatable and remains robust against previous measures. The system processes 10,000-token inputs with 2,048-token outputs while maintaining lossless decoding, meaning the model fully verifies every drafted token. Notably, per accelerator performance stands at approximately 127.9 tokens per second per GPU, roughly three times that of the best reported configurations, positioning this outcome as a significant target for future optimizations in model serving. This achievement is significant for the AI/ML community as it underscores the feasibility of efficiently serving massive models in real-time, which can propel practical applications in various domains, from chatbots to advanced AI tools. The configuration is entirely stock and public, establishing a new standard for transparency in performance claims. Furthermore, a new drafter program, Fovea-E, trained on diverse conversations, enhances the model’s output metrics further, highlighting the critical interplay between model design and practical efficiency. As performance replication improved rather than declined, this work sets a solid foundation for building more powerful and efficient AI systems capable of handling increasingly complex tasks.