Running Kimi K3 on MI355X at Better Performance per Dollar Than B300 (www.wafer.ai)

The recently reported performance of the Kimi K3 model running on AMD’s MI355X GPU is significant for the AI/ML community, marking a potential shift in the landscape of hardware utilization for large open-source models. With Kimi K3 boasting an impressive 2.8 trillion parameters, it presents an opportunity for developers to harness a more cost-efficient alternative to established NVIDIA GPUs, notably the B300. The MI355X delivers superior performance per dollar, achieving 952 tokens per second across a node and offering 2.4 times better cost efficiency compared to the B300 while still being competitive in aggregate throughput. Key technical advancements include resolving significant compatibility issues within the ROCm framework that hindered Kimi K3's initial performance, such as establishing stability for speculative decode paths and optimizing prefill speeds considerably. These enhancements have led to a striking increase in throughput and efficiency, highlighting how the MI355X’s architecture, particularly its high-bandwidth memory capacity, could facilitate the successful deployment of extraordinarily large models. As open-source AI continues to evolve, AMD's growing capabilities may challenge CUDA's prevailing dominance in high-performance machine learning applications, suggesting a noteworthy evolution in the ecosystem of AI infrastructure.