Skymizer and MiTAC to Show Kimi K3 2.8T on a Single Air-Cooled Server at SC26 (skymizer.ai)

Skymizer and MiTAC Computing have announced a significant advancement in AI deployment technologies with their showcase of the Kimi K3, a 2.8-trillion-parameter mixture-of-experts model, running on a single air-cooled server at SC26. This setup utilizes eight HTX301 LPU cards combined with 2 TB of card memory, designed specifically to manage the model's substantial memory requirement while allowing for on-premises deployment for enterprise customers. The model leverages quantization-aware training to optimize its weight configuration and memory use, enabling efficiency and scalability in AI applications. This initiative is particularly noteworthy as it addresses the growing enterprise demand for maintaining greater control over AI models and data within their own infrastructure. By efficiently fitting such a large model into a single server, Skymizer and MiTAC aim to simplify the complexities associated with deploying ultra-large AI models in sectors like finance, healthcare, and government. Their strategy emphasizes a memory-residency and scheduling approach to optimize performance, while plans for AI-assisted delivery services will help enterprises streamline model selection and deployment processes. This showcases a clear direction towards making high-performance AI solutions more accessible and manageable for enterprises.