I tried to do Kimi K3 inference, I still couldn't figure out bottleneck (ramshankar07.substack.com)

In an exploration of serving the Kimi K3 model, a cutting-edge 2.8 trillion-parameter MoE (mixture of experts) AI model, a developer attempted to offer more competitive pricing against third-party APIs, which typically charge around $15 per million output tokens. By renting high-performance GPUs and using the vLLM framework in partnership with Mooncake, the developer sought to identify the true operational costs of serving K3. However, despite achieving impressive output rates, they discovered that maintaining profitability and effective pricing is challenging. The model's GPU usage is limited by CPU bottlenecks, highlighting inefficiencies in resource allocation and suggesting that additional computational power isn't always the solution. The project's insights are significant for the AI/ML community as they underscore the complexities of model deployment and cost management. With the required sustained utilization rates near 75% to be financially viable, service providers need to optimize not just their infrastructure but also usage patterns to ensure profitability. The exploration brought to light constraints around the infrastructure setup and potential areas for future improvement, particularly emphasizing the need for advanced scheduling and resource management techniques. This inquiry not only contributes to discussions around model serving economics but also underlines the importance of cross-optimization between CPUs and GPUs in machine learning operations.