Apple Silicon Inference Engine: Kapitsa (github.com)

Apple has announced the launch of Kapitsa, a new inference engine designed specifically for Apple silicon, capable of running advanced coding models like Qwen, Nemotron, and Recursive's Flint locally. Kapitsa significantly enhances local processing capabilities by enabling fast prompt decoding and serving external APIs from OpenAI and Anthropic. In tests, Kapitsa demonstrated superior performance, achieving decoding speeds of 61 tokens per second on an M1 Pro, outpacing other engines such as llama.cpp in various configurations, making it a powerful tool for developers working with AI-driven coding agents. The significance of Kapitsa lies in its speed and efficiency, boasting the fastest decoding rates in 32 of 36 tested model-and-Mac combinations against nine competitors. Key innovations include KFlash, which utilizes speculative decoding for increased throughput, and an SSD prefix cache that dramatically reduces restart times from 17 seconds to 0.16 seconds for up to 16K-token follow-ups. Additionally, features such as prompt prefill and prefix reuse optimize response times even further, streamlining workflows for developers. Overall, Kapitsa represents a major leap forward in local AI processing, solidifying Apple’s position in the AI/ML ecosystem.