Mtplx – 2.24x faster TPS – The native MTP inference engine for Apple Silicon (github.com)

Mtplx has unveiled a new native MTP inference engine that significantly boosts processing speeds on Apple Silicon, achieving approximately 2.24 times faster token processing per second (TPS) compared to traditional methods at a temperature setting of 0.6. This performance leap utilizes a math-correct approach to rejection sampling without requiring an external drafter, ensuring that the model's probability distribution remains intact during high-speed operations. Running on the Apple M5 Max, the system scales with memory bandwidth, marking a major advancement in efficiency for AI/ML frameworks. The MTPLX engine is not only open source under the Apache-2.0 license, which encourages modification and commercial use, but it also integrates seamlessly with existing OpenAI and Anthropic APIs, potentially extending its applications across various AI projects. It features an interactive installation process with customizable options, a browser-based chat interface, and supports live token download progress and fan control for optimal performance. This innovative runtime development represents a significant milestone in maximizing the efficiency of AI inference engines, particularly for macOS users, reinforcing the potential for faster and more reliable machine learning deployments.