Lithos-metal: Qwen3.8-27B on 40-core M5 Max at 168 tokens/s (github.com)

Lithos-metal has launched a fully open-source inference engine designed to facilitate low-latency LLM (Large Language Model) inference on Apple silicon devices. This innovative engine leverages Metal megakernels to enhance performance capabilities, achieving impressive token generation speeds—approximately 168 tokens per second on a 40-core M5 Max with 48 GB of unified memory. By combining advanced techniques such as layer-wise mixer megakernels, GPU-resident generation control, and DSpark speculative decoding, lithos-metal optimizes resource usage and minimizes execution overhead, making it ideal for coding agents and other AI applications. This development is significant for the AI/ML community as it democratizes access to powerful LLM inference capabilities on widely used Apple hardware, enabling developers to run sophisticated models locally. The engine's design empowers users to tailor configurations based on their specific models and memory availability while supporting dynamic memory allocation and efficient task scheduling. With compatibility across various Apple silicon backends and support for multiple client applications, lithos-metal represents a major step towards enhancing the accessibility and efficiency of AI inferencing on consumer-grade devices, effectively bridging the gap between high-performance computing and everyday computing environments.