Optimizing On-Device Inference for Apple Silicon (www.perplexity.ai)

Apple has unveiled Lily, a specialized local inference engine optimized for Apple Silicon that enhances on-device processing of language models, specifically targeting the Qwen3.6-35B-A3B. This innovation seamlessly integrates local and cloud-based intelligence by efficiently managing tasks between complex cloud computations and private, on-device analyses. The engine is designed to rapidly process prompts and maintain a high token-generation rate, crucial for a smooth user experience. The significance of this development lies in its potential to transform AI/ML capabilities on personal devices. By achieving superior performance metrics—averaging 1.23 times the prefill throughput and 1.35 times the decode throughput compared to existing solutions, such as MLX-LM—Lily not only accelerates the processing speed of language models but also demonstrates a new potential for running advanced AI applications directly on consumer hardware. By using a specialized Rust runtime and custom Metal kernels, it maximizes efficiency while minimizing the need for more resource-intensive frameworks like PyTorch. This tailored approach enables rapid, real-time interactions that were previously limited to cloud-based solutions, heralding a new era of privacy-focused, efficient AI processing on personal devices.