Lily-Qwen3.8-Flash-Next (github.com)

The recent release of the lily-qwen3.8-flash-next Metal inference server marks a significant advancement for AI and machine learning applications on Apple Silicon. This server is tailored for the Qwen3.8-Flash-Next model and is a fork of Perplexity’s lily, boasting improvements such as hand-written Metal kernels and innovative features like speculative decoding and an expert cache. These enhancements have resulted in performance benchmarks demonstrating prefill speeds 2.7 to 4.2 times faster and decoding speeds 2.1 to 3.6 times faster than existing solutions on similar hardware. For the AI/ML community, this development represents a crucial step in optimizing model inference speed and resource efficiency. The architecture, featuring 48 layers with a Gated DeltaNet and sparse attention mechanisms, allows for advanced techniques like cached conversations and durable prefixes to enhance user experience. With its compatibility with OpenAI’s API and support for up to 128 GB of unified memory, the new server opens up possibilities for more robust and responsive AI applications operating on Apple’s GPU infrastructure. The implications of this optimized server extend to a wide range of practical uses, from real-time conversational agents to complex data processing tasks.