One Model, Two Accelerators: M6 Neural Engine and GPU for Prefill and Decode (huggingface.co)

🤖 AI Summary
A recent development in AI technology reveals a new hybrid model that utilizes both the Apple Neural Engine (ANE) and GPU on the M6 chip for enhanced performance in prefill and decode tasks. This innovative architecture allows the system to run the Qwen3.8-27B model simultaneously on both accelerators, achieving up to 40% faster prefill speeds and maintaining competitive decode speeds compared to the existing Splash 1.7.0 engine. Specifically, the V8.2 version demonstrated a remarkable improvement in energy efficiency, consuming 41% to 50% less energy per generated token, which is significant for sustainability in machine learning operations. The hybrid model's strategy leverages the strengths of each component; the Neural Engine excels in running feed-forward networks (FFNs), while the GPU handles token mixers effectively. Alongside this, technical enhancements, such as the introduction of an on-chip scratchpad for quicker write-backs, have resulted in better throughput during long-context requests, allowing the model to reach target speeds of 700 to 1000 tokens per second. These advancements promise not only faster processing times but also a notable reduction in overall energy consumption, making it a compelling solution for developers focused on high-performance AI/ML applications.
Loading comments...
loading comments...