Faster Local LLMs with iPhone Offloading (github.com)

A new development allows users to enhance local performance of the Qwen3.8-27B language model by utilizing an iPhone in tandem with a MacBook. This setup can significantly accelerate the model's prefill speed—boosting it by 29-44% depending on the context size—thanks to a unique offloading mechanism where the Mac processes the first 40 layers while the iPhone handles the remaining layers on its GPU. Notably, this approach allows for handling up to 229k tokens of context, leveraging the iPhone’s memory to increase the model's capacity beyond what the Mac holds alone. This advancement is significant for the AI/ML community as it presents an innovative strategy to improve computational efficiency in local setups, reducing latency and enhancing model responsiveness. The integration of sophisticated techniques like split prefill and the use of the iPhone's Neural Engine for old key attention processing showcases how existing consumer hardware can be repurposed to optimize AI performance. The results indicate that as models grow larger, this hybrid architecture could facilitate more complex tasks without needing extensive cloud resources, paving the way for more accessible and powerful local AI solutions.