OMLX: Use a second Mac to speed up local LLM prefill 2.9x (github.com)

oMLX has released version 0.7.4.dev1, introducing a groundbreaking feature called the Prefill Accelerator. This enables users to connect a secondary Mac with a high-performance GPU to enhance prefill speeds for large language models (LLMs) that may not fit entirely on a primary Mac. For instance, when pairing an M3 Ultra with a fast M5 Max, prefill speeds for models like DeepSeek V4.1 and Qwen3.8 Flash-Next increased significantly—up to 195% faster—demonstrating a substantial boost in performance for long prompts, which is crucial for developers working with memory-intensive AI applications. The integration of this feature is significant for the AI/ML community as it allows for efficient distributed inference without the requirement of complex network setups. Users can utilize standard Thunderbolt or 5GbE connections to synchronize processing across devices, enabling a seamless user experience. Beyond speed improvements, oMLX has optimized its structure by maintaining a bounded buffer pool during processing, leading to a 43-48% increase in speed for DeepSeek V4.1 on the M5 Ultra. With adaptations that minimize memory pressure during consecutive long prompts, this update enhances the overall usability and capability of AI deployments in real-time applications.