Prefill a 284B model on Nvidia. Decode it on Apple Silicon. Over plain 10GbE (github.com)

A groundbreaking advancement in AI model processing has been achieved with the successful prefill of a staggering 284 billion parameter model using NVIDIA's infrastructure, followed by decoding it on Apple Silicon. This setup employs a seamless collaboration between two distinct production inference engines—DeepSeek-V4-Flash and oMLX—each optimized for their respective tasks. Notably, the entire process utilizes standard 10GbE Ethernet without the need for specialized cache formats or data transfer protocols, demonstrating a novel approach to model interoperability. This development is significant for the AI/ML community as it highlights an innovative solution for disaggregating model tasks based on hardware capabilities—compute-bound tasks for NVIDIA and memory-bound tasks for Apple Silicon. By computing the finished cache on the prefill engine and directly transferring the necessary data to the decoder, the team has managed to maintain model integrity without traditional cache sharing, achieving high efficiency and accuracy. The results indicate that large prompts are effectively handled, achieving impressive decoding speeds and allowing for a greater prompt ceiling than previously possible, thereby pushing the boundaries of what is feasible in multi-engine AI systems.