Distributing LLM Inference in DwarfStar (antirez.com)

In a recent exploration of distributed inference for large language models (LLMs), a new approach dubbed "DwarfStar" has emerged, utilizing the Mac Studio M3 Ultra and M5 Max. This initiative addresses the high costs associated with NVIDIA hardware necessary for running extensive models, making local inference more accessible. With configurations like the Mac Studio M3 Ultra featuring 512GB of unified memory, users can achieve prefill speeds of approximately 150 t/s and decoding rates of 10-13 t/s, marking a notable option for developers seeking alternatives to expensive data center setups. The significance of this development lies in the potential for distributed inference across multiple Mac systems, which can enhance processing speeds and efficiency. By employing techniques like memory duplication and vertical execution splits using Apple’s RDMA, users can run models like DeepSeek v4 PRO across several machines, optimizing resource utilization and speed. Furthermore, the exploration of ensemble methods—allowing models to collaboratively generate predictions while sidestepping traditional limitations—could yield improvements in performance, as combining insights from different models may lead to more nuanced outputs. With promising results from recent research on model ensembles, DwarfStar could redefine how smaller developers approach LLM inference, leveraging cost-effective hardware solutions to stay competitive in the evolving AI landscape.