Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio (machinelearning.apple.com)

Apple has announced the integration of MLX, an open-source framework, with the Mac Studio powered by Apple silicon, allowing researchers to conduct local and distributed inference of large language models (LLMs). The latest version enables users to run Muse Spark Open, a frontier-class model with 816 billion parameters, achieving impressive performance benchmarks such as generating over 36 tokens per second on a cluster of four Mac Studios. This advancement serves as a significant milestone for the AI/ML community, enhancing accessibility to high-performance model training and inference within personal and small-scale setups. The MLX framework leverages Apple silicon's unified memory architecture and introduces advanced communication protocols over Thunderbolt, including RDMA, facilitating efficient model parallelization techniques like pipeline and tensor parallelization. These features enable researchers to distribute workloads across multiple machines, essential for handling massive models that cannot be accommodated by a single device. Key technical innovations include a dramatic increase in memory bandwidth with the M5 Ultra chip, improving inference speed, power efficiency, and computation capabilities. This setup positions the Mac Studio as a compelling option for developers and researchers focused on cutting-edge AI applications, particularly in private, home, or office environments.