Make your own Darkbloom – The Expert Network pools Macs into one model (github.com)

In a groundbreaking move for the AI/ML community, the Expert Network has unveiled a method to harness the power of multiple ordinary computers to collaboratively run Mixture-of-Experts (MoE) models. Instead of relying on a single device, this network pools resources to execute complex models, activating only a fraction of the available experts per computation. The significant advantage lies in the ability to store the bulk of the model's weights remotely on SSDs while transferring only necessary activation data between devices, effectively optimizing inference performance and reducing local memory requirements. The innovative architecture employs two tiers: a single workstation uses low-latency memory to handle essential computations while streaming expert data from connected SSDs, and a distributed network assigns expert partitions across multiple nodes. This results in impressive performance benchmarks, with speeds of up to 19.3 tokens per second when operating across two machines, outperforming single-device setups. The system's design is highly scalable; adding more nodes enhances both expert cache and SSD bandwidth, enabling more extensive and efficient model training and deployment. Developers can easily integrate this system on Apple Silicon through a pip-installable package, positioning it as a promising solution for AI developers seeking cost-effective ways to leverage powerful ML models.