Decoupled DiLoCo: Resilient, Distributed AI Training at Scale (deepmind.google)

Google has announced the Decoupled DiLoCo, an innovative training system that enhances resilience and efficiency for distributed AI training at scale. This system successfully trained a 12 billion parameter model across four U.S. regions using standard broadband connectivity, achieving speeds over 20 times faster than traditional methods. By integrating communication into longer computation periods, it mitigates blocking bottlenecks and maximizes resource utilization, making it feasible to optimize AI training without the need for custom infrastructure. The significance of Decoupled DiLoCo lies in its ability to leverage unused compute power from various hardware generations, such as TPU v6e and TPU v5p, within a single training run, thereby extending the lifespan of existing hardware and boosting total compute capacity. This paradigm allows for effective mixing of different equipment, addressing logistical challenges and capacity limitations in AI training. As Google further explores resilient AI infrastructure, this approach could potentially revolutionize how AI models are pre-trained, opening new avenues for efficiency and performance across the AI/ML community.