Decoupled DiLoCo for Resilient Distributed Pre-Training [pdf] (storage.googleapis.com)

Google DeepMind and Google Research have unveiled Decoupled DiLoCo, a revolutionary distributed training framework aimed at enhancing the pre-training of large-scale language models (LLMs). Unlike traditional methods that rely on the tightly coupled single program multiple data (SPMD) paradigm, which makes them vulnerable to hardware failures and synchronization delays, Decoupled DiLoCo introduces a system where multiple independent “learners” conduct local optimization steps and communicate asynchronously with a central synchronizer. This approach aims to maximize training efficiency while maintaining model performance, mitigating the substantial downtime often encountered in failure-prone computing environments. The significance of Decoupled DiLoCo lies in its ability to prioritize availability and partition tolerance over strict consistency, thus breaking the synchronization barrier that traditionally hampers training processes. By allowing learners to operate independently, the framework effectively localizes the impact of hardware failures, preventing entire systems from stalling due to isolated issues. Empirical evidence shows that the system achieves comparable model performance across various architectures and tasks, including text and multi-modal evaluations, all while enabling high availability. This innovative approach to LLM pre-training could pave the way for more resilient and efficient training practices in the AI/ML community, particularly as model sizes and training scales continue to grow.