The Hitchhiker's Guide to Training on Compute Islands – Core Automation (www.coreauto.com)

In a groundbreaking study, researchers successfully trained a sparse mixture-of-experts model across two geographically distant compute clusters, located nearly 9,000 miles apart, utilizing different types of GPUs. This innovative approach involved pairing a 160 B300 cluster in the U.S. with a 160 GB300 cluster in Asia, linked via a high-capacity 25 Gbps connection. By employing a novel "butterfly mixing" strategy with pairwise exchanges for model updates, the team showcased improved synchronization that cut down traditional communication overhead, achieving almost the same evaluation metrics as standard synchronous training. This work is significant as it addresses the growing challenge of utilizing fragmented compute resources effectively. The AI/ML community faces a bottleneck not due to a lack of processing power but because of the inefficiencies in deploying widely scattered infrastructure. By turning "compute islands" into cohesive and productive training systems, this methodology could enable teams to harness available resources better, potentially increasing the efficiency of frontier training endeavors—crucial for advancing AI capabilities amidst ongoing hardware and power constraints. The study further demonstrates the adaptability of training architectures to overcome traditional limitations, paving the way for future innovations in distributed learning.