What kept GPUs waiting: an E1 study of distributed training overhead (nodeox.com)

Researchers at Nodeox conducted a significant study on distributed training overhead in deep learning, highlighting critical delays that occur outside of GPU computation. Their findings, based on phase-level records from experiments involving different model configurations, reveal that substantial time is lost on CPU-side tasks such as encoding updates and unnecessary model persistence. Notably, the study identified instances where two hosts in a 16-A40 run experienced encoding delays of nearly 18 seconds, compared to just 1.5 to 2.3 seconds on other hosts. These delays are attributed to CPU oversubscription as well as continued full-model saving practices, which were unnecessarily prolonging round times. These insights are vital for the AI/ML community as they provide a clearer understanding of the bottlenecks in distributed training systems and suggest targeted engineering improvements. By moving the coordinator closer to the GPUs and changing the model-saving methodology to be on-demand, researchers reported a reduction in synchronization intervals from approximately 13 seconds to under 10 seconds. The study emphasizes the importance of phase-level measurements and granularity in identifying and addressing inefficiencies in training workflows, ultimately paving the way for improved training times and better resource utilization in future large-model experiments.