Field Notes on Scaling Moe Expert Parallelism with DeepEP (nousresearch.com)

A recent update from a research team showcases their journey in scaling Mixture of Experts (MoE) training using a specialized framework called DeepEP integrated within a modified version of Torchtitan. Faced with challenges like poorly scaling intranode kernels and benchmarking issues, the team addressed performance bottlenecks by optimizing kernel launch configurations and tuning GPU compute resources. Their newfound configuration allows DeepEP to achieve significant performance improvements, reaching about 57% of the theoretical limit, compared to the baseline which only managed 34%. This work is significant for the AI/ML community as it highlights the trial-and-error nature of developing scalable architectures for deep learning, particularly in distributed systems. The ability to optimize expert parallelism can greatly enhance training efficiency and performance, making it a vital advancement for large models. The team's exploration into the CPU overhead during training, which accounted for a substantial portion of total processing time, underscores the importance of both GPU and CPU optimizations in achieving seamless scaling in deep learning workflows.