MoE-Hub Taming Software Complexity for Seamless MoE Overlap on Multi-GPU Systems (arxiv.org)

Researchers have introduced MoE-Hub, a novel solution addressing critical scalability challenges in Mixture-of-Experts (MoE) architectures when deployed on multi-GPU systems. The current limitations stem from inter-GPU communication bottlenecks and a mismatch between the dynamic token-to-expert mapping of MoE and the static communication model used by GPUs, resulting in performance lags and programming complexities. MoE-Hub proposes a hardware-software co-design that adopts a destination-agnostic communication approach, allowing immediate data transfers post-routing while delegating address management to specialized hardware within the GPU hub. This advancement is significant for the AI/ML community, particularly for researchers and developers working with large language models, as it enhances the efficiency and flexibility of parallel processing on GPU infrastructures. The evaluation reveals that MoE-Hub delivers impressive performance improvements, achieving between 1.40x to 3.08x speedup per-layer and 1.21x to 1.98x end-to-end speedup compared to existing systems. By streamlining communication control and optimizing resource utilization, MoE-Hub paves the way for more scalable and efficient AI model training, ultimately accelerating innovation in AI-driven applications.