The cut in the Mixture of Experts compute graph (idlemachines.co.uk)

Recent developments in the Mixture of Experts (MoE) architecture have unveiled significant limitations stemming from a critical ‘cut’ in the compute graph. This issue arises from the discrete nature of the routing decision, which utilizes a top-k selection method among expert networks. As a result, gradients are not properly transmitted back to the router, preventing it from learning optimal routing strategies. Instead, the training process relies on auxiliary losses to encourage balanced routing without directly informing the router about selection correctness, leading to challenges in expert specialization and potentially unstable training dynamics. The implications of this discovery are profound for the AI/ML community, particularly in the design of more efficient models. MoE architectures theoretically promise a robust increase in capacity with minimal compute overhead, as exemplified by architectures like Switch Transformer and Mixtral. However, the inability of the router to learn from selection signals risks over-reliance on popular experts while neglecting others, undermining the intended benefits. This highlights the necessity for innovative strategies to enhance routing effectiveness and expert utilization within complex model architectures, ultimately pushing the boundaries of model efficiency and performance.