Moe inference optimizations: 15% lower expert load by request reordering (blog.doubleword.ai)

Doubleword has announced a significant optimization for Mixture-of-Expert (MoE) models that enhances inference efficiency by approximately 15%. The technique involves reordering input requests to cluster similar prompts together, reducing the number of unique expert weights that need to be loaded during each forward pass. This strategy addresses the memory-bandwidth limitations often faced with MoE architectures, where different prompts typically require varying expert weights, thus hampering throughput compared to dense models. By implementing an embedding model to co-locate requests based on cosine similarity, Doubleword achieved a reduction in expert loads, resulting in improved throughput without needing any changes to the model architecture. The implications of this optimization are substantial for the AI/ML community, as it allows for faster inference times without incurring additional costs associated with hardware upgrades. In tests with Qwen/Qwen3.5-35B-A3B, the new method demonstrated up to a 21.3% reduction in expert loads through efficient batching strategies. This approach not only translates into quicker wall-clock operation times, particularly in production environments where parallel processing is common, but also opens avenues for further research into prompt ordering techniques that could enhance MoE performance even further. Overall, this advancement enhances the efficiency of MoE models, potentially making them more accessible for real-time applications.