Reading MAI's efficiency gain. How to pick architectures like serious people (idlemachines.co.uk)

Microsoft's latest report on the MAI-Thinking-1 project highlights a significant advancement in model efficiency through a new metric termed "efficiency gain" (EG). This 35-billion active, one-trillion total sparse Mixture of Experts (MoE) reasoning model addresses a critical challenge in AI model design: balancing computational budget with loss performance. The report emphasizes the importance of assessing not only theoretical floating-point operations per second (FLOPs) but also the practical wall-clock time for training, which can diverge significantly depending on the model architecture and hardware setup. The EG metric allows researchers to evaluate how much better or worse a model design performs relative to a baseline across these two dimensions. By using EG, researchers can make informed decisions about architecture choices that optimize both resource usage and training time. The report illustrates this with a comparison of different MoE architecture configurations, revealing that apparent wins in FLOPs may not translate to real-world performance due to inefficiencies in kernel execution. This nuanced understanding fosters a deeper exploration of model designs by providing a defensible means to quantify trade-offs, ultimately guiding the AI/ML community towards more efficient and practical innovations in model development.