Loop the Loopies (arxiv.org)

Researchers have introduced the Loopie series, which features two Mixture-of-Experts (MoE) models: a 20 billion-parameter model with 2 billion active parameters and a 6 billion-parameter model with 600 million active parameters. This development addresses a longstanding issue in Loop Transformers, where increasing compute usually yields better performance than simply scaling up parameters. Through extensive ablation studies, the Loopie models significantly outperformed a vanilla 30B-A3B model, leveraging a novel post-training method that enhances reasoning capabilities and reaches state-of-the-art reasoning performance. The significance of the Loopie series lies in its innovative approach to model efficiency and effectiveness, redefining how researchers can optimize resource use in AI/ML workloads. By demonstrating robust performance within a limited compute budget, Loopie sets a new benchmark for future models, suggesting that smarter architectures can supersede brute-force scaling. This advancement not only contributes to more efficient AI systems but also strengthens the foundation for developing more sophisticated reasoning capabilities in machine learning.