Aurora: A Leverage-Aware Optimizer for Rectangular Matrices (blog.tilderesearch.com)

The Aurora optimizer has been introduced as a solution to address critical issues raised by the Muon optimizer in training large machine learning models, particularly in preventing neuron death in multi-layer perceptrons (MLPs). The research highlights that while Muon's update mechanism can lead to significant portions of neurons being inactive, Aurora incorporates a strategy of row-norm uniformity that prevents this issue without sacrificing the essential property of orthogonality in gradient updates. By leveraging this approach, Aurora was able to train a 1.1 billion-parameter model with remarkable efficiency, achieving a 100-fold improvement in data efficiency compared to existing methods. Notably, Aurora not only resulted in superior performance in terms of training speed during the nanoGPT speedrun but also outperformed larger models on standard evaluation benchmarks such as HellaSwag. With a minimal overhead of just 6% over Muon, Aurora serves as a drop-in replacement that promises to enhance the training of deep learning models, particularly those with wider architectures. The code for Aurora has been open-sourced, allowing the AI/ML community to leverage these advancements in their own model training efforts. This development may signal a pivotal shift in optimizing large-scale models, enhancing both performance and efficiency in various applications.