Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon (tridao.me)

The introduction of Gram Newton-Schulz represents a significant advancement in optimization algorithms for training large language models, particularly through its integration into the Muon optimizer. This novel approach reworks the traditional Newton-Schulz routine by iterating on a smaller symmetric Gram matrix, resulting in a reduction of computational overhead by up to 50% for trillion-parameter models like Kimi K2. Unlike earlier versions, Gram Newton-Schulz leverages specialized symmetric matrix multiplication routines, enhancing performance and reducing the time required for each optimization step. This development is particularly impactful for the AI/ML community as it maintains the optimization quality comparable to previous methods while providing a near "free lunch" performance improvement—achieving stability with just a 0.01 validation perplexity deviation. The implementation of custom kernels for GPU architectures optimizes resource utilization, allowing researchers and developers to train state-of-the-art models more efficiently. With a focus on preserving accuracy and facilitating GPU performance, the release of open-source implementations of Gram Newton-Schulz will likely accelerate innovation and scalability in the training of advanced language models.