Polar Factor Beyond Newton-Schulz – Fast Matrix Inverse Square Root (jiha-kim.github.io)

The Muon optimizer has introduced an advanced method for performing fast matrix inverse square roots, enhancing its application in machine learning. This development centers on computing the orthonormal polar factor of tall matrices efficiently, defined as \( \mathrm{polar}(G) = G(G^\top G)^{-1/2} \). The significance of this approach lies in its reliance on fast GPU computations while ensuring numerical stability, particularly in lower precision formats like bf16. By leveraging rectangular General Matrix-Matrix Multiplications (GEMMs) and optimizing via minimax polynomials and online selection techniques, the method allows for quick approximations that perform significantly better in practical machine learning scenarios. Key technical innovations include a streamlined process that eliminates the need for high iterations traditionally involved in polar decomposition, focusing instead on utilizing a Gram-side representation to manage smaller \( n \times n \) matrices. This efficient transformation is coupled with an emphasis on stability through techniques such as Jacobi scaling and early-ridge addition, which prevent complications from roundoff errors. The underlying framework provides opportunities for improved convergence rates in scenarios requiring large dimensional computations, thus making it a vital tool for the AI/ML community seeking robust and efficient optimization techniques.