SoftServe: A Scalable Quasi-Newton Method for Deep Learning (arxiv.org)

SoftServe, a newly introduced family of Quasi-Newton (QN) methods, aims to enhance deep learning optimization by tackling issues of non-convexity and significant parameter sizes that have historically limited the effectiveness of these methods. Unlike traditional QN approaches, SoftServe eliminates the need for line searches and arbitrary curvature corrections. Its innovative design derives positive-definite curvature estimates from advanced variational objectives, enabling it to handle environments with negative curvature. The method's diagonal and Kronecker-factored variants are specifically built to maintain positive definiteness while efficiently scaling to large neural networks. This development is particularly significant for the AI/ML community as SoftServe demonstrates superior performance on challenging, ill-conditioned problems, including recurrent networks and deep autoencoders. It outperforms established optimization baselines like Adam and Muon, achieving lower loss rates in complex scenarios such as a 136M-parameter physics-informed diffusion model. By leveraging the stable coupled Newton-Schulz iteration, SoftServe replaces costly matrix operations with GPU-optimized multiplications, paving the way for more efficient learning in deep neural architectures.