Show HN: The Hessian of tall-skinny networks is easy to invert (github.com)

A new package has been released that simplifies the computation of the Hessian inverse product for deep networks, a critical operation in various machine learning optimization tasks. Unlike traditional approaches that require cubic scaling with respect to the number of parameters—making them unfeasible for many modern neural networks—this implementation employs an innovative algorithm using hierarchical, structured block matrices to enhance computational efficiency. This breakthrough could significantly reduce the time and resources needed to perform this calculation in large-scale models. The advancement is particularly significant for the AI/ML community as it enables more efficient training and optimization of deep learning models, which often involve complex parameter landscapes. By leveraging partitioned matrix techniques and providing a code library for manipulating such structures, this new approach not only streamlines the computation but also allows researchers and practitioners to explore more sophisticated network architectures without the prohibitive computational overhead traditionally associated with Hessian inversions. This development could lead to enhancements in various areas, including model tuning and theoretical studies of neural network behavior.