Noise-Driven Escape from Metastable Phases Explains Grokking in DNNs (arxiv.org)

Recent research has unveiled that deep neural networks (DNNs) undergo first-order phase transitions influenced by variations in L2 regularization strength, shedding light on the phenomenon of "grokking" in machine learning. Grokking describes the delayed onset of model generalization, often appearing after a period of overfitting. The study demonstrates that networks can become trapped in metastable states—collective configurations of weights that hinder convergence—until noise from stochastic gradient descent (SGD) allows them to escape across energy barriers. This escape process aligns with Arrhenius scaling, illustrating how noise can facilitate a transition from low-accuracy states to effective learning. This research holds significant implications for the AI/ML community, as it clarifies the underlying mechanisms of grokking and suggests a potential method for enhancing model convergence. By establishing that the number of metastable states correlates to the learnable features in a DNN, the findings indicate that task complexity naturally amplifies this hysteresis effect. The insights gained not only deepen our understanding of DNN behavior but also pave the way for designing more efficient learning strategies that leverage noise to improve generalization outcomes.