Pretraining in Stanford CS312: Sharpness and the Edge of Stability (twitter.com)

In a recent lecture from Stanford CS 312, researchers delved into the concept of "sharpness" in the context of pretraining language models. Sharpness is defined as the highest curvature of the loss function, measured through the largest eigenvalue of the Hessian matrix. It was demonstrated that during gradient descent, models can exceed a predicted threshold of sharpness without experiencing instability, a phenomenon termed the "edge of stability." This discovery is significant because it challenges existing theories that suggest models should diverge when sharpness surpasses a critical value, indicating a complex relationship between model training dynamics and loss landscapes. Through experiments with a 35M-parameter transformer model, researchers observed that sharpness could reach values significantly above the predicted instability threshold while still enabling effective training. The findings suggest that as models improve, they may oscillate within regions of higher sharpness, creating a self-stabilizing feedback loop in the loss landscape. This ongoing exploration into sharpness dynamics has broader implications for optimizing deep learning techniques, as understanding these mechanisms could lead to the development of more effective training methods and improved model performance across various architectures. The results further highlight a gap in understanding the fundamental behavior of models during training processes, encouraging further investigation in this area.