Overparameterization's Puzzling Success: Lottery Tickets or Escape Dimensions? (infoscience.epfl.ch)

Recent research challenges the common analogy that overparameterized neural networks succeed because they contain well-initialized subnetworks, akin to increasing lottery tickets improving winning chances. This perspective implies that subnetworks can operate independently, leading to a flawed interpretation of learning in wide networks as a simple multi-start optimization process. The new analysis suggests that overparameterization's effectiveness stems from how it alters the geometry of the loss landscape. Specifically, wider networks create more available dimensions for optimization, making it easier to escape local minima, with bad minima becoming increasingly rare as network width increases. This insight is significant for the AI/ML community as it refines foundational understandings of neural network performance and reinforces the importance of aligning intuitive explanations with current theoretical advancements. By shifting the focus from subnetwork independence to the overall structure and dimensionality of the optimization landscape, researchers and practitioners can better comprehend how large networks function and improve strategies for training and deploying neural models.