There Will Be a Scientific Theory of Deep Learning (arxiv.org)

A new paper proposes the emergence of a scientific theory of deep learning, termed "learning mechanics," aimed at characterizing key properties and statistics of neural network training processes. The authors synthesize recent advances in deep learning research into five main areas: idealized settings that simplify learning dynamics, tractable limits revealing fundamental insights, simple mathematical laws for understanding macroscopic behaviors, hyperparameter theories that clarify their impact on training, and universal behaviors across various systems. This perspective emphasizes the importance of describing the training dynamics with quantifiable, falsifiable predictions. The significance of this development lies in its potential to deepen our understanding of how neural networks function, transitioning from anecdotal practices to a principled framework. By addressing skepticism around the feasibility and relevance of a fundamental theory, the authors highlight open questions and future research directions essential for progressing in the field. This theoretical groundwork could significantly inform both mechanistic interpretability and practical implementations in AI/ML, making it an important cornerstone for future advancements.