Read the History, Not Just the Code (blog.ptidej.net)

Naomi Saphra, a researcher at Harvard's Kempner Institute, is advocating for a shift in how the AI/ML community approaches interpretability. Rather than solely analyzing the finished models, she emphasizes the importance of understanding the training process itself. Drawing an analogy from evolutionary biology, Saphra argues that like organisms, AI models can only be fully understood in light of their development—particularly through stochastic gradient descent. Her insights reveal that what appears essential in a model may actually be "vestigial," meaning it was once critical but no longer serves a purpose, akin to "dead code" in software development. Saphra’s work highlights the challenges of technical debt and the impact of early training decisions on a model's capabilities. She points out that models often latch onto easy solutions, which restricts their overall learning ability, mirroring problems in software engineering where early decisions shape the system. Additionally, she notes the lack of access to intermediate training checkpoints, complicating debugging efforts. This perspective encourages researchers to value the training history, akin to the commit history in coding, to gain deeper insights into model behavior, making her approach a significant contribution to the ongoing discourse in AI interpretability.