A short note on interpretability and minds (ericjmichaud.com)

A new blog post by Eric J. Michaud explores the intricate relationship between interpretability in neural networks and broader cognitive sciences, building upon his PhD research. Over 18 months, he distilled significant ideas regarding mechanistic interpretability—arguing that neural networks’ computations can be understood in terms of simpler, modular components. He emphasizes that just as cognitive science has sought to describe human understanding, interpretability in AI can reveal insights about the workings of artificial systems, potentially leading to a deeper comprehension of human cognition itself. Michaud posits that the internal workings of deep learning models, like language models, can be modeled as a decomposition of relevant cognitive tasks or computations, adhering to power-law distributions. This suggests that not all components of a model are equally important; some are crucial across many tasks, while others are rarely activated. Understanding this could shed light on the nature of thought and language, hinting at a possible convergence of AI and cognitive science. The implications of these ideas could foster a new scientific legacy by bridging insights from both fields, ultimately enriching our comprehension of intelligence in all its forms.