The Dead Salmons of AI Interpretability (arxiv.org)

A new study highlights significant concerns in AI interpretability by drawing parallels to a famous neuroscience experiment that illustrated pitfalls in statistical inference. Researchers found that when a dead salmon was placed in an MRI scanner and shown social images, it produced results that misleadingly suggested neural activity linked to emotions. This analogy illustrates how AI interpretability methods such as feature attribution and causal analysis can yield seemingly valid insights from models that are randomly initialized, raising alarms about the reliability of these findings. The study advocates for a pragmatic statistical-causal reframing of AI interpretability, proposing that interpretations should be treated as parameters of a statistical model derived from computational traces. This approach urges the AI/ML community to rigorously test interpretations against various computational hypotheses and quantify uncertainty. Addressing critical theoretical issues around identifiability, the authors highlight the need for AI interpretability to evolve into a more rigorous science, ultimately aiming to reduce false discoveries and enhance generalizability in the field. This shift could open new research pathways and improve the reliability of AI systems in real-world applications.