Explaining Attention with Program Synthesis (arxiv.org)

Researchers have introduced an innovative method for interpreting the attention mechanisms of transformer models by using program synthesis. By generating executable Python programs that reproduce the attention patterns of specific transformer attention heads—such as those found in models like GPT-2 and Llama-3B—they aim to make the oft-opaque operations of deep learning frameworks more transparent and understandable to humans. The approach involves calculating attention matrices from training examples and utilizing a pre-trained language model to create these programs based on the summarized attention data. Remarkably, fewer than 1,000 generated programs were able to achieve over 75% similarity in attention patterns while allowing for the replacement of neural attention heads with minimal impact on model performance. This development marks a significant step toward enhanced interpretability in AI, as it provides a scalable pipeline for reverse-engineering complex model components into human-readable code. The ability to replace up to 25% of attention heads with these programmatic surrogates, only causing a 16% increase in perplexity, demonstrates the potential for symbolic reasoning in neural networks. Such advancements can pave the way for more transparent AI systems, facilitating trust and enabling researchers and practitioners to better understand and fine-tune models in various applications.