Are we interpreting the model of interest? (ethahtz.github.io)

A recent analysis revealed that enabling the `output_attentions=True` option in a vision-language model significantly altered its performance, dropping accuracy from 70% to 20%. This stark decline arose because the attention mechanism changed from a more optimized version to an eager implementation, leading to numerical discrepancies that resulted in different outputs. The study compared three attention backends—Eager, Scaled Dot Product Attention (SDPA), and Flash Attention 2 (FA2)—and found that using different attention implementations produced notable variations in generated text, prompting concerns about the interpretability of models and the accuracy of studying model behaviors based solely on their computational artifacts. The investigation underscored that while attention mechanisms are theoretically equivalent, numerical precision and implementation choices can profoundly affect model outputs. Results showed that as floating-point precision increased, differences diminished significantly, particularly in FP32, where the same inputs produced identical outputs. This observation suggests that model interpretability is contingent on maintaining consistency in computational configurations, raising important questions for researchers regarding which models to analyze and how to ensure reliable comparisons across different training and inference conditions.