The Existence and Behavior of Secondary Attention Sinks (arxiv.org)

Recent research has uncovered a new class of attention sinks termed "secondary sinks," which differ from the previously recognized primary sinks in their behavior and characteristics. While primary sinks, often associated with the beginning-of-sequence (BOS) token, are consistent throughout layers of a model and draw significant attention, secondary sinks are identified in middle layers of neural networks. These secondary sinks exhibit a variable presence across layers and attract a smaller, yet notable amount of attention mass. The study conducted extensive experiments across 11 model families, revealing that these sinks emerge primarily due to specific middle-layer multi-layer perceptron (MLP) modules, which map token representations to vectors aligned with the primary sink direction of that layer. The implications of this research are significant for the AI/ML community, as it provides deeper insights into the attention mechanisms employed by large-scale models. Understanding the formation and properties of secondary sinks can lead to improved model interpretability and refinement of attention-focused architectures, especially in larger models where their location and lifespan appear to be more deterministic. By identifying three sink levels in QwQ-32B and six in Qwen3-14B, the study lays groundwork for future explorations into optimizing attention weights and enhancing overall model performance in tasks involving complex data representations.