Forget Attention: Importance-Aware Attention Is All You Need (arxiv.org)

A groundbreaking approach in hybrid language modeling has been introduced with the proposal of SISA (SSM-Informed Softmax Attention), which effectively integrates attention mechanisms and state space models (SSMs). Unlike existing models that compartmentalize these functions—such as Jamba and Hymba—SISA advances the field by embedding an SSM-derived importance term directly into the attention score calculation. This unique method allows for a comprehensive operation performed as a single SDPA call on enhanced query/key vectors, promoting both efficiency and effectiveness without the need for recurrent states or custom kernels. The significance of SISA lies in its demonstrated performance improvements, achieving a LAMBADA-greedy accuracy of 17.3%, outperforming traditional Transformers (13.9%) and previous models like Mamba-3 (15.5%). Furthermore, SISA showcases remarkable speed with a retrieval convergence rate seven times faster than standard Transformers, while maintaining a perfect NIAH from the first 1,000 steps. This innovation represents a pivotal shift in SSM-attention hybrids, introducing a score-level fusion design that could redefine how AI models leverage attention in natural language processing.