Attention Is Not What You Need: Grassmann Flows as an Attention-Free Alternative (arxiv.org)

Researchers are challenging the necessity of explicit self-attention mechanisms in neural networks with the introduction of a new architecture called Grassmann flows. This approach moves away from traditional multi-head attention, which involves complex tensor manipulation and often obscures model interpretability. Instead, the proposed Causal Grassmann layer transforms token states into two-dimensional subspaces on a Grassmann manifold, utilizing Plucker coordinates to fuse geometric features back into the model. This innovative computation strategy allows for efficient information propagation through controlled deformations of low-rank subspaces. The significance of this development lies in its potential to provide a more structured and interpretable framework for sequence modeling in AI. On benchmarks like Wikitext-2 and SNLI, Grassmann-based models have shown competitive performance against size-matched Transformers, demonstrating only a slight trade-off in validation perplexities and accuracy. Moreover, the linear scaling of Grassmann mixing with sequence length offers practical advantages, suggesting that this method could pave the way for advanced geometric interpretations of neural reasoning, ultimately enhancing AI systems' understanding and robustness in complex tasks.