Π-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling (arxiv.org)

A new transformer model titled \Pi-Attention has been introduced, addressing the quadratic complexity issues associated with long-range sequence modeling in natural language processing. This innovative model combines periodic sparse attention to enhance the receptive fields of traditional sparse attention mechanisms like RingAttention, which typically struggle with limited adaptability. By employing a structure that integrates ring-local neighborhoods, deterministic $\pi$-stride skips, and an adaptive fusion gate, \Pi-Attention achieves a significant improvement in efficiency. The model showcases a receptive field growth of $\mathcal{O}(kL + \pi \log L)$, compared to RingAttention's $\mathcal{O}(kL)$, making it more effective for processing lengthy contexts. The significance of \Pi-Attention lies not only in its reduced computational demands—utilizing 50% fewer GPUs while attaining 8.3% lower perplexity in language modeling and other tasks—but also in its comprehensive approach to improving transformer performance. Extensive experiments suggest that this model not only rivals but can also outperform dense attention mechanisms. The combination of periodic skips, adaptive fusion, and head-level sparsity coordination is highlighted as crucial for effective long-context modeling, positioning \Pi-Attention as a potential game-changer in the field of AI/ML.