Criss-Cross Attention (www.jkobject.com)

A new attention mechanism called Criss-Cross Attention has been developed to address the computational bottlenecks in traditional self-attention models, which exhibit O(n²) complexity. This innovation is particularly significant for models dealing with extensive sequences, such as single-cell biology models like scPRINT-2, which processes up to 3,200 genes, leading to around 10 million pairwise interactions. By refocusing the attention mechanism, Criss-Cross Attention enables the model to learn a compressed representation of input data dynamically, improving efficiency while maintaining the model's expressive power. The Criss-Cross Attention approach operates with a doubly cross-attention mechanism between input tokens and a smaller set of learnable latent tokens, effectively reducing the computation cost from O(n²) to O(n × M), where M is significantly smaller than n. This shift not only allows the model to adaptively determine which aspects of the data to retain but also enhances the processing speed without sacrificing accuracy. The technique exemplifies a broader trend in AI/ML toward using latent representations for efficient context management, making it relevant for various applications beyond single-cell biology, including reasoning, multimodal fusion, and long-context understanding.