LongCat ZigZag Attention (www.k-a.in)

LongCat ZigZag Attention (LoZA) has been introduced as a groundbreaking sparse attention scheme designed to optimize full-attention models by allowing them to operate more efficiently within a limited computational budget. LoZA enables significant speed-ups in processing long-context scenarios, making it particularly valuable for tasks like retrieval-augmented generation and tool-integrated reasoning. By implementing LoZA into the LongCat-Flash model, researchers have developed LongCat-Flash-Exp, a foundation model capable of handling up to 1 million tokens, facilitating enhanced long-term reasoning and agentic capabilities in AI applications. The significance of LoZA is underpinned by its two-phase approach, utilizing calibration to identify which layers in a pre-trained model can afford to use sparse attention without losing performance, followed by a training phase to recover any lost effectiveness. Inspired by the lottery ticket hypothesis, which posits that dense networks contain effective sparse subnetworks, LoZA effectively transitions models from full to sparse attention. This method not only reduces computation by limiting attention to relevant tokens but also preserves critical information needed for tasks. Ultimately, LoZA's architecture, which opts for layer-level sparsity over head-level, simplifies computational complexity, paving the way for its integration into various open-source language models and potentially broadening its impact across the AI/ML community.