SubQ: Sub-quadratic LLM built for 12M-token reasoning (subq.ai)

SubQ has unveiled a groundbreaking sub-quadratic large language model (LLM) capable of processing an impressive 12 million tokens in a single prompt. This innovative model is engineered to handle comprehensive repositories, lengthy histories, and persistent states without sacrificing quality. By utilizing a fully sub-quadratic sparse-attention architecture, SubQ optimizes compute resources by focusing only on the most relevant relationships between words, achieving almost a 1,000x reduction in attention computation compared to traditional methods. This efficiency opens new possibilities for long-context reasoning and significantly reduces the operational costs associated with large-scale AI tasks. The significance of SubQ lies in its potential to revolutionize how LLMs scale for complex applications. With benchmarks showing its strong performance, including an accuracy of 95% in long-context tasks and competitive software engineering capabilities, it positions itself as a formidable player among leading models like Gemini and Opus. Additionally, SubQ aims to enhance coding agents by integrating with platforms like Claude Code and Codex, thereby streamlining the processing of extensive codebases and improving the efficiency of token-heavy queries. As the first model to implement this architecture, SubQ represents a pivotal shift in AI development, moving beyond mere incremental improvements towards foundational changes for more scalable, multi-modal inference.