Attention Once Is All You Need: Stateful Transformers (arxiv.org)

A new paper titled "Attention Once Is All You Need" introduces a transformative approach in machine learning, focusing on efficient streaming inference with stateful transformers. Unlike traditional request-driven models, which incur high prefill costs with increasing context size, this innovative framework employs a persistent key-value (KV) cache that allows for incremental updates as data arrives. This shift significantly reduces query latency to O(|q|), unaffected by accumulated context. Moreover, the concept of Flash Queries enables pre-evaluation of registered questions during idle GPU cycles, creating a more responsive system that maintains state between requests, a feat not possible in stateless architectures. The implications for the AI/ML community are profound. The reference implementation has demonstrated up to 5.9 times faster performance on streaming market-data benchmarks compared to existing inference engines like vLLM and TensorRT-LLM, all while maintaining full quadratic self-attention. This development not only improves processing efficiency but also paves the way for more sophisticated applications in real-time data analysis and continuous streaming environments, thus challenging and potentially reshaping the design of future transformer-based systems.