Cache-aware prefill–decode disaggregation – 40% faster long-context LLM serving (www.together.ai)

Together AI has introduced a new architecture called cache-aware prefill–decode disaggregation (CPD), which significantly optimizes long-context large language model (LLM) serving by enhancing throughput and reducing latency. CPD intelligently separates warm (reusable) and cold (new) requests, allowing for faster context reuse and up to 40% higher sustainable throughput under real-world traffic conditions. This innovation comes as long prompts, often exceeding 100K tokens, have become prevalent in applications such as coding assistants and conversational agents, challenging conventional serving architectures that struggle with increased time-to-first-token (TTFT) in mixed workloads. The CPD system employs a three-tier structure: dedicated pre-prefill nodes for cold requests, prefill nodes for warm requests, and decode nodes that focus on low-latency operations. By utilizing a hierarchical KV-cache system, which includes GPU memory and distributed caches, CPD facilitates quick access to previously computed contexts and streamlines processing. This architecture not only improves performance metrics, such as QTPS and TTFT, but also allows systems to gracefully scale under heavy load without being bogged down by lengthy cold prompts. Overall, CPD marks a significant leap forward in efficiently serving high-demand long-context LLM applications.