Disaggregation in Large Language Models: The Next Evolution in AI Infra (www.infoq.com)

Large language model (LLM) inference is fundamentally two-phase: a compute-heavy "prefill" (200–400 ops/byte, ~90–95% GPU utilization) and a memory-bound "decode" (60–80 ops/byte, ~20–40% utilization). Recent work and production frameworks have converged on disaggregated serving—splitting prefill and decode onto specialized clusters—to match hardware to workload. Open-source and research systems (vLLM, SGLang, TensorRT-LLM, DistServe) report substantial wins: vLLM’s PagedAttention and continuous batching showed ~2.7× throughput on Llama-8B, SGLang reported up to 6.4× throughput on Llama-70B, and DistServe demonstrated ~4.5× goodput plus up to 20× reduction in latency variance. Real deployments also cite KV-cache transfer latencies in the single-digit milliseconds and measurable infrastructure gains (15–40% lower total costs, 40–60% better GPU utilization, and big energy savings). Technically, disaggregation maps prefill to compute-optimized GPUs (e.g., H100 for high FLOPs and batching) and decode to memory-bandwidth-optimized devices (e.g., A100 or accelerators with better cache behavior), connected by low-latency fabrics (InfiniBand/NVLink). Key enablers are efficient KV-cache management (Paged/RadixAttention), continuous batching, accurate workload profiling, distributed state/caching (Redis), and GPU-aware schedulers for dynamic routing. For practitioners, the recommended path is profile→segment resources→pilot with parallel deployments→gradual migration, while monitoring GPU utilization, cache hit rates, and token latency to validate cost and SLO benefits.