Push-based KV transfer to improve TTFT, TPOT for disaggregated inference in vLLM (aws.amazon.com)

The recent announcement regarding vLLM, an open-source inference engine for large language models (LLMs), introduces a push-based key-value (KV) transfer method that significantly enhances time-to-first-token (TTFT) and time-per-output-token (TPOT) during disaggregated inference. This improvement arises from separating the prefill and decode phases of LLM inference, allowing for independent optimization. The push mode streamlines KV transfer by eliminating the sequential coordination steps required in the pull model, enabling immediate action once KV blocks are ready, thus reducing latency and improving resource utilization, particularly under high request loads. In benchmark tests using the Qwen3-32B model, the push-based architecture demonstrated up to a threefold reduction in TTFT and a 30% decrease in TPOT compared to the traditional pull model on Amazon EC2 instances equipped with Elastic Fabric Adapter (EFA). The design operates through a dedicated background thread that manages KV transfer while allowing the main execution of the inference model to continue seamlessly. This advancement not only maintains high application responsiveness but also marks a significant step towards optimizing LLM deployment efficiency. The feature, now integrated into vLLM, is designed to be accelerator-agnostic, making it applicable across various hardware configurations.