Decode-Latency Feedback Prefill: A Model-Free Controller (arxiv.org)

Researchers have introduced a novel approach called Decode-Latency Feedback Prefill (DLFP), a model-free controller designed to optimize the efficiency of concurrent autoregressive inference in AI models. The primary challenge addressed by DLFP is the interference caused by prefilling long prompts, which can delay token generation for simultaneous requests. By dynamically adjusting the size of prefill chunks based on observed decoding intervals, DLFP aims to minimize latency while retaining output accuracy and meeting service level objectives (SLOs). Significantly, the implementation of DLFP in the vLLM framework demonstrated a 27.7% reduction in P99 inter-token latency during trials with the Qwen3-0.6B model on NVIDIA A100 GPUs, although it also resulted in a 34.8% increase in time to the first token. However, the method faced limitations in scaling to larger model sizes such as Qwen3-8B and Qwen3-32B, highlighting a key boundary for its applicability. The findings not only showcase a potential breakthrough in reducing latency for multi-request processing but also underline the need for further exploration of completion-timed controllers in the context of concurrent inference, marking an important step in advancing AI/ML performance optimization strategies.