Beyond Prediction: Tail-Aware Scheduling for LLM Inference (yl3469.github.io)

A groundbreaking scheduling policy called UniBoost has been introduced to tackle the challenges of extreme output-length variability in large language model (LLM) serving. Traditional scheduling strategies, like Shortest Job First (SJF) and Size-Needed Shortest Job (SRPT), largely rely on predictable job sizes, which is impractical in the face of LLM output unpredictability. UniBoost innovates by employing soft, continuous priority boosting and KV-cache-aware preemption, allowing a single parameter to control the balance between first-come-first-served (FCFS) and SJF approaches. This adaptive methodology offers significant improvements in tail latency and response time, outperforming existing policies under diverse workload conditions. The significance of UniBoost lies in its ability to maintain performance amidst workload variations, crucial for real-time applications relying on LLMs. By dynamically adjusting how requests are prioritized based on actual progress rather than predicted lengths, UniBoost reduces the risk of tail latency penalties and ensures that shorter jobs do not face undue delays from longer ones. In comprehensive evaluations across various models, UniBoost demonstrated a remarkable capacity for improving the 95th and 99th percentile latency metrics, highlighting its potential for enhanced responsiveness in AI-driven applications while efficiently managing GPU resources.