Burstiness is all you need for LLM serving (systems.seas.harvard.edu)

A recent exploration into the effects of bursty workloads on large language model (LLM) serving has revealed a surprising outcome: higher burstiness can actually decrease latency. Traditionally, bursty request patterns are considered detrimental due to their potential to overload servers and create performance bottlenecks. However, findings indicate that as the coefficients of variation of interarrival times increase, both the median time per output token (TPOT) and end-to-end latency (E2EL) improve. This counterintuitive result has significant implications for optimizing request routing and load balancing in production inference clusters. By observing higher rates of exclusive decode iterations in bursty conditions, researchers found that less interference between prefill and decode processes leads to faster performance. This research suggests that traditional methods of routing LLM requests must be re-examined; specifically, it indicates that sticky round robin and join the shortest queue (JSQ) methods significantly outperform conventional routing strategies in these scenarios. While burstiness presents challenges in typical circumstances, under the right conditions, it can be harnessed to enhance overall inference speed, fundamentally changing how AI systems manage workloads.