DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference (arxiv.org)

Researchers have introduced DualPath, a novel inference system designed to alleviate the storage bandwidth bottleneck in agentic large language model (LLM) inference. Traditionally, inference performance has been hampered by the heavy reliance on KV-Cache storage input/output, leading to a saturated bandwidth at prefill engines while decoding engines remain underutilized. DualPath addresses this issue by implementing a dual-path KV-Cache loading architecture, which allows efficient data transfer from storage directly to decoding engines. This is achieved through Remote Direct Memory Access (RDMA) across a compute network and is coupled with a global scheduler to optimize load distribution between prefill and decode engines. The implications of DualPath are significant for the AI/ML community, demonstrating up to a 1.87 times increase in offline inference throughput and an impressive 1.96 times improvement in online serving throughput across various production workloads. This advancement not only enhances operational efficiency but also optimizes resource usage without compromising service level objectives (SLOs). The results underscore the potential for DualPath to serve as a foundational element in the future of scalable, high-performance LLM applications.