MCF: Move queries, not KV caches (arxiv.org)

A recent paper introduces the Memory- and Communication-Aware Framework for Long-Context LLM Inference (MCF), which significantly enhances the efficiency of long-context inference in large language models (LLMs). At its core is the Memory-Aware Replication and Scheduling (MARS) mechanism, which optimizes memory usage by intelligently managing key-value (KV) cache layouts during processing. MARS facilitates selective replication of frequently reused KV blocks, improving data locality and reducing latency during attention computation. As inference progresses, it smartly adjusts the KV cache layout to streamline decoding, minimizing the performance penalties typically associated with transitioning between prefill and decoding stages. The significance of MCF lies in its innovative query-centric execution model that shifts data transfer dependency from KV caches to lightweight query tensors. This novel approach notably lessens communication overhead as context length increases, creating potential for better scalability in distributed systems. By integrating a hardware-software co-designed execution layer utilizing FPGA technology, MCF enables efficient metadata-driven routing and collective operations directly within the network path. Experimental results demonstrate marked improvements in throughput and total time to first token (TTFT) across varying contexts and cluster sizes, suggesting MCF could set new benchmarks in LLM inference performance.