LLM-d 1.0, an open source stack for distributed inference (cloud.google.com)

Google has launched llm-d 1.0, an open-source, Kubernetes-native stack designed for distributed inference of large language models (LLMs) on Google Cloud TPUs and GPUs. This release signifies a pivotal moment for the AI/ML community, as it addresses the evolving nature of inference workloads, which have shifted from simple, prompt-response interactions to complex multi-step agent processes and reinforcement learning scenarios. Ll-d 1.0 enhances the efficiency and scalability of serving these workloads by employing cache-aware routing, tiered key-value caching, and priority flow control, ensuring that interactive requests remain responsive even during traffic bursts. The technical foundation of llm-d 1.0 includes session-aware routing, disaggregation for safer updates, and asynchronous processing that maximizes resource utilization while minimizing latency. This framework allows organizations to manage mixed workloads under a single API, facilitating an adaptable and efficient inference environment. Notably, benchmarks demonstrate that llm-d improves throughput by up to 2.9x and significantly reduces time-to-first-token for interactions, positioning it as a powerful tool for companies operating at scale. Major players like Snap and ZenteiQ have already reported enhancements in their AI infrastructure, leveraging llm-d's capabilities to refine performance and optimize resource allocation in production settings.