Kinference: Open-source inference control plane for vLLM (vllm.ai)

Kinference, an exciting new open-source project developed by Buoyant, has been launched to facilitate the construction of scalable inference platforms using vLLM. Designed as a control plane for managing inference workloads, Kinference addresses the challenges faced by users in deploying open-weight models, offering key features such as token rate limiting, intelligent load balancing, cost tracking, and policy-driven inference management. By simply directing inference clients to its HTTP endpoint, Kinference integrates seamlessly into existing technology stacks. The significance of Kinference lies in its advanced load-balancing capabilities that enhance inference performance. By leveraging KV-cache awareness, it significantly reduces median time-to-first-token (TTFT) by 95% compared to standard Kubernetes methods, optimizing GPU usage and minimizing latency. Kinference’s policy engine allows platform owners to orchestrate inference requests based on diverse factors, including request complexity and GPU resources, ultimately improving model efficiency and user experience. With comprehensive metrics reporting, Kinference enables precise performance monitoring, making it a vital tool for AI/ML developers looking to streamline their inference processes.