vLLM Agentic API (vllm-project.github.io)

The vLLM Agentic API has been launched, enabling the seamless continuation of conversations by maintaining item history and response states on the server, thus eliminating the need to rebuild transcripts with each API call. This new agentic server facilitates advanced inference by integrating model calls, built-in tool execution, and multi-turn conversation management, specifically designed to work with popular models like Codex and Claude. The API uses HTTP requests, server-sent events (SSE) for streamed responses, and WebSockets for interactive communication, making it versatile for developers. This API is significant for the AI/ML community as it streamlines the execution of complex workflows and enhances the interactivity of AI models. By allowing developers to connect to compatible open models served by vLLM, the Agentic API helps in maintaining clear execution boundaries while providing the flexibility to leverage tools for web searches and other tasks. The integration with SGLang and NVIDIA Dynamo highlights its robust architecture, supporting clear conversation states and efficient distributed inference for AI applications. With the open model capabilities, the community is encouraged to contribute ideas and improvements, fostering innovation in AI management.