Patterns for Building a Scalable Multi-Agent System (devblogs.microsoft.com)

The post outlines a practical architecture and patterns for moving LLM-driven multi‑agent systems from prototype to production, motivated by an ecommerce voice assistant that must route diverse customer queries (order tracking, returns, recommendations, promotions) to dozens or hundreds of specialized agents without blowing up latency or token costs. The core idea is dynamic agent selection backed by a semantic cache: embed agent names + sample utterances (recommend ≥5 per agent) and index them (example: Azure AI Search with OpenAI’s text-embedding-3-small) so incoming queries retrieve a small, relevant agent set via similarity scores. Once candidates are found, an AgentFactory (Factory pattern) instantiates code- or template-defined agents (Python/SDK vs YAML) and a SupervisorAgent orchestrates multi‑intent group chats—applying selection and termination strategies to sequence hand‑offs and produce a single coherent response. Key technical implications and optimizations: use a semantic-retrieval fast path to invoke a single confident agent directly, limit group-chat “chattiness” via max_iterations, and tune LLM params (temperature/top_p → 0 for deterministic responses; cap max_completion_tokens). Onboarding and SOPs plus golden datasets and metrics (recall@k, precision@k, BLEU, relevance) are critical for evaluation as the agent ecosystem grows. Together these patterns deliver scalable, low‑latency, cost‑controlled orchestration that generalizes beyond voice assistants to any intent‑driven agent composition.