Evaluating AI Agents: A Production Blueprint with Strands and AgentCore (aws.amazon.com)

Motorway, a UK-based online car marketplace, has collaborated with AWS's Prototyping and AI Customer Engineering (PACE) team to enhance its dealer stock search agent, significantly improving the way dealers find vehicles. The AI agent now employs natural language queries, allowing dealers to replace tedious manual filtering. However, ensuring the agent's reliability is crucial since missteps could lead to financial losses. In response, Motorway and AWS developed an advanced evaluation pipeline, utilizing the Strands Agents SDK alongside Amazon Bedrock's AgentCore. This system has refined search accuracy from one incorrect result in every eight queries to one in fifty and reduced issue detection time from hours to minutes. For the AI/ML community, this project represents a significant advancement in agent evaluation, particularly in tackling challenges unique to AI agents as opposed to traditional large language models (LLMs). The new evaluation framework includes a two-phase strategy that monitors tool usage, reasoning, and output quality through a structured pipeline with stringent quality gates. By allowing the agent to handle complex queries effectively while maintaining high levels of trust among users, the project sets a benchmark for future AI agent applications, promoting rigorous testing while also providing a reliable blueprint that can be adapted for various domains across different platforms.