Macro Evals for Agentic Systems (developers.openai.com)

A new framework called "macro-evals" for evaluating multi-agent systems has been introduced, addressing common failures in these systems that often stem from broader issues rather than single erroneous responses. This macro-eval workflow utilizes a synthetic electric vehicle order process to exhibit how specialist agents interact and manage tasks like pricing, compliance, and scheduling amidst changing operational conditions. The framework allows teams to analyze and discover recurring behavior patterns across traces, enabling focused inspections where issues are most concentrated. This initiative is significant for the AI/ML community as it promotes a structured method for evaluating complex agentic systems beyond traditional evaluations that focus solely on isolated outputs. By breaking down the evaluation process into lower-level insights and macro patterns, teams can better understand systemic issues and refine their workflows. The macro-evals not only enhance the quality of assessments but also improve communication between technical and business stakeholders by summarizing extensive data into actionable insights. Key technical components include the generation of trace documents, lower-level evaluations for individual agent performance, and the identification of high-impact patterns that can indicate where human oversight is necessary.