How we made evals 100× cheaper with multi-agent architecture (cleric.ai)

In a recent development, the team behind the AI agent Cleric has successfully transformed their architecture from a single-agent system to a multi-agent framework, leading to a staggering reduction in evaluation costs by 100 times. Initially, their evaluation process involved costly and time-consuming end-to-end investigations for each change to the agent, which became unsustainable as expenses soared over $12,000 monthly. By breaking down the system into approximately 40 specialized agents, each performing focused tasks like analyzing metrics or querying logs, they have optimized both cost and efficiency. Their new architecture allows individual agent evaluations to cost around 1 cent and complete in approximately 30 seconds, compared to the previous system's $1.50 and 30 minutes. This significant shift not only enhances testing speed and reduces costs but also boosts the overall iteration cycle, enabling the team to run over 5,000 evaluations weekly as opposed to the previous 180. The ability to rapidly test different models and approaches has led to further improvements in their key performance indicators while maintaining quality. However, the transition also introduces complexities such as increased system interactions, potential miscommunication between agents, and the challenge of ensuring cohesive functionality across the distributed architecture. Overall, this architectural evolution positions the AI/ML community to rethink evaluation strategies by emphasizing modularity and adaptability in agent-based systems.