AutomationBench-AA (artificialanalysis.ai)

AutomationBench-AA has been launched as an independent leaderboard for evaluating AI agents' capabilities in automating real SaaS workflows while adhering to business rules. In collaboration with Zapier, the benchmark tests models on 657 tasks across key business areas like Finance, HR, and Marketing, utilizing 40 simulated app environments, including popular platforms like Salesforce and Slack. The primary metric for scoring is the share of completed objectives without violating guardrails designed to ensure compliance with business rules. Leading models include Anthropic’s Claude Fable 5 and Opus 4.8, achieving scores of 48.6% and 48.5%, respectively. This initiative is significant for the AI/ML community as it sets a rigorous standard for evaluating automation capabilities, pushing models to not only solve tasks but do so responsibly. The results highlight a disparity in performance across tasks, with Finance workflows being the most challenging, where agents complete around one-third of their objectives compared to over half for Support and Operations. Moreover, the analysis reveals cost-effective options, as demonstrated by Gemini 3.5 Flash, which scores competitively at a fraction of the cost of other high-performing models. This transparent benchmark encourages further innovation in building robust autonomous agents capable of navigating complex SaaS workflows.