AI Agent Benchmark Compendium (www.philschmid.de)

A new AI Agent Benchmark Compendium aggregates over 50 modern benchmarks for evaluating agentic LLMs and is organized into four practical categories: Function Calling & Tool Use, General Assistant & Reasoning, Coding & Software Engineering, and Computer Interaction (GUI & Web). The compendium (with a companion GitHub repo) centralizes diverse evaluations—from single-turn function calls and large-scale API mastery to multimodal web navigation and real-world coding fixes—and invites community contributions via PRs and issues. Technically, the collection highlights key evaluation axes now shaping agent research: fine-grained function-calling (BFCL, ComplexFuncBench, NFCL) including nested/parallel/multi-step calls and extreme context lengths (up to 128k); massive API and tool-learning datasets (ToolBench, API‑Bank) that support instruction tuning; real-world agent workflows and MCP-driven server interactions (LiveMCPBench, MCP‑Universe); robust, contamination-free and adversarial tests for factuality, safety, and reasoning (GAIA, LiveBench, FORTRESS, SimpleQA/Verified, FACTS); practical software-engineering benchmarks with human-validated or hidden test sets (SWE‑bench variants, LiveCodeBench, SWE‑PolyBench); and rich web/GUI navigation suites (WebArena, VisualWebArena, WebVoyager, BrowseComp). For researchers and practitioners this compendium makes it easier to pick targeted benchmarks, compare agent capabilities across realistic dimensions (memory, tool orchestration, grounding, security), and accelerate development by aligning training and evaluation to real-world agent tasks.