Automating Benchmark Design (arxiv.org)

Researchers introduced BeTaL (Benchmark Tuning with an LLM-in-the-loop), a framework that automates the creation of dynamic benchmarks for evaluating large language models and agentic systems. Instead of relying on static, human-crafted tests that quickly become saturated, BeTaL parameterizes key design choices in base benchmark templates and uses LLMs to reason through that parameter space to produce tasks with target properties such as difficulty and realism. This LLM-in-the-loop approach applies environment-design principles to make dynamic benchmark generation faster and cheaper, reducing the need for continual manual curation as models evolve. Technically, BeTaL steers template parameters via LLM reasoning rather than brute-force search, enabling cost-efficient calibration to target difficulty levels. The authors validated the method by creating two new benchmarks and extending a popular agentic benchmark (τ-bench), then measuring how closely generated benchmarks matched desired difficulty. BeTaL achieved average deviations of 5.3%–13.2% from targets, a 2–4× improvement over baseline methods. For the AI/ML community, this offers a scalable path to maintain meaningful, evolving evaluations that better track model capabilities and help spot emergent strengths or failure modes without constant human redesign.