Randomness in Agentic Evals (arxiv.org)

A recent study on agentic systems reveals significant variability in performance estimates based on single-run evaluations, challenging the reliability of the commonly reported pass@1 score. The research involved analyzing 60,000 agentic trajectories across three models and two evaluation scaffolds, uncovering that pass@1 metrics could differ by 2.2 to 6.0 percentage points depending on the run, with standard deviations exceeding 1.5 percentage points. This variance suggests that purported improvements in performance may stem more from evaluation noise than actual advancements in algorithmic capability. To enhance the reliability of evaluations, the authors propose three best practices: conducting multiple independent runs per task for more accurate pass@1 estimates, utilizing statistical power analysis to determine necessary run counts for detecting meaningful improvements, and incorporating alternative metrics like pass@k and pass^k for a more comprehensive understanding of system performance. While these recommendations may increase evaluation costs, they are crucial for distinguishing genuine scientific progress from mere statistical fluctuations, thus providing a more solid foundation for future research and development in the AI/ML community.