Infrastructure configuration can swing coding evals by several % points (www.anthropic.com)

Recent findings regarding agentic coding benchmarks like SWE-bench and Terminal-Bench reveal that infrastructure configuration can significantly influence evaluation scores—by as much as 6 percentage points. This variability underscores the importance of considering resource allocation when comparing the software engineering capabilities of AI models. Unlike static benchmarks, which purely assess output, agentic evals operate within a dynamic environment where runtime configurations directly impact problem-solving success. Internal experiments demonstrated that stricter resource limits led to higher infrastructure error rates, which subsequently affected model performance outcomes. The implications for the AI/ML community are profound: as reliance on benchmark scores for model deployment decisions increases, it is crucial to standardize how these evaluations are run and reported. Currently, slight score differences on leaderboards might misrepresent genuine model capabilities due to discrepancies in infrastructure setup or timing. The study recommends that evaluations should specify both guaranteed allocations and hard limits for resources, allowing for more consistent and interpretable results. By addressing these confounding factors, the community can improve benchmarking practices, ensuring that AI models are assessed more accurately without ambiguity stemming from their operational environments.