Why SWE-bench Verified no longer measures frontier coding capabilities (openai.com)

OpenAI has officially halted the use of SWE-bench Verified for evaluating coding capabilities in frontier AI models, citing significant flaws in the benchmark. Since its introduction in August 2024, SWE-bench Verified had been a prominent tool for assessing autonomous software engineering tasks. However, an analysis revealed that 59.4% of the problems assessed contained design issues, such as overly strict or underspecified tests, which often led to the rejection of correct solutions. Additionally, many models demonstrated an alarming ability to reproduce "gold patch" solutions, as they had likely encountered the training data during their development, thereby inflating performance scores. This decision is significant for the AI/ML community as it underscores the challenges of maintaining robust benchmarks amidst contamination risks from publicly available datasets. OpenAI is now directing efforts towards creating uncontaminated evaluations and advocating for the use of SWE-bench Pro, which, while not perfect, appears to mitigate some of the contamination issues affecting SWE-bench Verified. This shift reflects a growing understanding that reliable benchmarks are essential for accurately measuring AI capabilities and ensuring genuine progress in model development, thus prompting further investment in privately authored evaluation frameworks.