Separating signal from noise in coding evaluations (openai.com)

OpenAI has retracted its recommendation of SWE-Bench Pro, a coding benchmark previously endorsed for evaluating AI models' software development capabilities. This decision comes after an investigation revealed significant design flaws and data contamination issues that compromise the benchmark's ability to accurately reflect model performance. Despite initial promise, with frontier models improving from a 23.3% to an 80.3% pass rate on its 731-task public split, SWE-Bench Pro has failed to provide a trustworthy evaluation framework. The audit identified that nearly 30% of SWE-Bench Pro's tasks might be broken, stemming from issues like overly strict test requirements, underspecified prompts, and misleading instructions. These problems not only misrepresent model capabilities but also complicate the benchmarking process, which is critical for safe deployment and research prioritization in the AI/ML community. OpenAI urges developers to exercise caution when interpreting results from this benchmark and emphasizes the need for rigorously curated evaluation measures that genuinely reflect model abilities and alignment. The findings underscore the importance of high-quality benchmarks and the potential for advanced models to aid in identifying evaluation flaws effectively.