Goodhart's Law Comes for Every Benchmark You Trust (cacm.acm.org)

The AI community is facing a significant challenge as Goodhart's Law reveals flaws in benchmark integrity, notably highlighted by issues with the BIG-bench dataset. OpenAI's GPT-4 accidentally incorporated this benchmark into its training data, illustrating how powerful web-scale crawlers can inadvertently "contaminate" benchmarks, rendering attempts at integrity checks ineffective. This dilemma echoes historical challenges in education and computing, where optimizing for specific metrics ultimately degraded the quality of learning and results. The systematic exploitation of benchmarks is further exacerbated by errors in popular datasets, such as MMLU, where nearly 6.5% of questions contained mistakes, thereby skewing model evaluations. To address these issues, experts advocate for the development of private, continuously refreshed test sets, akin to GSM1k, where questions remain unexposed online to mitigate data contamination. Additionally, contamination detection measures are deemed essential, although studies indicate that traditional methods have not proven effective. The overarching message is clear: as AI benchmarks become central to funding, rankings, and procurement decisions, it is vital for stakeholders to critically evaluate their reliance on public scores, treating them with skepticism, and to favor testing on proprietary data that accurately reflects real-world applications.