Measuring What Matters: Construct Validity in Large Language Model Benchmarks (arxiv.org)

A large systematic review of 445 LLM benchmarks by a team of 29 expert reviewers highlights that many widely used evaluations lack construct validity — they don’t reliably measure the abstract phenomena they claim to (e.g., “safety” or “robustness”). By surveying benchmarks published at major NLP and ML venues, the authors identify recurring problems in how phenomena are operationalized: tasks chosen as proxies are often narrow or misaligned with the underlying concept, scoring metrics are coarse or brittle, and reporting/practices make it hard to interpret what a score actually means. The upshot is that benchmark results can mislead researchers, practitioners, and policymakers about model capabilities and risks. This matters because benchmarking drives model development, deployment decisions, and research priorities; weak construct validity can produce false confidence or misplaced effort. The paper provides eight concrete recommendations and actionable guidance for building better LLM benchmarks — for example, more careful definition of target constructs, tighter alignment between tasks and constructs, richer multi-metric and distributional reporting, clearer provenance and documentation, and robustness checks to validate that metrics correlate with the intended real-world phenomena. For anyone designing, using, or comparing LLMs, the work is a timely call to raise methodological standards so evaluations actually measure what matters.