Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure (arxiv.org)

A recent study, titled "Clean Engineering, Unstable Measurement," raises significant concerns about the reliability of black-box language model (LLM) observers used to evaluate AI-generated content. The research involved two preregistered campaigns auditing the consistency of these models. With nearly 53,000 attempts, the results revealed considerable variability in rankings and evaluations, falling short of the required reliability thresholds—indicating that identical queries produced inconsistent rankings over time. This challenges the prevailing assumption that LLMs provide stable outputs in controlled settings. The implications of these findings are crucial for the AI/ML community, particularly for those relying on these models for governance and training data decisions. The study highlighted three primary reasons for the observed instability: biased label interpretations, severe gaps in candidate evaluations, and unpredictable rankings from identical inputs. Furthermore, metrics like self-hosting and provider switching failed to achieve desired consistency. The research advocates for a reassessment of reliability assumptions in AI evaluations, introducing a structured framework to improve measurement practices. This underscores the necessity for rigorous vetting of evaluation metrics before using them to shape AI models, ultimately aiming for enhanced trust and stability in AI assessments.