Why averaging LLM benchmark scores is fundamentally broken (arxiv.org)

A new study highlights the shortcomings of traditional benchmarking methods in AI and ML, specifically how simple averaging of scores can lead to misleading assessments, particularly when dealing with sparse evaluation data and varying item difficulties. The research, utilizing simulations across domains like NLP, clinical trials, and autonomous vehicle safety, found that the correlation between these average rankings and actual ground-truth rankings significantly falls as coverage decreases and difficulty discrepancies increase. Notably, while Spearman rank correlation dropped to as low as 0.809 under testing constraints, a two-parameter logistic Item Response Theory (IRT) model maintained a high correlation above 0.996. This finding is crucial for the AI/ML community as it questions the reliability of conventional benchmarking practices, which could undermine the evaluation of AI systems in safety-critical applications. The study advocates for the adoption of IRT methodologies, suggesting that they more accurately reflect performance across diverse and incomplete data sets, thereby providing a more robust framework for evaluating AI capabilities. As benchmarking is foundational for AI development and deployment, improving these metrics can lead to better insights and advancements in the field.