You don't need all the LLM benchmarks (alex.smola.org)

A recent study reveals that not all benchmark evaluations for large language models (LLMs) are necessary, as many are highly correlated. Analyzing over 5,400 models across various benchmarks (like MMLU and MTEB), researchers found that just a handful of subjects could predict overall performance with remarkable accuracy (R² ≈ 0.91 for MMLU). This insight suggests that selecting a smaller, well-chosen subset of benchmarks could significantly reduce computational costs and time while maintaining reliable performance assessments. To determine which benchmarks to prioritize, the study employed a Gaussian model to estimate covariance among scores. By treating benchmark selection as a sensor placement problem, they introduced objectives based on entropy and mutual information to guide the selection process. Their findings indicate that intelligently curating benchmark sets can explain over half of the variance in held-out scores, which is especially impactful for the more complex MTEB and merged benchmark scenarios. This approach not only streamlines evaluation efforts but also enhances the robustness of future LLM model assessments, emphasizing the importance of strategic benchmark selection in advancing AI/ML research.