How noisy are LLM evals in CI? (evalship.com)

A recent study has unveiled the prevalence of noise in large language model (LLM) evaluations during continuous integration (CI) processes across public repositories. By analyzing eval results from various repositories, researchers found that approximately 2.5% to 3% of passing evaluations failed in subsequent runs, often without actual code changes. This raises significant concerns regarding the reliability of eval results, which can lead to undeserved regression alerts on pull requests that introduce no real issues. The study emphasizes the importance of measuring and understanding this noise, as failing to do so could mislead developers and hinder progress. The implications for the AI/ML community are profound, as they highlight the necessity of accurately interpreting evaluation outcomes in CI systems. Researchers suggest athletes should rerun evaluations on unchanged branches to measure inherent noise and compare results against historical performance. Tools like evalship are being developed to automate this process, providing insights into a suite's noise and helping developers make better-informed decisions on pull requests. This research not only advocates for improved evaluation methodologies but also sheds light on the complexities involved in validating LLM performances in real-world applications.