Frontier AI Is Accelerating. Open Benchmarks Need to Keep Up (benchmarks.snorkel.ai)

Snorkel has announced a substantial $30 million commitment to expand its Open Benchmarks Grants (OBG), aimed at advancing the evaluation of frontier AI systems. This initiative will support researchers and open-source teams in developing diverse benchmarks for crucial areas such as safety, alignment, cybersecurity, and physical AI. The program includes new funding for innovative measurement methods and the formation of an Open Benchmarks Red Team, which will collaborate with builders to identify and rectify weaknesses in benchmarking processes, ensuring that assessment tools remain relevant and effective amidst rapid advancements in AI technology. The significance of this initiative lies in its recognition of the need for robust and continually evolving benchmarks to keep pace with the accelerated development of AI models. As the complexity and capabilities of AI systems grow, the benchmarks used to evaluate them must also expand in scope and depth, covering diverse environments, autonomy levels, and output complexities. By addressing potential issues like reward-hacking and contamination, and by involving domain experts in the evaluation process, Snorkel aims to maintain the integrity and utility of these benchmarks, thus fostering a more empirical and scientifically rigorous approach to AI research and development.