One Schema for Every Eval (evalevalai.com)

The EvalEval Coalition has announced a groundbreaking universal schema for AI evaluation that addresses the fragmentation in how results are shared and compared across different frameworks. This innovative schema allows for the seamless interchange of evaluation results from various sources, including HELM and EleutherAI, without the need for complex mappings. By capturing comprehensive experimental context—such as prompt templates and inference parameters—this initiative ensures that evaluation results are traceable, transparent, and reproducible. Significantly, the schema also liberates evaluation data from static PDFs and closed systems, transforming them into a structured, queryable global dataset. This design enables advanced meta-analysis and automated leaderboard creation. Each evaluation is linked with a unique UUID to facilitate data organization and prevent conflicts, while also allowing changes to models over time to be tracked accurately, thereby addressing concerns about API drift and model versioning. With its structured JSON format, the schema supports efficient data analysis using popular tools like Pandas or SQL, paving the way for more rigorous and reproducible AI research. The coalition encourages data contributions from eval providers to create a comprehensive public dataset that enhances the quality of AI evaluations across the community.