A Platform to Build and Share AI Evaluations (weval.org)

A new platform called Weval has been launched to enable the creation and sharing of in-depth qualitative evaluations for AI performance across diverse domains. Unlike traditional benchmarks that often focus on easily quantifiable metrics, Weval emphasizes evaluations that address critical aspects such as safety, honesty, and helpfulness. The platform features a public library of community-contributed benchmarks, allowing users to track model performance over time and propose their insights for public evaluation. This initiative is particularly significant for the AI/ML community as it provides a structured approach to measure complex and relevant traits in AI systems that haven't been adequately assessed by existing benchmarks. Weval's detailed evaluation methodologies cover areas from healthcare and legal reasoning to educational efficacy, grounding assessments in well-researched pedagogical practices. As a result, domain experts can contribute their knowledge to improve AI systems, ensuring these tools not only achieve operational efficiency but also align with ethical and societal norms. The platform aims to advance the understanding of AI behavior through comprehensive assessments, ultimately fostering the development of more responsible and capable AI systems.