Are your AI evals measuring the right things? (commandline.microsoft.com)

Microsoft has introduced the Adaptive Spec-driven Scoring for Evaluation and Regression Testing (ASSERT), a new framework aimed at improving the evaluation of generative AI applications. Traditional tests often yield misleadingly high pass rates if they focus on limited behaviors, potentially overlooking critical gaps. ASSERT addresses this issue by systematically structuring behavioral requirements into a taxonomy, ensuring that tests assess a diverse range of relevant scenarios. This method helps developers identify weaknesses effectively and supports transparency in how failures relate to the behaviors being tested. Significantly, ASSERT has been evaluated against Meridian Labs’ Petri Bloom framework, which generates evaluations from natural-language descriptions. In tests across various cybersecurity risks, ASSERT demonstrated superior technique coverage and distribution balance, marking a shift toward more rigorous evaluation standards. The evaluation showed that ASSERT produced a higher violation rate in initial conversation turns, suggesting it is better at surfacing policy violations early in interactions. This enhanced ability to capture nuanced behaviors and policy violations has crucial implications for ensuring the reliability and safety of AI applications, particularly in sensitive areas such as cybersecurity.