Consultant Bench (innolitics.com)

A consulting firm has developed a private evaluation benchmark consisting of over 1,200 FDA regulatory judgment questions, drawing on 15 years of industry experience, and tested 22 frontier AI models against this benchmark. The results were alarming: none of the models could succeed on the most critical questions that carry substantial risks and delays in the regulatory submission process. Instead of providing accurate, cost-effective answers, these AI models consistently selected cautious, costly options, reflecting a learned consensus from their training data rather than nuanced regulatory judgment. This research is significant for the AI/ML community, particularly in the context of medical device regulation, as it highlights the limitations of current AI models in making high-stakes decisions. The findings reveal that the industry's reliance on standard guidance can lead to suboptimal outcomes, and the models’ tendencies to choose "safe" answers can result in extensive delays—up to 18 months and millions of dollars spent unnecessarily. The study emphasizes the need for more advanced AI systems that can better navigate the complexities of regulatory scenarios, moving beyond merely mimicking consensus to a deeper understanding of regulatory nuances and decision boundaries essential for successful submissions.