Claude, GPT, Gemini Agents Fail 72% of U.S. Healthcare Workflows (apnews.com)

ActAVA.ai has unveiled CHI-Bench, the first comprehensive benchmark for evaluating AI agents in U.S. healthcare workflows. Testing 30 advanced agents, including those from Anthropic, OpenAI, and Google, the benchmark revealed that these systems failed to successfully complete approximately 72% of healthcare tasks related to prior authorization, utilization review, and care management. With trials demanding 60-80 consecutive steps through complex clinical processes, the top-performing agent, Anthropic's Claude Code, managed a meager 28% accuracy, highlighting a significant gap in reliability. The significance of CHI-Bench lies in its rigorous methodology and architecture, developed with input from over 20 academic and healthcare institutions. By exposing AI agents to intricate workflows, the benchmark addresses the critical issues of accountability and effectiveness in healthcare automation—one error in these workflows can lead to denied authorizations or treatment delays. This research sets a new standard for the industry, emphasizing the need for AI systems capable of consistent, end-to-end execution in high-stakes environments, and calls for further advancements in AI reliability in healthcare applications.