HealthBench Base and Prof: Evaluating models in realistic health conversations (artificialanalysis.ai)

OpenAI has introduced two new benchmarks, HealthBench and HealthBench Professional, aimed at evaluating AI models' performance in realistic health conversations. HealthBench assesses model responses across 5,000 conversations with users and health workers, focusing on themes such as emergency referrals and global health. The evaluation criteria include accuracy, completeness, communication quality, context awareness, and instruction following. HealthBench Professional narrows down the scope to 525 clinician interactions, featuring complex scenarios that challenge models on consultation, writing, and research tasks. Scores are reported in both raw and length-adjusted formats, with adjustments penalizing overly lengthy responses. This launch is significant for the AI/ML community as it sets a new standard for evaluating conversational AI in healthcare, highlighting the importance of clear and effective communication in medical contexts. Initial results reveal that OpenAI’s GPT-6 Astra leads the HealthBench Professional leaderboard with a length-adjusted score of 66.0%, while Claude Sonnet 5.5 tops the HealthBench leaderboard at 62.1%. The benchmarks emphasize the trade-off between completeness and communication quality, illustrating the complexities of generating accurate, concise, and contextually relevant responses in a nuanced field like healthcare. This advancement could drive further innovation in developing AI models tailored for medical applications.