General purpose LLMs outperform specialized clinical AI on medical benchmarks (www.nature.com)

A recent comparative study revealed that general-purpose large language models (LLMs) outperform specialized clinical AI tools in medical evaluations. Researchers assessed two clinical AI systems, OpenEvidence and UpToDate Expert AI, against advanced models including GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6 across three benchmark stages: testing medical knowledge through MedQA questions, expert alignment with HealthBench items, and real clinical queries (RCQ) drawn from actual physician interactions. The results indicated that the frontier LLMs consistently surpassed the clinical tools in all metrics, with Gemini scoring the highest accuracy of 97.4% compared to OpenEvidence's 89.6% and UpToDate's 88.4%, demonstrating the potential of general-purpose models in clinical settings. This finding is significant for the AI/ML community as it raises questions about the effectiveness of proprietary clinical AI systems, which often lack independent verification. The study suggests that general-purpose LLMs may excel due to their extensive training data and superior alignment capabilities, outperforming domain-specific adaptations. The implications of this research could inform procurement and regulatory frameworks in healthcare, emphasizing the necessity for independent evaluations of AI technologies before deployment. Moreover, as generative models increasingly enter healthcare systems, the need for rigorous assessment remains paramount to ensure they are safe and effective in real-world applications.