The Comparator in Clinical AI (sparsethought.com)

A recent study by Brodeur et al. published in *Science* revealed that OpenAI's o1-preview model achieved a 78.3% accuracy rate in diagnosing cases from NEJM clinicopathologic conferences, outperforming human physicians. This research, which included six distinct experiments ranging from curated clinical cases to real emergency department scenarios, sought to compare the AI model against both GPT-4 and a range of physician benchmarks. The results indicate that AI can effectively assist in clinical settings, particularly during initial triage when information is least available. However, the significance of this finding is complicated by the evolving role of AI in healthcare. By 2026, a majority of physicians are reported to incorporate AI tools into their workflows, primarily for documentation and summarization, rather than differential diagnoses. Notably, the study highlighted a buried finding: physicians using AI tools like GPT-4 often performed worse than when working independently, suggesting potential pitfalls in human-AI collaboration that deserve further investigation. Overall, this study reflects a pivotal moment in clinical AI, emphasizing that the key questions have shifted from whether AI can assist physicians to how it impacts collaborative workflows in real-world healthcare settings.