Benchmarked how the latest LLMs perform on expert level medical questions (www.natomy.com)

Recent benchmarking of state-of-the-art AI models on the MedXpertQA dataset, which comprises expert-level medical questions, revealed significant insights into their performance in multimodal and text-only medical reasoning. The study examined 2,000 questions with clinical images and 2,450 text-only questions across a variety of organ systems and task types, employing chain-of-thought reasoning in a zero-shot context without prior fine-tuning. The findings highlight that models such as Gemini 3.8 Flash achieved remarkable accuracy, particularly in areas like the lymphatic and nervous systems, while others like Claude Sonnet 5 struggled more significantly. This research is crucial for the AI/ML community as it underscores the varying strengths and weaknesses of different models in handling complex medical queries. The report not only outlines the general performance trends but also delves into specific organ systems and task types, highlighting discrepancies in model accuracy. However, it also notes limitations within the MedXpertQA dataset, such as potential ambiguities in answer choices and occasional issues with image quality, suggesting that the true effectiveness of these models might be even better than indicated by the raw score. As AI continues to be integrated into healthcare, these insights will inform future improvements and applications in medical diagnostics and decision-making.