Radiology's Last Exam (RadLE) (arxiv.org)

A recent study has introduced "RadLE," a benchmarking system aimed at evaluating the capabilities of advanced multimodal AI models in medical imaging, particularly against expert human radiologists. This evaluation involved 50 expert-level diagnostic cases and included assessments of five prominent AI systems—OpenAI's o3 and GPT-5, Gemini 2.5 Pro, Grok-4, and Claude Opus 4.1—through their web interfaces. While human radiologists achieved the highest diagnostic accuracy at 83%, AI models, with GPT-5 leading at 30%, demonstrated significant limitations, especially with challenging cases. Notably, the study classified visual reasoning errors made by AI into a proposed taxonomy to enhance understanding of their performance shortcomings. The significance of this research lies in its clear demonstration of the current limitations of generalist AI in healthcare settings, particularly in interpretations of complex medical images. As AI tools become more integrated into clinical practice, the findings urge caution against their unsupervised use, emphasizing the necessity for rigorous evaluation and understanding of AI reasoning errors. This framework not only highlights critical performance gaps but also directs future development efforts to create more reliable models capable of supporting medical professionals effectively.