The algorithm will see you now (worksinprogress.co)

Radiology is often held up as the poster child for medical AI: CheXNet (2017) and newer commercial systems can flag pneumonia and many other conditions faster than humans on benchmarks, operate on a single consumer GPU, and over 700 FDA-cleared radiology models now account for roughly three-quarters of medical AI devices. Vendors also build workflow tools that reorder worklists, draft structured reports, or, in rare FDA-cleared cases, operate autonomously. But the field’s real-world uptake and impact have been much more limited than early promises suggested. The reasons are technical, regulatory, and human. Most imaging AIs are single-task “islands of automation” trained on curated datasets (CheXNet used ~100,000 X‑rays) and often validated on narrow site collections (38% reported single‑hospital testing), so performance can drop by as much as ~20 percentage points out-of-sample. Training sets underrepresent children, women, and minorities and omit challenging, real-world images, producing brittle models and domain-shift failures. Legal, reimbursement, and workflow constraints further block full automation, and human factors—clinicians’ overreliance on flawed tool outputs—have historically degraded outcomes (e.g., mammography CAD). For AI/ML practitioners, radiology highlights critical needs: diverse, multi-site datasets, robust external validation, multi-task/ensemble orchestration, better human-AI interaction design, and regulatory-aligned evaluation to move beyond benchmark wins to reliable clinical benefit.