GPT-5o-mini hallucinates medical residency applicant grades (www.thalamusgme.com)

Thalamus confirmed that its Cortex “Transcript Normalization” feature — which uses OCR and NLP to parse clerkship grades and generate percentiles and visualizations — produced a small number of automation-extracted grade inaccuracies during the current residency recruitment season. The company says affected extractions were corrected immediately, and in every investigated case program directors identified the accurate grade by checking the original PDF transcript and MSPE. Thalamus emphasizes that all original application documents remain unaltered and fully accessible, the extraction outputs are intended for context only (not for automated filtering, sorting, or rejection), and reviewers are strongly encouraged to verify extracted values against official records. The tool, first released in 2020, is updated yearly to reflect transcript formatting and grading-schema changes. Why it matters: this incident highlights a recurring risk as LLM- and NLP-driven tools are adopted in high-stakes workflows — automated “hallucinations” or mis-extractions can create confusion or potential disadvantage unless coupled with clear guardrails and human review. Thalamus recommends programs use the normalization feature as a reference, employ the built-in “blinder” if desired, and collaborate with medical schools to refine mappings for unusual grading systems. The company commits to ongoing validation and invites inquiries to support rapid investigation and correction.