AI medical tools found to downplay symptoms of women, ethnic minorities (arstechnica.com)

Multiple recent studies from leading US and UK institutions warn that large language models being deployed in clinical settings routinely downplay symptoms and provide less empathetic guidance for women and ethnic minorities. Research from MIT’s Jameel Clinic found models including OpenAI’s GPT‑4, Meta’s Llama 3 and the healthcare-focused Palmyra‑Med recommended lower levels of care for female patients and suggested self‑treatment more often; a separate MIT analysis showed reduced compassion in responses to Black and Asian users with mental‑health concerns. The London School of Economics reported similar gendered minimization in Google’s Gemma when generating social‑work case notes. These findings arrive as vendors and hospitals rapidly adopt LLM tools (Gemini, ChatGPT) and niche medical apps (Nabla, Heidi), while major companies tout AI diagnostic advances. The significance is immediate: biased triage, summaries or advice from AI could reinforce existing disparities and worsen outcomes unless models are rigorously audited and deployed with clinician oversight. Technically, the problems likely stem from imbalanced or biased training data, label and evaluation choices, and alignment/rewarding methods (e.g., RLHF) that don’t enforce equitable calibration across subgroups. Remedies include subgroup-specific benchmarks, counterfactual testing, calibration of severity predictions, transparent dataset documentation, and human-in-the-loop safeguards. Without these, scaling LLMs in healthcare risks automating and amplifying systemic under‑treatment.