Deceptive Grounding: Entity Attribution Failure in Clinical RAG (arxiv.org)

A recent study has unveiled a critical issue in clinical Retrieval-Augmented Generation (RAG) models, identifying a phenomenon termed "deceptive grounding" (DG). This flaw occurs when these models provide clinically relevant information from retrieved documents but assign it to the wrong medical entity. Even when a RAG response appears faithful and accurately cited, it can misrepresent data, such as misattributing clinical evidence from drug Y to the queried drug X. The study assessed 13 models and found DG rates ranging from 8% to a staggering 87% under peak adversarial conditions, with medical models showing alarming rates of up to 86.7%. This revelation has significant implications for the AI/ML community, especially in healthcare applications where accurate information is paramount. The study highlights that employing domain specialization actually exacerbates this entity-attribution failure rather than providing a solution. Notably, the implementation of an entity-attribution verification system demonstrated impressive effectiveness, achieving a precision of 97.0% and a recall of 98.7%. This points to a crucial need for improved evaluation frameworks in AI models to safeguard against misleading outputs in clinical settings, underscoring the necessity of integrating robust verification mechanisms in future deployments.