Bounding Hallucinations: Merlin-Arthur Protocols for Mutual-Information Bounds (aleph-alpha.com)

A new preprint titled "Bounding Hallucinations: Merlin-Arthur Protocols for Mutual-Information Bounds" addresses the pervasive issue of hallucinations in language models by introducing a novel training approach. The proposed method transforms the training process into a game involving three players: Arthur (the language model), Merlin (who provides supportive context), and Morgana (who intentionally removes crucial evidence). This setup incentivizes the model to accurately identify when it cannot answer based on the provided document, thereby reducing the occurrence of misleading outputs that arise from guessing or hallucination. The significance of this research lies in its potential to enhance the reliability of language models by implementing a framework that quantifies how much of the generated answer can be traced back to the specific document provided. Through this innovative approach, the study reports a reduction in incorrect responses by up to 35 percentage points on various question-answering benchmarks and a notable improvement in grounding scores, indicating a stronger correlation between answers and their documentary sources. This advancement not only fosters greater trust in language model outputs but also refines retrieval-augmented generation techniques by ensuring that AI-generated answers stem from verifiable information rather than mere fluent guessing.