We're training LLMs to hallucinate by rewarding them for guessing (lightcapai.medium.com)

A recent manuscript from OpenAI researchers sheds new light on why large language models (LLMs) hallucinate—produce confident but incorrect statements. The team argues that current training and evaluation frameworks incentivize models to guess answers rather than admit uncertainty. This “grading curve” encourages LLMs to perform well on benchmarks by maximizing apparent correctness, even at the cost of generating misleading information. Essentially, models are rewarded for being good “test-takers” rather than truthful responders, resulting in persistent hallucination issues despite advances in model architecture. The study dives into the statistical and algorithmic roots of hallucinations, framing them as errors in binary classification amplified by misaligned evaluation metrics. Rather than viewing hallucinations as mysterious flaws, the authors propose that they emerge naturally from the way we train and score LLMs. Highlighting a socio-technical approach to mitigation, the researchers suggest modifying benchmark scoring to penalize incorrect answers more heavily and reward expressions of uncertainty. This shift could promote more transparent and trustworthy AI systems, steering the field away from models optimized merely to guess and toward ones that better communicate their confidence and limitations. The findings elevate the conversation on improving evaluation metrics as a critical lever for reducing hallucinations in future language models.