The Nature of Hallucinations (blog.qaware.de)

Sebastian Macke’s essay reframes “hallucinations” not as a bug but as a predictable consequence of how LLMs are built: they generate text by sampling the most likely next tokens to maximize coherence, not by verifying truth. When a model lacks a fact (e.g., a non-existent mountain), the probabilistic next-word process and resemblance to known patterns produce plausible-sounding but false outputs. Reinforcement learning and benchmark-driven training amplify this tendency—models are rewarded for guessing (like a test-taking strategy) and penalized for saying “I don’t know,” so uncertainty is suppressed by design. That diagnosis has practical implications for the AI/ML community. Current fixes—larger models, more memorized facts, and retrieval-enabled agents—treat symptoms; the root lies in training signals and evaluation metrics. Promising work (an unpublished OpenAI model that admitted incorrectness on IMO problem 6) suggests models can learn calibrated uncertainty, but doing so requires reworking reward functions and benchmarks, with trade-offs in user experience, latency, and scored performance. If solved, however, models could be smaller, cheaper, and more reliable—preferring lookup or abstention when appropriate—transforming both safety and the economics of deploying local, energy-efficient AI.