OpenAI says models are programmed to make stuff up (www.theregister.com)

OpenAI and a Georgia Tech coauthor published a paper, "Why Language Models Hallucinate," arguing that hallucinations are not just bugs but an expected outcome of how models are trained and evaluated. The paper shows pretraining embeds guesses when the corpus lacks a learnable pattern (e.g., rare factual items), and post‑training incentives—benchmarks and reward models that favor binary correctness over uncertainty—push models to provide answers rather than admit ignorance. Empirically, the authors note that a base model’s hallucination rate should be at least the fraction of facts that appear only once in the pretraining data; as an illustration, an OpenAI bot produced three different incorrect birthdays for researcher Adam Tauman Kalai because trainers rewarded answering over saying “I don’t know.” The significance is broad: hallucinations are a systemic evaluation and training artifact, not solely a model-capacity or dataset-cleanliness problem. That means mitigation requires changing objectives—rewarding calibrated uncertainty, adjusting benchmarks away from exam‑style binary scoring, and reworking post‑training optimization—because perfectly error‑free pretraining corpora are unrealistic. OpenAI suggests and claims partial progress (e.g., changes in ChatGPT‑5) toward making models express uncertainty more often, but the paper warns that without incentive realignment, hallucinations will persist as a rational strategy for models optimized to “score” rather than to be truthful.