Dealing with GPT Results, Or, Pots, Kettles and Hallucinations (ebellani.github.io)

A viral New York Times story about a lawyer who used ChatGPT to fabricate case citations crystallizes a persistent problem: GPT hallucinations—plausible but false outputs—can produce harmful outcomes in legal, medical, or safety‑critical settings and erode trust in AI. Eduardo Bellani’s piece (citing Waldo & Boussard 2024) argues hallucinations are not random bugs but predictable behavior: LLMs surface the most common patterns in their training corpus, so where there’s broad consensus they appear accurate, and where data is sparse or contested they invent content. Subtle, authoritative‑sounding errors are especially dangerous because non‑experts tend to equate frequency in the training data with truth. Technically this implies limits and practical countermeasures: truthfulness depends on data coverage, task framing, and model grounding. Bellani highlights epistemic trust—science’s reliance on evidence, methods and peer review—as the gold standard LLM outputs lack unless externally verified. Even newer models (example ChatGPT‑5 replies) can confidently contradict historical nuance. For practitioners, the takeaway is clear: use LLMs for exploration, drafting and templating, but enforce retrieval/grounding, uncertainty calibration, human‑in‑the‑loop verification, domain fine‑tuning, and retrieval-augmented generation when deploying in high‑stakes domains to mitigate hallucination risk.