Counterfeit judgments in large language models (www.pnas.org)

The PNAS piece shows that large language models can produce “counterfeit judgments”: outputs that look like thoughtful, human-style decisions (reasoned explanations, confidence scores, or categorical verdicts) but are not grounded in genuine evidence or reliable decision processes. Through controlled prompts and evaluation tasks the authors demonstrate that LLMs readily generate persuasive rationales and high-confidence answers across domains—moral, aesthetic, legal and factual—while exhibiting poor calibration, high sensitivity to wording, and vulnerability to spurious correlations from training data. The result is a plausible-seeming but potentially misleading form of automated judgment that can be mistaken for expert reasoning. This finding matters because many deployments rely on models to summarize, evaluate, or decide (hiring screens, legal triage, medical summarization, moderation, peer review). Technically, the paper highlights that chain-of-thought-style explanations increase apparent transparency without reliably improving correctness; internal confidence signals (logits, tokens) are weak proxies for true uncertainty; and fine-tuning or prompt hacks can make models mimic specific decision styles without genuine grounding. The authors advocate mitigation measures — better calibration and uncertainty modeling, adversarial and out-of-distribution testing, provenance and evidence requirements, and human-in-the-loop approval — to avoid treating model-produced judgments as authoritative in high-stakes settings.