So, AI Made You a Superman? (medium.com)

An attorney, Steven Schwartz, relied on ChatGPT while drafting a legal brief and accepted generated case citations that didn’t exist; after submitting them he was sanctioned and fined by a federal judge. The episode crystallizes a common failure mode of large language models (LLMs): they produce fluent, professional-sounding text by probabilistically predicting the next token, not by verifying facts. Reinforcement Learning from Human Feedback (RLHF) makes outputs more polished and persuasive—boosting readability and automation bias—but does not guarantee factual accuracy. Long-tail details (case names, citations, statute numbers) are particularly vulnerable to coherent fabrications, and even models with internet access can still hallucinate or misattribute sources. Beyond one lawyer’s mistake, the case highlights systemic risks: cognitive outsourcing, processing fluency, and the Dunning–Kruger blind spots that lead professionals to trust AI too quickly. For the AI/ML community this means technical and policy priorities: strengthen retrieval-augmented generation and provenance, integrate deterministic fact-checking tools, improve grounding and uncertainty calibration, and design human-in-the-loop workflows that assign clear accountability. Practically, organizations and open-source projects are already drafting AI-usage policies; the safer path is responsible augmentation—use AI to accelerate work, but verify and take legal and ethical responsibility for outputs.