Mistake-filled legal briefs show the limits of relying on AI tools at work (apnews.com)

Judges and attorneys are increasingly encountering court filings that relied on generative AI and contained clear errors — notably fabricated case citations and false quotes. Damien Charlotin, a data scientist and lawyer, has tracked at least 490 such “hallucination” incidents in six months, mostly in U.S. filings by self-represented litigants, though even established firms and companies (e.g., a MyPillow brief with nearly 30 defective citations) have been caught. Courts have responded with warnings and, in some cases, fines. Beyond litigation, the problem surfaces across workplace uses of AI (search overviews, meeting notetakers, research) and raises legal and privacy risks when confidential data is uploaded to off‑the‑shelf tools. For the AI/ML community this is a practical warning: model hallucinations and weak source grounding have real-world costs. Technical fixes include stronger retrieval‑augmented generation, provenance-aware outputs, citation verification, and better uncertainty signaling; product fixes include human-in-the-loop workflows, audit logs, consent/recording controls, and enterprise training. Employers should treat AI as an assistant, validate outputs (especially legal facts), avoid feeding sensitive data into public models, and invest in user education. These incidents underscore that deployment safety, provenance, and usability — not just raw model quality — are critical to adoption in high-stakes settings.