Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought (arxiv.org)

Recent research highlights critical vulnerabilities in large language models (LLMs) used for legal reasoning, specifically in their so-called "chain-of-thought" processes. The study systematically examined whether these models reliably justify their legal decisions based on the statutes or precedents they cite. Despite high accuracy in naming the correct legal authorities (ranging from 66.7% to 100%), the models' ability to maintain consistent verdicts when these authorities were altered was significantly lower, with changes in legal authority resulting in verdict shifts from 0% to 76.7% across various benchmarks. Remarkably, the research also found that the models demonstrated a higher sensitivity to adversarial instructions embedded in case facts than to changes in legal reasoning, which raises important concerns about the reliability of LLMs as compliant and accurate tools in legal decision-making. This study is significant for the AI/ML community, as it underscores the limitations of LLMs in high-stakes applications like legal reasoning. The findings suggest that simply citing a legal authority does not guarantee that a model’s verdict is genuinely dependent on that authority, and exposes the models to potential adversarial manipulation. As legal professionals and developers look to integrate AI in compliance and auditing processes, these insights are critical in re-evaluating how generated legal explanations are perceived and utilized. This research calls for a rethinking of AI's role in legal contexts, pushing for more robust validation methods to ensure that AI-generated content is not only accurate but also faithfully aligned with the legal reasoning it claims to represent.