Everyone learned the wrong lesson from the AI-hallucination lawsuits. The headline was "AI invents fake cases." The real danger is subtler and harder to catch: AI inventing fake quotationsinside cases that are entirely real.

The failure everyone knows: fabricated cases

In Mata v. Avianca (2023), attorneys filed a federal brief citing several judicial decisions that simply did not exist. The citations had been generated by ChatGPT, which produced plausible-looking case names, reporter numbers, and quotations out of thin air. When opposing counsel and the court could not locate the cases, the lawyers were sanctioned. It became the canonical cautionary tale of generative AI in legal practice — and it taught the profession to run a cite-check.

A fabricated case is, thankfully, easy to catch. You look it up; it isn't there; you delete it. A single lookup exposes the lie.

The failure most people miss: fake quotes in real citations

Now consider a more insidious variant. The case is real. The reporter citation is correct. The court, the year, the parties — all genuine. But the sentence the AI attributes to that opinion, the quoted holding it puts in quotation marks, was never written by that court. The tool grounded its answer on a real document and then fabricated language it claims to have found there.

This is the trap of "grounded" or retrieval-augmented (RAG) legal-AI tools. Grounding a model on real source documents reduces the odds of a wholly invented case — but it does not guarantee the model accurately quotes what it retrieved. It can still paraphrase a holding into words the court never used, or splice a quotation from one passage onto a proposition it doesn't support.

This quotes-in-real-citations pattern has reportedly surfaced in practice with commercial, grounded research assistants. In one reported example involving Thomson Reuters' Westlaw CoCounsel — discussed in connection with a matter referred to as U.S. v. Farris — the tool is said to have produced quotations that did not match the language of the cited authority, even though the underlying citation was genuine. We present this only as a reported illustration: we have not independently confirmed the exact case name, citation, court, year, or holding, and readers should verify those details before relying on them. The pattern, however, is well documented — and it is the pattern, not any single anecdote, that should change how you review AI output.

Grounding is not a guarantee — the evidence

This isn't a fringe concern. A 2024 study from the Stanford RegLab and the Institute for Human-Centered AI (HAI) evaluated leading purpose-built legal-AI research tools — the kind explicitly marketed as grounded and hallucination-mitigated — and found they still hallucinated on a meaningful share of queries.

The takeaway isn't "these tools are bad." It's that grounding reduces hallucination but does not eliminate it, and the residue includes exactly the dangerous variety: confident, well-formatted, cite-check-passing quotations that are wrong.

For a running, public record of real court cases that involved AI-hallucinated citations, researcher Damien Charlotin maintains a widely-cited database tracking these incidents as they reach the courts. It's an excellent resource for seeing how often — and how far into the process — these errors travel. We point to it rather than duplicate it.

Why single-model self-verification fails

The instinctive fix is to ask the same tool to check itself: "Are you sure that quote is accurate?" This rarely helps, and often makes things worse.

• The model that fabricated the quote has no ground-truth memory of the opinion — asking it to "verify" just re-runs the same pattern-matching that produced the error
• Prompts like "confirm this is verbatim" frequently return a confident yes — sometimes with a freshly hallucinated supporting detail
• A model's confidence score doesn't correlate with quotation accuracy, so it can't self-flag the risky ones

Why cross-model disagreement catches it

A lone grounded model has one retrieval and one way of quoting it. Multiple independent models each retrieve and quote on their own. When you ask several models the same question and one of them produces a quotation the others can't reproduce — or that the others explicitly rate as unsupported — that disagreement is the alarm bell.

A verification checklist for AI-assisted legal research

Whether or not you use a consensus tool, never file unverified AI output. At minimum:

• Confirm the case exists — but don't stop there; a real cite is where the dangerous errors hide
• Pull the primary source and read the quoted language in context — verify every quotation verbatim against the opinion itself, not against the AI's summary
• Check that the quote supports the proposition — a real quote can be attached to a claim it doesn't actually support
• Cross-check with independent models — a quotation only one model produces is a flag to investigate, not a fact to file
• Never treat "grounded" as "verified" — RAG lowers the odds of invention; it does not remove your duty to check

Consensus across independent models raises your confidence that a quotation is real. It does not prove it. The primary source is still the authority — the AI is only there to help you find it faster and to tell you where the models disagree.