Ask a legal research tool whether a case exists, and it will usually tell you the truth. Ask it whether the case says what you think it says, and you are on your own.

That gap is the part of the legal AI problem the profession has not priced in. The fear that made headlines was fabrication: an AI invents Varghese v. China Southern Airlines, a lawyer files it, a judge notices. It is a vivid failure and an easy one to guard against, because a citation either resolves or it does not. Checking is mechanical, and the tooling to check it is now everywhere.

The failure that survives is quieter, and it is not about citations at all.

The finding nobody quotes

In 2024, researchers at Stanford published the first systematic profile of legal hallucination in large language models. Large Legal Fictions tested the models of the day against specific, verifiable questions about randomly selected federal cases, and found hallucination rates between 58% and 88% depending on the model.

Those headline numbers got the attention. A different result in the same paper deserved more of it.

The researchers also ran what they called a contra-factual setup: they asked questions containing a false premise. Not “what did the court hold in this case”, but a question that quietly assumed something untrue about it. The models frequently failed to correct the premise. They answered the question as posed.

Read that as a lawyer rather than as an engineer and it is a different kind of problem entirely. A junior associate who does not know an answer says so. A junior associate who thinks your framing is wrong tells you, sometimes at professional cost to themselves. That instinct — to refuse the question as asked — is most of what makes a second opinion worth having.

A language model has no such instinct. It is built to be maximally responsive to the prompt it receives. If your prompt contains your theory of the case, the theory comes back to you with citations attached.

Grounding was supposed to fix this

The industry’s answer was retrieval. Stop letting the model recall law from training data, and make it read from a real corpus instead. Purpose-built legal research tools were rebuilt on that principle and marketed accordingly, some of them with explicit promises of citations free of hallucination.

In 2024 the same Stanford group ran the first preregistered independent evaluation of those tools. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools put over two hundred legal queries through the leading commercial systems and had the answers hand-scored by legally trained evaluators. It has since been peer reviewed and published in the Journal of Empirical Legal Studies.

The tools did better than an ungrounded model. They did not do what the marketing said. The study reported hallucination in a meaningful share of responses — enough that its authors concluded, in plain terms, that providers’ claims were overstated.

Two things follow, and they point in opposite directions.

The first is that grounding works. Retrieval measurably reduces the error, and any serious system should use it.

The second is that grounding is not sufficient, and the reason is structural. Retrieval changes what the model reads. It does not change what the model is for. A retrieval-augmented system still takes your question, still treats it as the thing to satisfy, and still has no mandate to tell you that the question is built on sand. Give it a false premise and better sources, and you get a false conclusion that is now well cited.

Why more accuracy does not close the gap

There is a tempting response to all of this: the models keep improving, so wait. Error rates fall, retrieval corpora get better, verification layers get bolted on, and eventually the numbers get small enough to trust.

That reasoning works for the fabrication problem. It does not work for the premise problem, because the premise problem is not an accuracy failure. A system can be entirely accurate about every proposition it asserts and still be useless to you, if the thing it never questioned was your starting assumption.

Consider what a partner actually does when handed a memo. They do not begin by verifying the citations. They read the conclusion, and then they look for the argument that would defeat it. The value is not in confirming that the sources exist. It is in the adversarial pass.

No amount of accuracy inside a single answer produces that pass, because a single answer has no one in it whose job is to disagree.

What a second opinion has to be made of

If the missing thing is disagreement, then it has to be built in structurally rather than requested politely. A prompt asking a model to “consider counterarguments” is still one system, reasoning once, in one direction, with all of its blind spots intact.

That is the reasoning behind how Pilot5 is built. Five independent models analyse a question in parallel, with no visibility into each other’s work, so their errors do not correlate. They then critique each other’s analyses anonymously, stripped of any signal about which model produced what, so the critique lands on the reasoning rather than on the reputation. One of the five holds a standing mandate to attack whatever the others converge on.

Where they agree, that agreement means something, because it survived the attempt to break it. Where one refuses to join, the objection is preserved as a Minority Report rather than averaged into the consensus — which is exactly the sentence a careful reader wants and a confident single answer never contains.

Every claim carries a provenance tag: traced to a retrieved source, or marked as inference. A claim that cannot be traced back to what was actually retrieved is relabelled as inference automatically, rather than being allowed to keep the authority of a citation it has not earned. And every deliberation leaves a timestamped record you can retrieve later, which matters when the question is not “was this right” but “what was in front of us when we decided”.

What this does not do

It does not verify law. It is not a legal research tool, it does not give legal advice, and nothing in it substitutes for a lawyer reading the opinion. Adversarial structure reduces the chance that a weak position goes unchallenged. It does not reduce it to zero, and a system that told you it did would be making the same category of promise this article is about.

The honest claim is narrower and more useful. The received worry about legal AI is that it will invent something and you will not notice. The more likely failure, on the published evidence, is that it will agree with you and you will not notice.

One of those is caught by checking citations. The other is only caught by something whose job is to argue.

Sources

  • Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024). Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. arXiv:2401.01301
  • Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies (2025); preprint arXiv:2405.20362