Why Knowing Your LLM Hallucinated Is Not Enough

There is a moment every developer building with LLMs knows well. The model says something wrong, completely wrong, with total confidence. You catch it. You log it. You note that a hallucination occurred. And then what?

Most teams stop there. They have a hallucination rate. They report it in evaluations. They try to push it down. But they are treating very different problems as if they were the same, which is a bit like a doctor logging "patient is sick" and calling it a diagnosis.

Here is what I mean.

Three Hallucinations That Need Three Different Fixes

Consider these three model outputs, each factually wrong:

• Output A: "Einstein won the Nobel Prize in 1925." (He received it in 1921.)
• Output B: "The Phase 3 trial demonstrated 78% efficacy." (The paper said 38%.)
• Output C: "Studies show that this treatment is universally effective across all patient populations." (One small study in one demographic showed promising results.)

All three are hallucinations. A binary detector flags all three as hallucinated: true and moves on.

But look at what is actually going wrong in each case, and they have nothing to do with each other.

Output A has the right entity (Einstein) and the right event (Nobel Prize), but the wrong year. The model pulled a real fact and attached it to the wrong timestamp. That is a temporal confusion, and it is a clean one: nothing else is wrong, just the date. The fix is better grounding in dated sources, or a retrieval layer that anchors claims to specific years.

Output B has the right study and the right metric, but the number is off by a factor of 2. The surrounding context is correct; the statistic is wrong. That is numerical distortion, and it often happens because models smooth over numeric precision during generation. The fix is structured extraction with explicit numeric verification, not general retrieval improvement.

Output C has facts that are arguably defensible in isolation. The study exists; it did show results. But the scope got inflated from "one study" to "studies show," and from "one population" to "all patient populations." Nothing was made up, exactly. But that specific-to-general leap turns a reasonable claim into a misleading one. That is overgeneralization, and it is not caught by fact-checking against the source document, because the source document contains the specific claim the model inflated.

Same hallucination label. Completely different failure modes. Completely different mitigations.

Hallucination is used more broadly in the operational sense in my definition of it throughout the essay: the generation of outputs that contain unsupported claims or misleading content, as a result of either incorrect retrieval, generation, or logical inference. Overgeneralization and relation errors can also be considered problems of reasoning by other researchers, rather than hallucination. The difference makes sense from a research perspective. It makes no difference from a practical point of view.

The Problem With "Hallucinated: True"

When you reduce hallucination to a binary, you lose the information you actually need to improve things.

Teams spend weeks tuning RAG pipelines and are confused when their hallucination rate barely drops. RAG helps with some types (factual grounding, date anchoring) and does almost nothing for others. Overgeneralization, negation flips, confident fabrication of things that never existed anywhere in the retrieval corpus, better retrieval does not touch those.

You also get evaluation results that do not transfer. A model that scores well on one benchmark can still fail badly in production because the benchmark was measuring the wrong type of error for the use case. A medical application should weigh negation flips and numerical distortions very differently from a customer support bot. But if your metric is just hallucination rate, you can not make that distinction.

There is also the attribution problem. When something goes wrong in a production system, you need to know why. "The model hallucinated" is not a root cause. It is a symptom. Was it a retrieval failure? A generation failure? A reasoning failure? The type tells you. The binary flag does not.

What Typed Detection Actually Looks Like

I have spent the last few months working through a typed detection approach that replaces the binary flag with a fingerprint: a typed, attributed output that tells you not just that something went wrong, but what kind of thing and where.

The taxonomy has eight types: temporal confusion, numerical distortion, entity substitution, source blending, confident fabrication, relation errors, negation flips, and overgeneralization. The categories came from recurring failure patterns across retrieval, reasoning, and generation stages, not from prior taxonomies, though some overlap with NLP error classification work. The key design constraint was operational: each type needed a distinct mitigation, otherwise the distinction was not worth making.

The key insight is that hallucination mitigation is type-dependent. Some failures are retrieval problems. Others require span-level verification or cross-source contradiction checks. This is one reason retrieval augmentation alone so often disappoints; it is the right fix for one class of failures and largely irrelevant to the others.

Certain categories can be caught using lighter models. Year inconsistencies and numeric mismatches do not necessarily warrant a heavier model; careful extraction and matching are mostly sufficient here. Other categories are dependent on an LLM-as-judge check for reasoning: invented citations, inverted cause-and-effect relations, and mixing of facts drawn from two separate sources to form false statements. 

As far as real-world examples go, sophisticated errors are typically combinations of several categories at once: incorrect information fetching causing entity misidentification, mixing sources leading to overgeneralization, etc. It is precisely for such cases that the role of attribution becomes just as important as classification.

The output for the Einstein example above is not hallucinated: true. It is:

temporal_confusion (confidence: 0.85)

→ "1925" in claim doesn't match context
→ correct value: 1921
→ span: characters 30–34

That is a root cause, not just a flag.

Why the Distinction Matters Now

LLMs are deep into production in medicine, law, and finance, places where not all errors are equal. A negation flip in a clinical summary, where the model says a treatment "did not show efficacy" when it did, is not in the same category as a one-year date error in a historical overview.

Binary detection treats them the same. That is the problem.

If you are building in a domain where getting the type wrong has real consequences, I would genuinely like to hear what errors you are hitting most. The patterns that show up in health and legal contexts are different from what I have seen in general-purpose retrieval systems, and those differences matter for how you build detection.

If we want LLM systems that are actually trustworthy in production, we need to stop asking only whether a model hallucinated, and start asking how.