When Your Benchmark Leaks the Answer

A detector I built was scoring 0.067 recall on temporal errors, meaning it caught about one in fifteen of the wrong dates it was supposed to find. Wrong dates are supposed to be the easy category: extract the years from the claim, extract the years from the source, compare. There is no semantics to get wrong. I assumed the extraction was broken and went looking for the bug.

The extraction was fine. The benchmark was the problem, and not in a way that showed up anywhere in the code. The contexts had been written in the wrong voice.

That's the part worth passing on, and it has nothing to do with hallucination detection. It applies to anyone who builds a synthetic evaluation set, which by now is most of us.

A Detector That Failed Because of Prose Style

The setup: a claim, a source context, and a question about whether the claim is supported. The detector is part of HallucinoType, an open-source package I maintain, and its benchmark was built the way most synthetic evaluation sets are built. Take a faithful claim, inject an error of a known type, keep the label. Two hundred fifty pairs, stratified across eight failure categories, thirty-five of them faithful so I could measure false positives.

When I wrote the contexts for the date items, I wrote them the way a person naturally writes when they know the claim is wrong. Something like the treaty was signed in 1928, not 1938. Read that as a human, and it is unambiguous. Read it as a program that treats the context as a reference document, and the string 1938 is sitting right there in the source. The detector extracted it, matched it against the claim, found agreement, and passed the item.

Every one of the thirty temporal items had this property. Seven of thirty numerical items did too. Rewriting the contexts as ordinary reference prose, the kind a retrieval system would hand you, moved temporal recall from 0.067 to 1.000. I changed no code.

The numerical items did not move. They stayed at 0.600, which told me their misses had a different cause and saved me from congratulating myself on a fix that only worked once.

The generalization is short enough to put on a sticky note. A context that argues with the claim is not the context a production pipeline supplies. I had unconsciously written my source documents in a fact-checking register, because I was thinking like an annotator rather than like a retrieval index. The register leaked the answer key into the input, and the system read it, exactly as it was built to.

What makes this uncomfortable is that nothing about the benchmark looked wrong. The labels were correct, and the errors were real errors; any reviewer would have signed off on it. The defect lived in a stylistic property of the prose that no one thinks to specify, and it moved a headline number by a factor of fifteen.

The Same Bug Wearing a Different Hat

The same mistake showed up a second time, and I did not recognize it at first. A second detector in the same system checks whether a claim names the wrong person, company, or place. It works by extracting entities from the claim and looking for them in the context. If the entity appears in the context, the detector skips the claim, on the theory that the source confirms it.

I upgraded the entity recognizer, the component that reads a sentence and tags which words are people, places, or organizations, from a regular-expression fallback to a proper statistical model. Recall fell from 0.200 to 0.067.

A better component made the system worse, which is the kind of result that stops you mid-sprint. The recognizer was not at fault. The skip rule was. A source document can mention a person in a role that has nothing to do with the claim under evaluation, and still mention them truthfully. In one item, the claim misattributed who was second to walk on the Moon, and the context named the substituted astronaut correctly in a different sentence, doing a different thing. The weaker recognizer missed that mention and flagged the error. The stronger one found it, read it as confirmation, and waved the error through. Of twenty-seven items the detector wrongly skipped, twenty-six followed this pattern.

The heuristic was never checking the right relationship. It asked whether the entity appears in the document when it needed to ask what the entity is doing in the sentence. Improving the model's ability to answer the wrong question just made it answer the wrong question more reliably.

This is a hazard anywhere a rule sits on top of a learned component. Ablating downward, swapping in a deliberately weaker component to confirm the strong one is earning its cost, is something I do routinely. Ablating upward is rarer, and it tells you more: a rule that degrades when its inputs improve is a rule whose logic was wrong all along, and no amount of model quality saves it.

A third instance, smaller but the same shape: a pattern for matching units of measurement was absorbing a trailing word, which let bare four-digit years slip past a filter meant to exclude them from numeric comparison. Fixing one regular expression moved numerical precision from 0.857 to 0.947. A lot of apparently semantic behavior turns out to be lexical.

What the Headline Number Was Hiding

None of these three defects were visible in the metric I would have reported at a demo. On the binary question of whether a claim is unsupported, the full system reached precision 0.991 and recall 0.986: almost nothing it flagged was fine, and almost nothing that was wrong got past it. Those are the numbers that go in an abstract.

Averaged across the eight failure categories the system is supposed to distinguish, precision was 0.578 and recall 0.723. One category sat at 0.067 recall. Another fired on nearly everything, reaching 0.960 recall at 0.198 precision, meaning it claimed credit for errors that more specific detectors had already identified correctly.

The binary number was not wrong. It was answering a question so coarse that every interesting failure averaged out of it. A system can be excellent at deciding that something is broken and close to useless at saying what broke. If the only number on your dashboard is the first one, you won't find out until the fine-grained output reaches someone who depends on it.

None of this is new. It's the same argument as reporting per-class results instead of overall accuracy on an imbalanced dataset. Everyone agrees with it in principle and skips it anyway, because the aggregate is the number that makes the case for the work.

Not Getting Fooled by Your Own Corpus

Four practices came out of this, all cheap, none of them clever.

1. Write your evaluation inputs in the register your production system receives. If your system reads retrieved documents, your test contexts should read like documents, not like annotations about documents. Voice is a feature your model can see, and the voice of someone who already knows the answer is a particularly dangerous one to hand it.
2. Ablate upward, not just downward. Replace a component with a better one and check that every metric moves in the direction you expect. When something moves the wrong way, the rule sitting on top of that component is making an assumption you have not written down.
3. Report per-stratum results next to the aggregate, always in the same table. Not in an appendix, not on request. If a category is at 0.067, that fact should be as easy to see as the number you are proud of.
4. Hold out items you did not write. A corpus built by the same people who defined the categories will flatter the categories. Mine did. That is the single largest caveat on everything above, and no amount of internal rigor substitutes for a test set authored by someone else.

The first one I would not have thought of before it cost me a day, and it's the one I now suspect is quietly wrong in a lot of synthetic eval sets. Injected-error benchmarks are easy to build, and their labels are correct by construction, which makes them feel safer than they are. The label being right does not mean the input is representative.

The Register Your System Actually Speaks

The failures worth writing up are rarely the ones where the model underperforms. They are the ones where the measuring apparatus was quietly reporting on something other than what you thought. A detector that scores 0.067 because the test data argues with itself is not a model problem. Neither is a rule that gets worse as its inputs get better, or an aggregate that averages away the only result that mattered. A bigger model fixes none of it.

What fixes them is reviewing the evaluation harness as carefully as the thing it measures, defects and all. That's unglamorous work, and where most of my debugging time went. Probably where most of yours goes too.