Metamorphic Testing For LLMs: The Oracle Problem's Most Underused Answer

Metamorphic testing (MT) is a practical way to generate test cases and verify results when exact oracles are hard to define. Metamorphic relations (MRs)are fundamental here: expected relationships between multiple inputs and their outputs for the same function or algorithm. As the technique has matured, researchers have explored ways to discover these relations, automate test generation, combine metamorphic testing with other software engineering methods, and use it to validate real systems.

In this article, I will explain what metamorphic testing is and its limitations. Some recent results are also put in perspective about MT's applicability for LLM testing. I also explain what this changes about using MT for LLM testing and where to start for LLM teams that want to implement MT. 

What We Want To Solve

Testing techniques often run into a brick wall: after you run a program with some input, how do you know whether the output is correct? For a login form, easy: you know the expected result. For a compiler optimizing a hundred-thousand-line program, a route-finding algorithm on a live map, or an LLM answering an open-ended question, there often isn't a practical way to compute the "right" answer. This is the oracle problem, and for language models it's not a corner case; it's the default condition. Text inputs are cheap and abundant, but correct-answer labels aren't. Especially once you're fine-tuning a model on your own data or deploying it privately for exactly the compliance reasons that make third-party labeling awkward.

One option to handle this is by human verification of outputs. Another option is to use a second model to grade the first. The latter just relocates the oracle problem one level up, since now you need to trust the grader. MT offers another alternative where you don't need to know in advance if an output is correct or not.

History

Are successful test cases (test cases that pass) useless? For MT, the answer is no. Since test-case generation strategies serve specific purposes, every generated test-case should carry some useful information about the code under test. One of the most challenging (and interesting) tasks in software testing is to examine how to make use of such useful, but implicit, information to support further testing. 

In MT, we first identify some necessary properties of the target function or algorithm. These take the form of MRs. These MRs are then used to transform existing (source) test cases into new (follow-up) test cases. But because the follow-up test cases depend on the source test cases, they should also possess some of the useful information. If the actual outputs of source and follow-up test cases violate a certain MR, then we can say that the code under test is faulty with respect to the property associated with that MR. 

Although MT was initially proposed as a method for generating new test cases based on successful ones, it soon became clear that it could be used regardless of whether the source test cases were successful or not. In addition, it actually provided a lightweight but effective mechanism for test result verification — MT was thus recognized as a promising approach for alleviating the oracle problem.

What Metamorphic Testing Actually Is

MT doesn't ask whether a single output is correct. It asks whether a necessary relationship (an MR) between multiple outputs holds, given a known relationship between their inputs.

A simple example makes the mechanics concrete before adding the complexity of natural language. Consider, for example, the sin(x) function. Verifying that sin(x) has been computed correctly for an arbitrary x is often impractical. This would require an independent, trusted implementation to compare against. But the sin(x) function has a property that must hold regardless of implementation: negating the input negates the output, so -sin(x) equals sin(-x). For the purposes of MT, the sin(x) function, -sin(x) = sin(-x) is our MR. Pick any value for x1 — this is the seed (or source) test case — derive a second input x2 where x2=-x1 (the follow-up test case), run the function on both, and check the relation -f(x1) = f(x2). If it fails, the implementation is wrong. No need for a known-correct answer about f(x1) or f(x2).

The natural-language version of the same idea, applied to an LLM, looks like this: feed a model a premise and hypothesis and ask whether the hypothesis is entailed, contradicted, or neutral with respect to the premise. Then paraphrase the hypothesis — reword it without changing its meaning — and ask again. The MR is simple: paraphrasing shouldn't change the classification. If it does, you've found a fault, and you never had to know the "correct" label for either version to catch it.

The table below shows what that looks like in practice. Each row is a source/follow-up pair: the same premise, an original hypothesis and its paraphrase, and the model's classification for each. None of these rows required a labeled dataset or a human judgment about what the "right" classification actually is — the fault is visible purely from the fact that two inputs the relation says should agree, don't.

The first two rows indicate faults: the model's classification flips under a transformation that should have left it unchanged. This is a metamorphic oracle violation regardless of whether "entailment" was the correct label for the original pair in the first place. The last two rows show that the MR holds.

Three things about this are easy to get wrong:

• An MR doesn't have to decompose neatly into "transform the input this way, expect the output to change that way" — some genuinely tie the follow-up input to the source's output.
• An MR doesn't have to be an equality. Plenty of useful MRs are subset, monotonicity, difference, or "stronger/weaker" relations.
• MT isn't only for oracle-free situations. It has caught real faults in small, thoroughly specified, extensively tested code bases where a conventional oracle did exist.

The Classic Limitations In A Nutshell

Before getting to LLMs specifically, it's worth naming what was already known to be unresolved in MT generally. None of it goes away just because the system under test got bigger:

• MR identification is still mostly art. Systematic techniques exist but need either an existing seed set or apply only in narrow domains.
• "Diversity" of relations was never formalized. A small set of diverse MRs is known to get most of the available fault-detection benefit. However, "diverse" has always been a matter of tester intuition rather than a measurable property.
• A violation tells you something is wrong, not what. This is OK for plain verification. However, this is a real cost if you want to debug or localize the fault.
• It alleviates the oracle problem; it doesn't retire it. No matter how good MRs are, the code under test could satisfy all MRs and still be wrong in a way none of them can capture.

The Evidence: Running MT on LLMs at Scale

A 2025 study ran a systematic literature search across 1,024 papers. From 44 papers that explicitly defined metamorphic relations for NLP, the study distilled them into a catalog of 191 unique MRs. The authors built LLMORPH, a framework implementing 36 representative MRs, and ran it against GPT-4, Llama 3.1, and Hermes 2. Four tasks have been studied: question answering, natural language inference, sentiment analysis, and relation extraction — for a total of 561,267 metamorphic test executions.

A few findings are worth understanding before you decide how (or whether) to apply this to your own system.

MT does find real faults, at a meaningful rate. Across all 36 relations, the average violation rate was 18%, ranging from 0% to as high as 80% depending on the specific relation and task. Relation extraction was the most fault-prone task tested; question answering the least.

It's genuinely complementary to labeled data, not a replacement for it. The authors compared MT's verdicts against ground-truth labels where available. In the large majority of cases, both oracles agreed. But in about 11% of all test groups, MT caught a problem — typically in a follow-up output — that the ground-truth check on the original input missed entirely. This was because the source output alone was correct even though the relation as a whole broke. In roughly 27% of cases it was the reverse: the source output was already wrong in a way MT's relation-based check didn't flag. This was usually because a wrong output led to an equally wrong but internally consistent follow-up output. Neither oracle subsumes the other.

The false-positive rate is real, and it's an NLP problem, not an LLM problem. Manual review of 967 flagged violations found a true-positive rate of about 62%. This means that more than a third of flagged "faults" weren't faults at all. The dominant cause wasn't the LLM under test. It was the input transformation itself misfiring (changing a paraphrase too much or too little). Or, it was the output comparison misjudging semantic equivalence (the BERT-based similarity scoring used to compare free-form answers has known blind spots, e.g., failing to recognize that "unknown" and a differently worded refusal to answer mean the same thing). Critically, this false-positive rate lines up with what earlier MT-for-NLP research reported on non-LLM systems: This is an intrinsic cost of testing natural language outputs with automated relations, not something specific to testing LLMs.

Effectiveness is relation- and task-specific — and that's actionable. Synonym substitution behaves very differently depending on task: swapping "tallest" for "highest" in a question is harmless, but swapping "great" for "superb" in a sentiment-analysis input can legitimately shift the sentiment score. This can turn a real behavioral difference into what looks like a relation violation. At the same time, a handful of relations held up consistently well across tasks and models, with a high failure rate paired with a low false-positive rate, making them reasonable defaults to prioritize rather than something you need to discover from scratch.

Real violations aren't flaky (inconsistent). Despite LLMs' well-known nondeterminism, the authors re-ran ~99,000 failing test groups ten times each. Most (62%) failed in the majority of runs, and 28% failed in all ten. The inconsistency that did show up was concentrated almost entirely in the false positives caused by input-transformation noise, not in genuine faults. A real violation, once found, is very likely to reproduce. Practically, that means you don't need to rerun a flagged case many times to trust it.

What This Changes About Applying MT to AI Systems

This is a different picture from treating MT as a speculative fit for LLM testing. It's not a silver bullet — a roughly 38% false-positive rate on raw violations means that we still need human verification. MT still misses more than a quarter of the failures that labeled data would have caught. But it's also not just theoretically promising anymore: it's a technique with a quantified failure-detection rate, a quantified (and now explainable) false-positive rate, and evidence that its output is stable enough to act on.

The practical framing this supports: use MT where labels don't exist or aren't affordable. This could be regression testing across prompt changes. It could be fine-tuning runs, or model swaps, where you don't need to know the "right" answer, only whether behavior changed. Labeled evaluation sets could be treated as the tool for everything else. The two are complementary lenses on correctness, not competing ones. The study's confusion-matrix breakdown is the first real evidence of how much each one catches that the other doesn't.

Where to Actually Start

The lower-risk entry points are the ones closest to what's now been validated rather than the technique's more speculative extensions:

• Don't invent relations from scratch. A public catalog of MRs across 24 NLP tasks exists. Start by picking a handful that are already documented as task-independent and effective, rather than guessing.
• Budget for manual triage from day one. A true-positive rate around 60% is the expected baseline for NLP-oriented MT. This is not a sign of a broken implementation. Plan review capacity accordingly, and consider it against the near-zero cost of the alternative (no automated check at all).
• Be deliberate about relation choice per task. The same relation (synonym substitution, tense changes, added negation) can be near-free of false positives on one task and noisy on another. Task-specific validation before rollout matters more than relation count.
• Use it for regression testing, not just one-off audits. Because it needs no ground truth, MT is well-suited to catching whether a prompt, fine-tune, or model swap silently changed behavior. This is exactly the CI/CD use case where labeled data is least available and most expensive to keep current.

Start with a handful of relations, not a taxonomy. Three to six well-chosen, genuinely different relations captured most of the benefit in the earlier oracle-substitution literature. The LLM-specific results reinforce it: a few consistently high-signal relations outperform a large pile of near-duplicate ones.

Wrapping Up

Metamorphic testing won't replace labeled evaluation for LLMs. The ~38% false-positive rate on flagged violations is a real cost, not a footnote. But the technique now has what it didn't have several years ago: a large, systematic, multi-model empirical record showing it catches faults labeled data misses. Its violations are reproducible rather than noise, and a documented catalog of relations exists so teams don't have to build the concept from first principles. If any part of your LLM-based system currently gets tested by "we looked at the output and it seemed fine" — and for most LLM features, that's still most of them — this is worth a pilot on one task before it's worth a policy.