DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Do We Test Just to Find Bugs?
  • Why Your QA Engineer Should Be the Most Stubborn Person on the Team
  • AI-Assisted Testing: Real-Life Use Cases vs. Myths
  • Building a New Testing Mindset for AI-Powered Web Apps

Trending

  • Member Spotlight: Raghava Dittakavi
  • Persisting Cookies Across JAX-WS Service Calls Using a Custom SOAP Handler
  • Spring Security JWT Validation Best Practices
  • Post-Quantum TLS for Cloud APIs and Microservices

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Testing, Tools, and Frameworks
  4. Carry the Bag of Reality Into the New Year? What Would Gödel Say?

Carry the Bag of Reality Into the New Year? What Would Gödel Say?

As Gödel showed, a system cannot judge its own rules from within. QA and the business are outside — and that is why they matter, even with 3,000 tests.

By 
Denys Lypnytskyi
·
Nov. 18, 26 · Opinion
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
33 Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

November is upon us, and one or two releases remain before the New Year. We hope that next year there will be fewer illnesses, epidemics, and — which matters for our work — less dirt in repositories and healthier code. But reality teaches us that problems don’t solve themselves. And so I look at that ball of mud which pretends to be test scenarios, and I understand that I will have to somehow coexist with it for one more whole year.Godel and mutation testing


This article is not about global problems. It is a little about computer science, more about programming practice, and about organizing testing for large and old projects. Namely: why the cleanup of legacy tests should not be postponed endlessly, how to approach it, and finally, why 100% mutation coverage does not mean we are safe. Why should QA be strengthened, not as a team of nitpickers, but as the last citadel of human judgment in a world of AI agents. And, of course, what Gödel has to do with it.

The Bag of Dirt

I keep working on a project with roughly three thousand high-level (that is, non-modular / non-unit) tests. They were called integration and regression tests, but the closer I looked, the more clearly I saw: behind them stood neither a testing strategy nor any kind of order (the package layout didn't count). Into one big bag of dirt, people simply kept tossing in more and more unnecessary tests.

It all started quite reasonably. Real client documents and real problems. The program processed the input, developers compared the result with the expected finding and fixed bugs. Then people, and later AI agents, started filing “known defects” (KD-x tickets), writing patches and tests for them. When a change broke something else, a new regression test appeared. Over the years, this turned into a scaffolding that nobody took down.

What do these three thousand tests actually check? How do they relate to the business rules that existed back then and changed over time (some were even versioned)? Does the business know at all what our tests protected, and whether they match the new rules? And the QA team? Do they have confidence that the pyramid has no potholes, no broken blocks, no missed scenarios? Everyday communication with the teams confirmed it: No, there is no confidence…

But throwing away testing garbage by the New Year, it seems, is not going to happen without a hitch. If we go through this set piece by piece and scrupulously, we probably won’t manage it even in a year. Maybe there is some optimized plan for how to at least start this cleanup?

For example, why not run this dirt through a filter based on mutation testing? The idea looked tempting: write the harness needed for mutation testing of the existing suite; raise the mutation score as high as we can (kill the mutants); and then find the tests whose removal does not lower the achieved score — essentially duplicates (and kill them too). After all, if the remaining suite still detects almost all the selected mutations, doesn’t that mean we didn’t throw the baby out with the bathwater? Could such a suite be considered reliable enough to start getting rid of the tests accumulated over the years?

That was the dream. But then reality caught up with me.

1. Tests as Archaeology

The tests lived on simply because nobody dared to delete them. Sometimes a test pins down behavior that nobody can explain, and changing it is scary. What if someone depends on that oddity? Each of these tests is evidence of the program’s encounter with reality. Once upon a time, this evidence was valuable. Years later, the old timers and the new team members can no longer understand what it proved. Obviously, accumulating examples is not the same as accumulating understanding. Three thousand tests may describe three thousand situations — or a handful of scenarios with a lot of background, and practically no variation that affects the behavior under test.

Besides, the suite is not homogeneous. Some tests resemble characterization tests in Michael Feathers’ sense: they fix a chosen aspect of observed behavior. Others are, in essence, a full snapshot of the system — a stored reference result against which every next run is compared. But that is a death mask: it precisely conveys the features of the face, yet says nothing about what the face should be. And these are not at all proper checks of business requirements. The difference matters: the expected result may simply have been taken from the output of the program itself, and nobody checked it against the business rules. I will come back to this.

Bottom line: the number of files says almost nothing about the quality of the suite. But its age and disorder don’t prove it is useless either. I needed another way to evaluate and filter it.

2. Mutation Analysis: Unpacking the Bag

So, to carry out the plan we have several tools that solve different problems:

  • JaCoCo - which lines and branches are executed when the tests run?
  • Mutation analysis (for example, PIT) - do the tests notice specific changes in the implementation?
  • Targeted tests - which exact behavior are we protecting, and which deviation do we consider an error?

Setting JaCoCo aside (but not excluding it) for sifting out old, dirty tests, I took mutation testing as a finer scalpel. The idea of mutation testing is simple: make a small change to the program — invert a condition, swap a comparison operator, remove a check, replace a return value — and see whether the tests fail.

If they don’t fail, the mutant survived, and we go looking for why. Perhaps the mutation is equivalent to the original code and the observable behavior doesn’t change. Or perhaps the tests execute this code but never check its result.

The situation started to change. Next to the three thousand junk tests, a concrete list appears of changes the tests never noticed. Every surviving mutant had to be examined to find out which check was missing. This work suits an AI agent well: it can analyze the mutated code, identify the unaffected behavior, and propose a killer test.

An example from the project

The kind code is a short suffix (e.g., A1, B2) that follows the application number found by Scanner. This logic was extracted from the “God” class Scanner into CodeBinder without changing it.

Before the extraction, PIT reported 62 mutants in this logic, and almost all of them were killed incidentally by Scanner’s end-to-end tests. After just a small refactoring, there were more survivors.

Inside Scanner, this logic only ever saw what the Scanner produced. As a standalone class, CodeBinder accepted any input, so cases that could never occur before became part of its contract, and nothing tested them. In this example, a kind code may appear no more than three characters after the number. A mutant that replaced <= with < survived because no test placed the code exactly at the boundary. The holes had always been there, the refactoring made them visible.



Gradually, the new tests, built around mutants as a new frame, began to look self-sufficient: the coverage they gave was high. And the three thousand old tests started to look like an even bigger burden.

A real run

What the experiment showed so far: mutation coverage of the old suite (67%) versus the new one (84%) showed the new suite had 600 tests against roughly 3000 in the old one. The new suite kills the mutants PIT generates in modules A and B — the question is, how much does this prove that it covers the same real documents?


Then I did what seemed rational at the time: I stopped running the old tests altogether and started relying on the new ones, built around mutants. Why drag the burden of thousands of files along if you can check the code’s behavior directly? This is exactly where I ran into a problem that turned out to be more important than mutation analysis itself.

3. Mutants Know the Code, But Not the Business

Let’s do a thought experiment: suppose the mutation score is 100%. In practice, it never happens. Equivalent mutants, tool limitations, and the chosen set of mutation operators always leave a gap. Still, let’s imagine…

Such a seemingly complete check analyzes the code we have, but is this the code that should actually be there? If the program is far from correct, a perfect score only proves that we reliably guard its current state. Without the philosophy, we are left with three blind spots, and none of them disappears as the coverage percentage grows:

  1. Errors of omission. You can only mutate what is written. If the rule “for an updated document, take the valid date” was never implemented, there is no condition in the code to invert and no branch to remove. Missing code produces no mutants, and mutation analysis is helpless here.
  2. The wrong oracle. A made-up example: the program extracts a citation from a document, and the test checks that the first citation in the text was extracted. Mutation analysis confirms the test is sensitive to every change in this code. But the business needs the citation standing in a particular context — in some documents, it is the second citation. The test will flawlessly protect the wrong behavior, killing mutants and cementing the error.
  3. Unrepresented inputs. A mutation changes the program, not the world. It does not know that a certain wording in a document has a special meaning for the client, that two documents which look alike require different results, and that a rare combination of conditions in real data leads to different scenarios. If such a document is not in the suite, a mutation will not create it.

History of the method

When DeMillo, Lipton and Sayward formulated the mutation testing method in the late 1970s, they relied on the competent programmer hypothesis: the program is already almost correct, and its errors are small deviations from the right logic. A mutant is exactly such a deviation. So the mutation score measures how sensitive the tests are to small changes around the existing implementation


What, then, do we claim at 100%? That the tests distinguish the original implementation from each of the generated modified versions. This is a valuable property, but it answers a question about the tests, not about the program. A mutation is a change in the code. A business situation is an event or a variation from the real world. There is no automatic bridge between these worlds.

Don’t flatten the pyramid

Mutation testing also has technical blind zones. PIT’s standard operators do not mutate string literals — and therefore not regular expressions, where, in extraction tasks, a noticeable part of the logic lives. Stateful behavior, compensating and rollback paths, and sagas are checked only by tests that play the scenario through end to end; combinations of application configurations need their own tests. So the pyramid still matters: unit tests at the base, integration and contract tests in the middle (contracts also let other teams rely on stubs of your service), a thin layer of end-to-end tests at the top. Nobody argues.

4. Where Do Expected Results Come From?

At this moment, I raised my eyes to the bookshelf and saw one of my favorite books — Gödel, Escher, Bach by Douglas Hofstadter. The book opens with a puzzle. There is a formal system, "MIU": the starting string "MI" and four rewriting rules — for example, to a string ending in "I" you may append "U", and three "I"s in a row may be replaced with "U". The question: can you obtain the string "MU"?

However much you apply the rules, "MU" never appears. But from inside the system you can’t understand that: the mechanical application of rules can go on forever without ever giving an answer. The answer is found outside — if you look at the system as an object and notice an invariant: the number of "I"s in the string is never divisible by three, and in "MU" there are zero of them. Hofstadter calls this jumping out of the system.

What Gödel gave us

Gödel’s theorems bring this thought to mathematical rigor: in a sufficiently expressive consistent formal system, some statements can neither be proved nor refuted by its own means, and such a system cannot prove its own consistency. To software testing, of course, Gödel’s theorems do not apply directly — this is an analogy, not a consequence. But a useful analogy: code and tests that check each other can confirm their mutual consistency. Whether this consistency corresponds to the real world, they cannot establish from the inside.


In testing, the challenge of deciding whether an observed result is correct has its own name — the oracle problem. The oracle is whatever makes that judgment, and a test’s ability to detect faults depends in part on how reliable its oracle is. In the survey by Barr, Harman et al., oracles are divided into several types:

  • Specified - the expected result is derived from requirements or a specification.
  • Derived - the expected result is taken from another version of the program, a previous run, or a reference implementation. Snapshots and golden masters belong here.
  • Implicit - only that the program didn’t crash, hang, or throw an exception is checked.
  • Human - an expert confirms the correctness of the result.

Now let’s look at our bag of dirt through the filter of this classification. The old tests with references taken from the program’s output are derived oracles: they check consistency with the past, not correctness.

The tests an AI agent created to kill mutants often cement this “correctness” even harder, adding guarantees of inviolability to it. To kill a mutant you need an expected result, and the most accessible source is whatever the current code does. The surviving mutant is killed, the score grows, and the oracle is once again taken from the legacy system. If you don’t demand otherwise from the agent, it tightens this loop by itself.

So even with a perfect mutation score, the main mutant survives — or rather, the main question. And this question is reflected in Barry Boehm’s classic dilemma — the dilemma between verification and validation: “Are we building the product right?” and “Are we building the right product?”. Mutation analysis lives entirely on the verification side: it works inside an already chosen model and evaluates the tests against predefined criteria. Where the criteria themselves came from, it does not know.

If the answers to the questions “why these documents?” and “are the results correct?” are taken from the very system under test, then we are in the position of that agent rewriting strings in the MIU system. The cycle goes on, the rules are applied, the indicators grow, and the very main question remains unanswered. To get out of the loop, you need a foothold outside: a requirement, an expert, scenarios of reality — not a digital twin, but a document with a confirmed result.

5. How to Unpack the Bag Now

Which tests to carry into 2027: the old “bag of reality”; or the new tests with maximum coverage — appeared to be the wrong starting point. Oversimplifying the task won’t work, and there remain at least three questions: (1) which functionality of the program are we checking, (2) why do we consider the expected results correct, and (3) which real scenarios will we stop protecting if we delete a particular test?

Each needs its own kind of evidence and its own testing strategy:

Risk

Question

What closes it

Who answers for it

Verification

Do the tests notice code changes?

Coverage, mutation analysis, targeted checks of conditions and branches

Developers and the AI agent

Oracle

Are the expected results correct?

Requirements, KD-x tickets, independently justified references, expert confirmation

QA and the business

Representativeness

Are we checking the right documents?

Real documents, known hard cases, feedback from production

QA, support, and the business


The strategies don’t conflict, but they don’t replace one another either. The new suite built around mutants was not wasted time. It simply covered the first position. The old suite was poorly organized, but so far, the only claim on the third. The second is covered right now by neither.

So my life hack, my hope for a quick solution — that is, my filter for a quick cleanup of the bag of mud, worked only partially. At the next step, I was already preparing — for myself and for my team — a plan for the coming year, consisting of five stages. The order in it matters: each stage relies on the previous.

Stage 1. Bring back the safety net. The old suite is neither deleted nor disabled, but moves to a nightly run. This is cheap, and every failure that the new suite misses points directly at knowledge the new tests don’t have. Such cases are the most valuable finds of the whole process, and they are worth collecting in a separate list. At first I did otherwise; I now consider that a mistake.

Stage 2. Measure the old tests with the same tool, and on the same area of code. Mutation analysis shows which tests kill unique mutants and which, by this criterion, fully duplicate each other. There is a technical trap here: for speed, PIT stops the run of a mutant at the first failing test, so the usual report doesn’t show how many tests kill it. To find duplicates, you need the full “mutant × test” matrix. In PIT, it is enabled by the fullMutationMatrix option — despite being experimental. The run will be several times longer, but for a one-off analysis that is acceptable. The result of the stage is not a deletion list but a map: where the tests overlap densely, and where a single test holds mutants on its own.

Stage 3. Dig up the provenance. The first section was called “Tests as Archaeology” not only for the metaphor. Most old tests have a history: the commit that added them and the KD-x ticket that commit belongs to. The ticket often records what is neither in the code nor in the test: what exactly the client considered an error and what result they expected. This is the closest thing to a specified oracle. Linking a test to a ticket through the commit history is mechanical work, and it is worth handing to the AI agent.

Stage 4. Sort the tests by purpose. Each test gets one category (category Label A):

  • Business invariants - rules that must hold regardless of the implementation.
  • Known defects/bugs (KD) - real cases where the program produced a wrong result; ideally with a link to a ticket.
  • Representative data - examples of important classes of input documents.
  • Implementation checks - targeted tests of conditions, branches, and edge cases, including those written against mutants.
  • Unexplained historical tests - value and correctness not yet established.

The second category (Label B) records the oracle source: (1) a requirement, (2) a ticket, (3) an expert, or (4) “taken from the program’s output”, etc. Together, the two labels are the “input/expectation” matrix, written directly into the tests, for example, as JUnit tags. Once it lives in the code, this knowledge no longer depends on anyone’s memory, mine included.

The third category (category Label C - more complex but more promising here), employs the AI agent — this time not for generating tests, but for text context analysis:

  • group similar;
  • find variations in expected results within a group;
  • propose hypotheses about the business rules behind these facts;
  • identify odd tests that look like duplication but may check different properties.

The main rule for the AI agent at this third stage: do your job without a reference to an external source (a ticket, a requirement, a comment in the code, or correspondence) remains a hypothesis. Otherwise, the agent will once again take the answer from the system we are checking, and the loop will close. It may see a pattern that isn’t there, or mistake a historical error for a business rule.

Stage 5. Confirm, and only then delete. Here QA enters the process, and here, its role is irreplaceable. The agent proposes a hypothesis: QA and the business confirm or reject it. A confirmed hypothesis turns an unexplained test into a business invariant or a known bug — a rejected one — into a candidate for fixing the expectation, and sometimes the program itself. A test may be deleted only when all the conditions come together:

  • by the full matrix, it kills only those mutants that other tests kill too;
  • its input belongs to a class of documents already represented in another test being kept;
  • there is no ticket behind it, or the rule from the ticket is protected by another test;
  • during the nightly runs it never once failed alone;
  • QA confirmed the deletion.

How can we tell that things are moving? Not by the mutation score: it shows only the first row of the table above. It is more useful to count the share of tests that have categories and are confirmed by oracle source. When it approaches one hundred percent, the bag of dirt stops being a bag of something: each test knows why it exists.

Overall, the point is neither to keep all three thousand tests nor to delete them all, but to understand what knowledge they contain — which part of it is confirmed, and which checks should remain. Perhaps far fewer tests will remain. Perhaps some of the old scenarios will survive almost unchanged. But that should be the result of analysis, not a side effect of a high metric.

Instead of a Conclusion: So, Carry the Bag Into the New Year?

Let’s return to the plan we all started with: raise the mutation score, find the tests without which it doesn’t drop, and throw them out as junk. From that very plan, the first part survived. An agent that kills mutants is genuinely useful: it finds checks nobody wrote, and does it faster than a human. But it gives exactly one kind of confidence: that the tests notice code changes. The sorting survived too, with amendments — only now more clusters are needed than “duplicate or not”. And that very “background data” I considered noise may turn out to be the rare but useful case. The last part did not survive: throwing tests out only because my metric said they add nothing.

The plan lacked a foothold outside the loop. When one person writes the code, and another (often an agent) writes the tests to kill mutants in that code, the system checks only its own consistency — like the MIU rules, which you can apply forever without ever learning whether MU is reachable. The answer to the question “but should it be this way?” always comes from outside.

Outside means QA and the business. In the process described, they have three concrete roles, and none of them can be hidden inside the system — left to the developer, or delegated to the agent. First: confirming oracles, saying the expected result is correct because the rule requires it, not because the program once produced it. Second: judging the hypotheses about business rules embedded in the tests — accept, reject, or set aside and look for confirmation. Third: giving the final go-ahead to delete background, duplicate, and meaningless tests. This is not the role of a team of nitpickers slowing down the release. It is the only place in the chain where someone answers the question “are we building the right product” rather than “are we building it right”. And that is exactly why, in a world where so much code and so many tests are written by AI, these people’s time is worth more, not less.

So no, carrying the bag into the New Year in its current form is not worth it. But throwing it away is not the answer either. Inside it lie three thousand encounters between the program and reality, and our task is neither to keep them all nor to burn them, but to find out which of them are still valuable to us — not as souvenirs, but as finds.

You can make a program follow its rules flawlessly. But before trusting the result, you need to understand why we consider those rules correct.

References

  1. Hofstadter, D. R. Gödel, Escher, Bach: An Eternal Golden Braid. Basic Books, 1979. The MIU puzzle and “jumping out of the system” (Chapter I).
  2. Gödel, K. “Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I.” Monatshefte für Mathematik und Physik, 38, 1931, pp. 173–198.
  3. DeMillo, R. A., Lipton, R. J., Sayward, F. G. “Hints on Test Data Selection: Help for the Practicing Programmer.” IEEE Computer, 11(4), 1978, pp. 34–41. The competent programmer hypothesis.
  4. Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., Yoo, S. “The Oracle Problem in Software Testing: A Survey.” IEEE Transactions on Software Engineering, 41(5), 2015, pp. 507–525. Classification of test oracles.
  5. Boehm, B. W. “Verifying and Validating Software Requirements and Design Specifications.” IEEE Software, 1(1), 1984, pp. 75–88. “Building the product right” vs. “building the right product”.
  6. Feathers, M. C. Working Effectively with Legacy Code. Prentice Hall, 2004. Characterization tests.
  7. PIT Mutation Testing. Documentation: pitest.org. Mutation operators, fullMutationMatrix option.
  8. JaCoCo Java Code Coverage Library: eclemma.org/jacoco.
Bag Question answering Testing

Opinions expressed by DZone contributors are their own.

Related

  • Do We Test Just to Find Bugs?
  • Why Your QA Engineer Should Be the Most Stubborn Person on the Team
  • AI-Assisted Testing: Real-Life Use Cases vs. Myths
  • Building a New Testing Mindset for AI-Powered Web Apps

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook