More Tests, More Confidence? Test Suites Are Investment Portfolios

Learn how to manage test suites as investment portfolios by measuring evidence value, reducing maintenance costs, and pruning liabilities.

Stelios Manioudakis

CORE ·

Jul. 28, 26 · Analysis

Likes (1)

Comment

Save

1.1K Views

Every test suite starts as an asset. The first hundred tests are almost universally useful. They cover the system's core behaviors, and they catch regressions. They give the team justified confidence that the software is behaving as intended. The team is proud of them. The CI pipeline is fast. The signal-to-noise ratio is high.

Then the suite grows. It grows because the system grows and the coverage targets demand it. It grows because new engineers arrive and write tests in new styles, because incidents trigger the addition of regression tests that reproduce specific bugs, because automated test generation tools produce tests at scale. By the time the suite reaches several thousand tests, the team's relationship with it has changed in ways they may not have fully realized. The CI pipeline takes forty minutes. Flaky tests fail intermittently and are ignored. Large portions of the suite were written for features that have since been refactored or removed. Yet nobody is sure which portions, so nothing is deleted. The suite still passes, technically, but the team has quietly learned not to trust it.

At this point, the suite has become a liability. It is consuming maintenance time without generating proportionate evidence about system quality. The team knows this, dimly, but the path to addressing it is unclear. How do you prune a test suite you do not fully understand? How do you govern a test asset that has accumulated without design?

This article answers such questions. I borrow from investment portfolio management because test suites and investment portfolios share the same fundamental problem: assets that once generated returns can become liabilities through neglect, accumulation, and failure to rebalance. The principles that make a portfolio manageable over time are the same principles that make a test suite manageable over time. Disciplines like classification, return measurement, periodic rebalancing, and governance can translate directly into test suite practices.

The Core Argument

Volume is not quality. A test suite of 10,000 tests that are poorly designed, poorly maintained, and weakly assertive generates less confidence than a suite of 1,000 tests that are precisely targeted, well-maintained, and behaviourally specific.

Test suites become liabilities through three mechanisms: accumulation without governance, flakiness without remediation, and maintenance cost that exceeds evidence value.

The solution is portfolio thinking: classify tests by their return on investment, measure their actual contribution to quality evidence, prune ruthlessly, and govern continuously.

Architectural decisions in test design determine the long-term cost structure of the suite. The wrong architecture, applied at scale, makes the liability irreversible without a full rewrite.

How Test Suites Become Liabilities

The transition from asset to liability is gradual and never announced. It happens through recognizable mechanisms that compound over time. Understanding them is the prerequisite for preventing them.

Mechanism 1: Accumulation Without Governance

Test suites grow continuously and shrink rarely. Code is deleted or refactored constantly — engineers remove features, rename functions, restructure modules. Tests, however, are treated as permanent. The instinct is conservation: who knows what a test is protecting, even if it seems redundant? Deleting a test looks risky. Deleting code does not. This asymmetry produces a suite that expands indefinitely relative to the codebase it is meant to cover.

The result is a suite with substantial dead weight. There are tests for deleted features that now test nothing meaningful. Other tests duplicate each other's coverage region. Many tests' original purpose has been lost. Another important category is the tests whose assertions were already weak when written and have become weaker as the system evolved.

Dead weight tests do not merely occupy space. They consume CI time, they produce noise in failure reports, and they create cognitive load when engineers try to understand what the suite is covering. This is a form of testing technical debt: Accumulated through individually reasonable decisions, expensive in aggregate, and resistant to removal because nobody can confidently identify them without significant analysis.

Mechanism 2: Flakiness Without Remediation

A flaky test is one that passes and fails non-deterministically for the same code. It is not a test that catches intermittent bugs; it's an unreliable test. It depends on timing, on network calls, on shared state, on randomness, or on other environmental conditions that the test does not control.

Flaky tests are one of the most corrosive forces in a test suite, and they act through a specific psychological mechanism. When a test fails intermittently, engineers learn that its failure is not meaningful. They re-run the suite. The test passes. They merge. Over time, the team develops a learned response to failure that generalizes from the specific flaky test to the suite as a whole: failures are probably noise. Re-run and proceed.

This generalization is catastrophic. A team that has learned to distrust its test suite has no test suite, in any meaningful sense. The suite still runs, still produces a pass/fail result, still appears in the CI pipeline. But the team does not act on its failures with the urgency that meaningful test failures require. The suite has become wallpaper.

# A flaky test: the canonical pattern

def test_user_session_expires():

user = create_test_user()

session = login(user)

# Wait for session to expire (30 second timeout in test config)

import time

time.sleep(31)

result = validate_session(session.token)

assert result.valid == False

# This test fails when:

# - The CI runner is under load and sleep(31) completes in 28s of wall clock

# - The session timeout is configured differently in different environments

# - The test database is shared and another test refreshed this session

# - Clock drift between the test process and the session service

# Each failure produces a re-run. Each re-run that passes reinforces the team's belief that the failure was noise.

# Meanwhile: the session expiry logic may be genuinely broken. The flaky test is providing cover for a real defect.

The flaky test's danger is not that it fails. It is that it teaches the team not to care when tests fail.

Mechanism 3: Maintenance Cost That Exceeds Evidence Value

Every test has a maintenance cost. When the code it tests changes, the test must change with it. When the test infrastructure changes, the test may break. When the team's understanding of the system evolves, the test's assertions may become outdated, wrong, or irrelevant. The maintenance cost of a test is a function of its coupling to implementation details, its reliance on shared state and infrastructure, and the specificity of its assertions.

A test with high maintenance cost and high evidence value is a reasonable investment — it is expensive to maintain, but it catches important issues. A test with high maintenance cost and low evidence value is a pure liability — it is expensive to maintain and contributes little to the team's understanding of the system's behavior. The problem is that most test suites contain a large proportion of the latter category, and nobody has ever measured it.

# High maintenance cost, low evidence value

# This test breaks every time the internal implementation changes,

# but it verifies almost nothing about behaviour.

def test_user_service_calls_repository():

mock_repo = MagicMock()

service = UserService(repository=mock_repo)

service.get_user(user_id=42)

# Assert that the internal implementation called the repository

mock_repo.find_by_id.assert_called_once_with(42)

# This test verifies HOW the code works (it calls the repository), not WHAT it does (it returns the correct user for a given ID).

# When UserService is refactored to use a cache before the repository, this test breaks. But nothing about the user-facing behavior changed. The test is coupled to implementation, not to behavior.

# Low maintenance cost, high evidence value

# This test survives implementation changes as long as behavior is preserved.

def test_get_user_returns_correct_user():

user_id = create_test_user(name='Alice', email='[email protected]')

result = user_service.get_user(user_id)

assert result.name == 'Alice'

assert result.email == '[email protected]'

# This test survives: adding a cache, changing the repository implementation, refactoring the service class, changing the internal call structure. It fails only when the behavior changes: when get_user returns wrong data.

# That is exactly when a test should fail.

The difference between testing implementation and testing behavior is the single most important architectural decision in test design, and it determines the long-term maintenance cost of every test in the suite.

Your Test Suite Is An Investment Portfolio

Investment portfolio management rests on a small number of principles that have proven durable across a century of application. Not all assets generate equal returns. Returns degrade over time without active management. Diversification reduces risk. Concentration in low-return assets is a silent tax on the portfolio. Rebalancing is not a one-time activity but an ongoing discipline.

Each of these principles translates directly into test suite management, and this is not a metaphor. It’s a lens that makes obvious the not-so-obvious.

Investment Principle	Test Suite Equivalent
Not all assets generate equal returns	Not all tests generate equal evidence. A unit test that verifies a critical financial calculation returns more quality evidence per unit of maintenance cost than an assertion-free smoke test of the same code.
Returns degrade over time without active management	Tests written for one version of the system may generate less evidence as the system evolves. A test written when a feature was new may be testing a behavioral region that is now low-risk. New high-risk regions have appeared without corresponding tests.
Diversification reduces risk	A test suite that covers only the happy path is concentrated in a low-risk region. Coverage across success paths, error conditions, boundary cases, and failure modes provides a diversification that catches more defects that matter.
Concentration in low-return assets is a silent tax	A suite dominated by implementation-coupled unit tests with weak assertions imposes a maintenance cost that exceeds its evidence value. The tax is paid in CI time, in maintenance effort, and in the false confidence the suite provides.
Rebalancing is an ongoing discipline	Test suites must be actively pruned, reclassified, and restructured as the system and its risk profile evolve. A suite that is not actively managed drifts toward a portfolio of low-return legacy tests surrounding a shrinking core of high-value evidence generators.
Know what you hold and why	Every test in the suite should have a reason to exist that can be stated in terms of the behavior it verifies and the risk it mitigates. Tests whose purpose cannot be stated are candidates for removal.

Viewing your test suite as an investment portfolio is very practical. It provides a vocabulary for conversations about test suite investment that many organizations avoid. A VP of Engineering who would never accept an investment portfolio managed without return measurement, diversification analysis, or rebalancing should apply the same standard to a test suite that consumes significant engineering resources.

Classifying Tests by Return on Investment

The first practical step in portfolio management is classification. Before you can manage a test suite for return, you need to understand what your current suite contains. The following classification system provides the vocabulary for that analysis.

The 4 Test Asset Classes

Borrowing from financial asset classification, test assets can be organized into four classes based on their evidence value and maintenance cost. The classification is not precise — tests exist on a spectrum — but it provides sufficient resolution to make management decisions.

Class	Evidence Value	Maintenance Cost	Management Action
Core Holdings	High	Low-Medium	Protect, expand. These are the suite's most valuable assets. They verify critical behavior, survive implementation changes, fail for the right reasons, and are cheap to maintain relative to the evidence they generate.
Speculative Assets	High	High	Monitor and restructure. These tests verify important behavior but are expensive to maintain — typically because they are coupled to implementation, depend on complex infrastructure, or are end-to-end tests with significant environmental dependencies.
Legacy Positions	Low-Medium	Medium	Review and prune. These tests were once valuable but have drifted from the system. They may be testing deleted features, duplicating other tests, or asserting against behavior that is now low-risk.
Liabilities	Low	High	Remove immediately. These tests generate little evidence and consume significant maintenance resources. Flaky tests, assertion-free tests, and tests that verify implementation rather than behavior at high maintenance cost all fall here.

Measuring Evidence Value

Evidence value is a function of three parameters: the criticality of the behavior the test verifies, the quality of the test's assertions, and the degree to which the test covers a region of the behavior space that is not covered by other tests. It cannot be measured with a single metric, but it can be assessed with a structured set of questions.

# Evidence value assessment questions for any test

# 1. BEHAVIOUR CRITICALITY

# What behavior does this test verify?

# If this behavior were wrong in production, what would the consequence be?

# [Critical / Important / Standard / Trivial]

# 2. ASSERTION QUALITY

# What does this test assert?

# Would this test fail if the behavior were wrong?

# Does it assert the outcome, or the implementation?

# [Strong: verifies correct output / Weak: verifies execution / None: assertion-free]

# 3. COVERAGE UNIQUENESS

# Is this the only test that covers this behavior?

# Or do 5 other tests cover the same region?

# [Unique / Partially duplicated / Fully duplicated]

# 4. ORACLE RELIABILITY

# Is the expected value in the assertion correct?

# Was it calculated carefully, or copied from the implementation?

# [Independent calculation / Copied from output / Unknown]

# Evidence value score (illustrative, not prescriptive):

# Critical behaviour + Strong assertion + Unique coverage + Independent oracle

# -> Core Holding

# Trivial behaviour + Weak assertion + Fully duplicated + Unknown oracle

# -> Liability

Evidence value cannot be automated entirely — the criticality assessment requires human judgment about the system's risk profile. Everything else can be partially automated.

Measuring Maintenance Cost

Maintenance cost is more tractable to measure than evidence value because it leaves traces in version control and CI data. The following signals are reliably available in any modern engineering environment and correlate strongly with actual maintenance burden.

# Maintenance cost signals — extractable from git and CI data

# 1. CHANGE FREQUENCY

# How often has this test file changed in the last 6 months?

# High change frequency = high coupling to implementation details

# git log --follow --format='%ad' --date=short -- tests/test_user_service.py

# | sort | uniq -c -> count of changes per date

# 2. FAILURE RATE IN CI

# What proportion of CI runs did this test fail?

# High failure rate with passing re-runs = flaky

# High failure rate correlated with code changes = potentially useful

# Extractable from CI API (GitHub Actions, Jenkins, CircleCI all expose this)

# 3. TIME TO EXECUTE

# How long does this test take to run?

# Tests that take >5s in a unit test suite are almost always

# inappropriately coupled to I/O or network.

# pytest --durations=0 | sort -k2 -rn | head -20

# 4. DEPENDENCY COMPLEXITY

# How many mocks, fixtures, and setup steps does this test require?

# High setup complexity = high coupling = high maintenance cost

# Count fixture dependencies and mock patches in test file

# 5. LAST MEANINGFUL FAILURE

# When did this test last fail, and was the failure meaningful?

# A test that has not had a meaningful failure in 2 years either covers low-risk behavior or covers behavior

# that is also covered by other tests.

All five signals are extractable without reading test code — they come from CI data and version control history. A maintenance cost dashboard can be built from these signals in a day.

Architectural Decisions That Determine Long-Term ROI

The decisions made when writing tests determine whether the suite will remain manageable at scale or will compound into a liability faster than it can be pruned. The following architectural decisions have the highest long-term impact on test suite ROI.

Architecture Decision 1: Test at the Right Level

The most consequential architectural decision is which level of the system to test at. Tests at different levels have different evidence value, different maintenance costs, and different execution speeds. These characteristics are not fixed — they are determined by the system's architecture and by what the test is trying to verify.

The classical framework — the test pyramid — prescribes many unit tests, fewer integration tests, and fewer still end-to-end tests. This is a sensible heuristic for many systems, but it is applied too dogmatically in most organizations. The right distribution depends on what the system does, where its risk concentrates, and what each level of test is capable of verifying.

# The test level decision and its consequences

# Level: Unit test

# Verifies: individual function behavior in isolation

# Evidence value: high for logic-heavy code; low for glue code and I/O

# Maintenance cost: low if behavior-coupled; high if implementation-coupled

# Execution speed: milliseconds

# What it cannot verify: integration behavior, configuration, infrastructure

# Level: Integration test

# Verifies: behavior of components working together

# Evidence value: high for interface contracts and data flow

# Maintenance cost: medium (depends on infrastructure complexity)

# Execution speed: seconds

# What it cannot verify: full user journey, performance, environment-specific behavior

# Level: End-to-end test

# Verifies: user-facing behavior through the full stack

# Evidence value: high for critical user journeys; low for edge cases

# Maintenance cost: high (UI changes, environment dependency, slow feedback)

# Execution speed: seconds to minutes

# What it cannot verify: the root cause of a failure (too much indirection)

# The portfolio implication:

# Unit tests should dominate in volume because they are cheapest to run and maintain.

# Integration tests should cover interface contracts and data boundaries.

# End-to-end tests should cover critical user journeys ONLY.

# End-to-end tests that cover what unit tests already verify are Liabilities.

Architecture Decision 2: Control Your Dependencies

Tests that depend on external systems — databases, APIs, message queues, file systems — are more expensive to run. They are more prone to flakiness, and more difficult to maintain than tests that control their own dependencies. Dependency architecture in tests is as important as dependency architecture in production code.

# Uncontrolled dependency: test behavior depends on external database state

def test_user_count_is_correct():

# Assumes the test database is in a known state.

# Other tests may have added or deleted users.

# Test order matters. Parallelization breaks this test.

count = user_repository.count()

assert count == 5 # This number means nothing outside a controlled context

# Controlled dependency: test owns its state

def test_user_count_reflects_created_users(db_session):

# db_session fixture creates a clean, isolated database for this test

# and tears it down afterward.

user_repository.create(db_session, email='[email protected]')

count = user_repository.count(db_session)

assert count == 3 # This number is meaningful: we created exactly 3.

# The second test can run in parallel, in any order, in any environment.

# It is deterministic because it controls its own data.

# This is the architectural property that prevents flakiness at scale.

Test isolation is an architectural property, not a testing technique. It must be designed into the test infrastructure, not retrofitted to individual tests.

Architecture Decision 3: Make Failures Diagnostic

A test that fails is only useful if its failure tells you something specific about what went wrong. A test whose failure message is "AssertionError" on a complex object comparison tells you that something is wrong somewhere. A test whose failure message is "Expected discount for GOLD tier to be 10%, got 8% for input price=100.00" tells you exactly where to look.

Diagnostic failures are an architectural choice, made when the test is written, that determines the return on investment of every future failure. A test that fails usefully is worth more than a test that fails mysteriously. This is because the cost of investigating a mysterious failure is substantial and is paid every time the test fails.

# Failure that tells you nothing

def test_order_totals():

orders = [order_service.create(item, qty) for item, qty in test_data]

totals = [o.total for o in orders]

assert totals == [10.00, 25.50, 8.99, 150.00, 3.50]

# On failure: AssertionError: [10.00, 25.50, 9.99, 150.00, 3.50]

# Which order failed? What was the input? What was expected? Unknown.

# Failure that tells you exactly what happened

import pytest

@pytest.mark.parametrize('item,qty,expected_total', [

('widget', 1, 10.00),

('gadget', 3, 25.50),

('doohickey', 1, 8.99),

('thingamajig', 2, 150.00),

('gizmo', 1, 3.50),

])

def test_order_total(item, qty, expected_total):

order = order_service.create(item, qty)

assert order.total == expected_total, (

f'Order total for {qty}x {item}: '

f'expected {expected_total}, got {order.total}'

)

# On failure: AssertionError: Order total for 1x doohickey:

# expected 8.99, got 9.99

# Immediate diagnosis. Zero investigation required.

Diagnostic failure messages are not cosmetic. They are the mechanism by which a failing test pays back its maintenance cost. Invest in them.

Architecture Decision 4: Design for Speed

A test suite that takes forty minutes to run in CI is a suite that will be circumvented. Engineers will merge without waiting for results. Test results will be checked retrospectively, if at all. The feedback loop that makes continuous testing valuable will be broken by the latency of the pipeline.

Speed is an architectural property of the test suite, determined by decisions made when tests are written. The primary driver of suite slowness is not the number of tests — it is the presence of slow tests that could be fast with different architectural choices.

# Speed anti-patterns and their architectural fixes

# Anti-pattern 1: sleeping in tests

# time.sleep(5) # Wait for async operation

# Fix: use explicit synchronisation or event-driven waiting

# result = wait_for(lambda: cache.has_key('result'), timeout=5)

# Anti-pattern 2: hitting real external services

# response = requests.get('https://api.payment-gateway.com/charge')

# Fix: stub at the HTTP boundary, not at the function boundary

# with responses.activate():

# responses.add(POST, 'https://api.payment-gateway.com/charge',

# json={'status': 'success'}, status=200)

# result = payment_service.charge(amount=100)

# Anti-pattern 3: starting a full application server for unit tests

# app = create_app() # Full Django/Flask app with all middleware

# client = TestClient(app)

# Fix: test the function directly; use the test client only for

# integration tests that specifically need the HTTP layer

# Anti-pattern 4: sequential tests that could run in parallel

# Fix: pytest-xdist for parallelization

# pytest -n auto tests/unit/ # runs in parallel across CPU cores

# Prerequisite: tests must be isolated (Architecture Decision 2)

# Speed target:

# Unit test suite: < 2 minutes

# Integration test suite: < 10 minutes

# Full suite including E2E: < 20 minutes

# If any suite exceeds these targets, it needs architectural attention, not a faster CI machine.

Speed targets are architectural constraints, not optimization goals. A suite that exceeds them should be redesigned, not scaled up on more powerful hardware.

Governing the Suite: The Ongoing Discipline

Portfolio rebalancing is a continuous process. A test suite that is audited and pruned once will drift back toward liability-heavy within a few months if governance is not ongoing. A continuous discipline that prevents accumulation from reoccurring involves at least the following basic ingredients.

Test Addition

Every new test added to the suite should pass a brief but explicit review against the classification criteria. This is the test equivalent of a code review. It should take as long as reading the test carefully and asking the four classification questions: what behavior does this verify, how strong are the assertions, does it duplicate existing coverage, and is the expected value independently calculated?

# Test review checklist (applied during code review, not separately)

# 1. WHAT BEHAVIOUR DOES THIS TEST VERIFY?

# If you cannot answer this in one sentence, the test needs revision.

# 2. WHAT IS THE CONSEQUENCE OF THIS BEHAVIOUR BEING WRONG?

# Critical / Important / Standard / Trivial

# If Trivial: question whether the test belongs in the suite at all.

# 3. DOES THIS TEST DUPLICATE EXISTING COVERAGE?

# Search for existing tests that cover the same behavior.

# If duplicate: consolidate rather than add.

# 4. ARE THE ASSERTIONS BEHAVIOUR-COUPLED OR IMPLEMENTATION-COUPLED?

# If implementation-coupled: request revision before merging.

# 5. COULD THIS TEST BECOME FLAKY?

# Does it use time, randomness, network, or shared state?

# If yes: what is the isolation strategy?

# 6. HOW LONG DOES THIS TEST TAKE TO RUN?

# Measure it. If >1 second in a unit test: investigate why.

These six questions add approximately two minutes to a code review. They prevent months of maintenance cost.

The Quarterly Portfolio Review

Once per quarter, the team should conduct a structured review of the test suite's health using the metrics available from CI data and version control. This review is not a full audit — it is a health check that identifies emerging problems.

Quarterly Portfolio Review: Key Metrics

Flakiness rate: proportion of test runs that contain at least one flaky failure. Target: <1%. Action threshold: >3%.

Suite execution time trend: is the suite getting faster or slower? A suite that grows by >10% in execution time per quarter without a proportional increase in behavior coverage is accumulating liability.

Test-to-code ratio by module: modules with very high test counts relative to code size may be over-tested in low-risk areas; modules with very low test counts may be under-tested in high-risk areas.

Change-failure correlation: when tests fail in CI, how often is a real defect found? A high false-failure rate indicates that tests are flaky or poorly specified.

Defect escape analysis: defects that reached production in the last quarter — were they in covered code? Were the covering tests assertive enough to have caught them? This is the ground-truth question for suite effectiveness.

Quarantine queue length: the number of tests currently in the flaky quarantine. This number should decrease each quarter. If it is increasing, flakiness is being generated faster than it is being resolved.

The Deletion Culture

The most important governance practice is one that runs against engineering instinct: the normalization of test deletion. In most engineering cultures, adding tests is virtuous and deleting tests is suspicious. This asymmetry is the primary driver of suite accumulation. It must be explicitly reversed.

Test deletion should be treated as a sign of quality maturity, not of quality regression. A team that deletes tests is a team that understands its suite well enough to identify what is not earning its place. A team that never deletes tests is a team that has lost track of what it holds.

Signs of a Healthy Test Portfolio

The suite execution time is stable or decreasing despite the codebase growing.

Flaky test rate is below 1% and trending downward.

When the suite fails in CI, the team investigates immediately rather than re-running.

Every test in the suite can be described, by any team member, in one sentence: what behavior it verifies and why that behavior matters.

The team deletes tests regularly, without controversy, as part of routine refactoring.

The last production incident was not in code with high test count — and when analyzed, the failure was either in genuinely untested behavior or in a known, accepted risk region.

Wrapping Up

More tests do not produce more confidence. They produce more volume. Whether that volume translates into confidence depends on the quality of the tests. It depends on factors like their assertion strength, their coverage of meaningful behavior, their maintenance cost, and their diagnostic clarity when they fail. Volume without quality produces a test suite that is expensive to maintain and unreliable as a quality signal. It is a liability that grows faster than it can be managed.

The portfolio model provides a framework for managing test suites as assets over time: classify by evidence value and maintenance cost, prune the liabilities, restructure the speculative assets, protect the core holdings, and govern continuously. The architectural decisions that determine long-term ROI must be made when tests are written, because they are difficult and expensive to retrofit.

The discipline this requires is not primarily technical. It is cultural: the normalization of test deletion, the institutionalization of test review alongside code review, and the replacement of volume metrics with evidence quality metrics. Teams that develop this discipline produce smaller, faster, more reliable test suites. They generate higher-quality evidence than the large, slow, noisy suites they replaced.

Test suite unit test

Opinions expressed by DZone contributors are their own.

Related

Trending