DORA Metrics Assume Your CI Pipeline Is Telling the Truth. What If It Is Not?

The DORA framework is built on a premise so foundational that it rarely gets examined. When the research team behind it studied thousands of engineering organizations and identified the metrics that predict software delivery performance, they were measuring what pipelines report. Deployment frequency from deployment logs. Lead time from commit timestamps. Change failure rate from incident records correlated with deployment events. Failed deployment recovery time from incident resolution timestamps. Deployment rework rate from the proportion of deployments consumed by fixing previously shipped work.

Every one of these metrics is a faithful record of what the pipeline said happened. None of them have a mechanism to assess whether what the pipeline said happened was accurate.

This is not a criticism of DORA metrics. The framework measures what it was designed to measure. The problem is the assumption that sits underneath the measurement: that a pipeline reporting green is reporting something true about the system it is validating.

In a significant number of engineering organizations, this assumption is not holding.

What the Pipeline Is Actually Reporting

A CI pipeline produces results based on what it is configured to check. Unit tests check whether individual functions behave as their developers specified. Integration tests verify that services interact as expected when the tests were written. The pipeline aggregates these results and reports a pass or fail.

The pass or fail tells you whether the code being deployed is consistent with the assumptions encoded in the tests. It does not tell you whether those assumptions are still accurate.

This distinction is invisible in normal operations. When a pipeline passes, it looks the same regardless of whether the tests are checking against current system behavior or against a snapshot of how the system behaved six months ago. The green is green either way.

The DORA metrics that depend on pipeline accuracy are the ones most directly affected by this distinction. Change failure rate measures the percentage of deployments that cause an incident requiring remediation. If tests are passing against stale assumptions about how downstream services behave, changes that will cause production incidents are getting through the pipeline and being counted as successful deployments- right up until the incident that reclassifies them. The change failure rate reflects what the pipeline caught. It does not reflect what the pipeline missed because it was not checking against current reality.

Deployment rework rate, the fifth DORA metric introduced in the 2024 research, captures the proportion of deployment activity consumed by fixing previously shipped work. It is the metric that most directly surfaces when change failure rate is underreported. When teams notice their deployment rework rate is high while their change failure rate appears controlled, the gap between those two numbers is often a signal that failures are occurring after the immediate post-deployment window- in the period when stale test coverage cannot catch them but real usage can.

"A note on reliability: the 2022 State of DevOps Report introduced reliability as an additional outcome dimension assessed through SLOs and SLIs. It is tracked separately from the five core delivery metrics rather than as part of the primary measurement framework. The five metrics that form the core DORA framework are the four original metrics plus deployment rework rate.

Where the Staleness Comes From

The mechanism that causes test coverage to drift from reality is well understood by any team that has run distributed systems at scale. It is less well understood as a systematic source of inaccuracy in DORA metrics.

In systems where services deploy independently on their own schedules, the mock files and integration assumptions that tests run against were accurate when they were written and become less accurate with every deployment of a downstream service. A service changes its error response format. A dependency updates its authentication behavior. An upstream API adds a required field that existing integrations do not send. Each of these changes is correct from the deploying service's perspective. Each of them potentially invalidates assumptions in the test suites of services that depend on them.

The consuming service's tests keep passing because they are not running against the updated service. They are running against a mock that reflects the updated service's behavior as of the last time someone thought to update it, which may have been before several of the changes that have since occurred.

In this state, the pipeline is not checking whether the deployment is compatible with the current system. It is checking whether the deployment is compatible with a historical snapshot of the system. The two checks are not the same, and DORA metrics cannot distinguish between them.

What This Does to Each Metric

The effect on individual DORA metrics is specific enough to be worth tracing through.

Deployment frequency is the metric least affected by pipeline accuracy. It measures how often deployments happen, which is an observable fact regardless of whether those deployments were properly validated.

Lead time for changes is similarly unaffected. The time from commit to production is a timestamp measurement that does not depend on the accuracy of what was validated during that time.

Change failure rate is where the inaccuracy concentrates most visibly. A deployment that passes a pipeline running against stale assumptions but causes a production incident days later gets counted as a failure. However, deployments that cause subtle degradation- service interactions that are slightly wrong, error conditions that are handled incorrectly because the error format changed- may not generate an incident that gets correlated with the deployment at all. They surface as unexplained production issues or as elevated user error rates that get investigated independently. These do not enter the change failure rate calculation. The metric underreports the actual failure rate in proportion to how far the test coverage has drifted from current system behavior.

Failed deployment recovery time measures recovery after incidents are declared. If incidents that originate in stale test coverage are not recognized as deployment-related, they also do not enter this calculation. The metric stays clean while the underlying reliability erodes.

Deployment rework rate is the most sensitive indicator of this pattern. It catches the work that the other metrics do not- the fixes deployed in response to issues that passed the pipeline, the hotfixes for behavior that tests did not catch, the rollbacks for failures that only manifested under real usage patterns the test suite never encountered. When the deployment rework rate rises while the other four metrics remain stable, the most common explanation is that the change failure rate measures a subset of actual failures rather than the whole.

What Accurate Pipelines Actually Look Like

A pipeline that is genuinely telling the truth about deployment safety shares a property that is straightforward to describe and requires deliberate investment to achieve: the assumptions it validates against reflect how the system currently behaves.

For unit tests, this is relatively automatic. The code is the specification, and the tests validate against the code. When the code changes, the tests break and require updating. The feedback loop is tight.

For integration tests that span service boundaries, this is structurally harder. The specification is a mock file that someone wrote to represent a downstream service's behavior. The downstream service continues to change on its own schedule. The mock does not update automatically. The gap between specification and reality accumulates silently.

The teams whose pipelines are telling the truth about integration behavior have addressed this gap architecturally rather than through process discipline. Instead of maintaining static specifications of how downstream services should behave, they derive their integration test coverage from observed real behavior. When a downstream service changes, new observations automatically update what the integration tests run against. The coverage stays grounded in how services currently communicate rather than in how they communicated when someone last thought to update a mock file.

This is the approach modern tools like Keploy take for API-driven systems. Rather than asking engineering teams to maintain mock files that represent downstream service behavior, it captures real traffic between services and generates test cases and dependency mocks from those actual interactions. When a downstream service changes its behavior after a deployment, the next round of captured traffic reflects that change. The pipeline validation running against Keploy-generated coverage is validating against current reality rather than against a historical specification. The change failure rate it contributes to reflects actual deployment safety rather than deployment safety as measured against assumptions that may have become outdated between the time they were written and the time the deployment ran.

The distinction matters for DORA metrics specifically because DORA metrics are only meaningful relative to the accuracy of the pipeline they are measuring. Deployment frequency, lead time, change failure rate, failed deployment recovery time, and deployment rework rate are all accurate representations of delivery performance when the pipeline is checking current system behavior. They are flattering but incomplete representations of delivery performance when the pipeline is checking historical assumptions.

The Metric That Tells You Which Situation You Are In

Teams that want to assess whether their DORA metrics are reflecting reality rather than pipeline assumptions have a relatively direct way to check.

Compare change failure rate against deployment rework rate over a rolling window. If change failure rate is low and deployment rework rate is also low, the pipeline is catching problems before they become incidents, and the rework burden is proportionally small. This is the pipeline pattern that accurately assesses deployment safety.

If change failure rate is low but deployment rework rate is elevated, the gap between them is the most direct signal available that the pipeline is missing failures that real usage is finding. The deployments look clean by the metric that catches immediate post-deployment failures. The rework burden reveals that the failures are occurring on a delayed timeline that the pipeline was not designed to detect.

The action this pattern calls for is not optimizing the four deployment metrics individually. It is examining where the pipeline's coverage is making assumptions that the system has since violated. The improvements to DORA metrics that come from closing that gap are not optimizations of the measurement. They are improvements to the actual delivery performance the metrics are supposed to be measuring.

DORA metrics are a reliable indicator of delivery performance when the pipeline they are measuring is reliable. Making the pipeline reliable requires more than fast feedback loops and automated deployments. It requires that the feedback the pipeline provides reflects the system as it currently exists rather than as it existed when the tests were written.

That is the assumption worth examining before concluding that a green pipeline is telling the truth.