DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Ingesting Public Data That Fails Silently: Patterns From a TypeScript and Cloudflare Workers Pipeline
  • Why Scraping Pipelines Break in Production
  • The Pipeline Failed, But the Infrastructure Changed
  • Engineering Data Observability: Detecting Schema Drift, Data Drift, and Silent Pipeline Failures

Trending

  • Where Should a Transaction End? Designing Consistency Boundaries in Distributed Java Systems
  • How to Migrate From C/C++ to Rust: A Practical Guide for Embedded IoT Systems
  • Kafka vs Flink vs Spark: Do You Really Need Real-Time?
  • Agent Harness, Memory Indexing, and AI Governance in Test Automation

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Testing, Deployment, and Maintenance
  3. DevOps and CI/CD
  4. Why Your CI Pipeline Is Flaky and How to Make It Reliable

Why Your CI Pipeline Is Flaky and How to Make It Reliable

Learn how to identify and fix flaky CI pipelines by addressing timing issues, shared test state, dependencies, caching, and test order.

By 
Sandeep Kumar Khandelwal
·
Nov. 19, 26 · Analysis
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
7 Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

A CI pipeline should give confidence to the developers. The code should be ready to move forward into the next stage when a build passes. When the build fails, then there should be a real problem that’s worth investigating.

Unfortunately, many of the developers and build engineering teams experience something different in real life. The same commit passes once, fails the next time, and then passes again after a retry, which makes investigation complex and puts everyone in a puzzle situation.

The situation explained above is called a flaky CI pipeline.

Teams need to understand where nondeterministic behavior enters the pipeline, identify and fix it.

What Makes a CI Pipeline Flaky?

A CI pipeline is flaky when it produces different results across runs without any change in the environment or variables.

Consider a simple example. A developer pushes a commit, and the pipeline fails. The developer retries the pipeline without changing the code or variables, but this time the Pipeline fails.

The second result immediately raises an important question: what has changed?

If the commit, configuration, and expected dependencies were identical, the pipeline probably contains an uncontrolled variable that needs investigation.

Finding that variable is key to reducing flakiness in pipeline
 builds.

Flaky Tests Are Only Part of the Problem

An integration test may depend on a remote API that occasionally responds slowly. A build may download the latest version of a dependency instead of a pinned version. Two parallel tests may attempt to modify the same database record. A Docker container may start successfully but not be ready when the test suite begins. There could be multiple such examples.

These failures appear in different places in the Pipeline steps, but they have something in common: the pipeline does not fully control its execution environment.

1. Timing and Race Conditions

Modern continuous integration systems carry out workloads in parallel in order to shorten the time taken for builds. Tests are run in parallel, the services start asynchronously, and several jobs may access shared resources.

While this strategy does help to enhance performance, it also causes timing-related failures which are outside our control and are the primary source of flaky behavior.

Consider this simplified test:

YAML
 
start_service()

sleep(10)

run_integration_test()


The test simply assumes that the service will become available within ten seconds. This assumption of the service being ready in 10 seconds can be true most of the time. However, under heavy CI load, the service might require 20 seconds. The application is healthy, but the test fails because the pipeline made an incorrect timing assumption that didn’t work in some cases.

A better and logical approach would be to check whether the dependency is actually ready:

YAML
 
start_service()
wait_until_healthy(timeout=30, interval=1)
run_integration_test()


The difference between the earlier pseudo code and the updated one is important. The first implementation waits for time (10 seconds). While the second approach waits for state.

2. Shared Test State

Tests become difficult to trust when multiple jobs share mutable state.

Common examples include:

  • Shared databases
  • Common test accounts
  • Reused directories
  • Shared object storage buckets
  • Fixed network ports
  • Global environment variables
  • Common queues or topics

Imagine two integration tests running concurrently. If both of the tests update the same user, then the final state depends on execution order.

Where practical, each pipeline execution should operate on isolated resources and clean them up afterward once the task is completed.

3. Uncontrolled Dependencies

A pipeline can become nondeterministic even when the application code does not change.

Suppose a build installs:

YAML
 
some-library >= 2.0


The pipeline succeeds today.

Tomorrow, a newer version 2.4 of the library is released. The same source commit now resolves to a different dependency and fails.

The source code did not change, but the build input did.

Lockfiles and pinned dependency versions reduce this uncertainty:

YAML
 
some-library == 2.3.1


This strategy applies beyond application libraries. Container images, operating system packages, build tools, runtimes, and infrastructure modules can all introduce unforeseen changes.

Reproducible builds require controlling as many of these inputs as reasonably possible to avoid such flaky behavior.

4. Caching Problems

Caching is one of the most effective ways to speed up CI pipelines. However, poorly designed caches can also introduce flaky behavior.

A common pattern is:

Build -> Restore Cache -> Compile/Test -> Update Cache

Problems arise when cache keys are too broad.

For example:

YAML
 
cache-key: dependencies


The above configuration may cause multiple branches or dependency configurations to reuse incompatible content.

A stronger cache key might include relevant inputs:

YAML
 
Cache-key: OS + runtime-version + lockfile-hash


The general rule should be: if changing an input can make cached content invalid, that input should influence cache validity.

Teams should also test pipelines from time to time without caches. If a build works only when a particular historical cache exists, then the pipeline may not be reproducible.

5. Test Order Dependencies

A reliable test suite should ideally produce the same result regardless of execution order.

Suppose two tests behave like this:

Test A:

YAML
 
create_user("demo")


Test B:

YAML
 
delete_user("demo")


If Test A executes first, then everything passes. However, if Test B executes first, then it may fail because the user does not exist.

Parallel execution often exposes these hidden dependencies because test ordering becomes less predictable.

Running tests in randomized order can actually be useful during the development phase because it intentionally exposes test coupling before it becomes a continuous CI problem.

A Practical Reliability Workflow

Teams do not need to fix every flaky component at once.

A more sustainable approach is:

Detect -> Classify -> Measure -> Prioritize -> Fix -> Verify

Start by collecting failure data. Classify failures so test problems are separated from other problems. Then prioritize based on frequency and impact. A test that fails once every six months may be less important to fix than one that causes dozens of retries every day. After implementing a fix, continue measuring the failure pattern. Otherwise, it is difficult to know whether the change actually improved reliability or not.

Reliability Before Speed

Engineering teams naturally want faster pipelines. However, a five-minute pipeline that developers must run twice may be less productive than a seven-minute pipeline that succeeds consistently.

Pipeline optimization should therefore consider both dimensions.

The goal should be fast and trustworthy feedback.

Final Thoughts

A faulty CI pipeline is, in most cases, not due to a single technology or one poorly written test case, but rather the result of uncontrolled variability in the tests themselves. Dependencies, infrastructure, shared state, caching mechanisms, external services, and execution environments can all be sources of the problem.

Although retries can temporarily hide these problems, they will not get rid of them permanently.

A reliable pipeline is achieved only by not treating intermittent failures as random events and instead treating them as measurable engineering signals that need attention and should be fixed.

Most important of all is to keep the developers' trust in the pipeline.

A CI system provides the most value when developers believe its results. When a pipeline turns red, engineers should have a reason to investigate. When it turns green, they should have a reason to trust it. Without this trust, there is very little developer productivity.

Pipeline (software)

Opinions expressed by DZone contributors are their own.

Related

  • Ingesting Public Data That Fails Silently: Patterns From a TypeScript and Cloudflare Workers Pipeline
  • Why Scraping Pipelines Break in Production
  • The Pipeline Failed, But the Infrastructure Changed
  • Engineering Data Observability: Detecting Schema Drift, Data Drift, and Silent Pipeline Failures

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook