Why Your CI Pipeline Is Flaky and How to Make It Reliable
Learn how to identify and fix flaky CI pipelines by addressing timing issues, shared test state, dependencies, caching, and test order.
A CI pipeline should give confidence to the developers. The code should be ready to move forward into the next stage when a build passes. When the build fails, then there should be a real problem that’s worth investigating.
Unfortunately, many of the developers and build engineering teams experience something different in real life. The same commit passes once, fails the next time, and then passes again after a retry, which makes investigation complex and puts everyone in a puzzle situation.
The situation explained above is called a flaky CI pipeline.
Teams need to understand where nondeterministic behavior enters the pipeline, identify and fix it.
What Makes a CI Pipeline Flaky?
A CI pipeline is flaky when it produces different results across runs without any change in the environment or variables.
Consider a simple example. A developer pushes a commit, and the pipeline fails. The developer retries the pipeline without changing the code or variables, but this time the Pipeline fails.
The second result immediately raises an important question: what has changed?
If the commit, configuration, and expected dependencies were identical, the pipeline probably contains an uncontrolled variable that needs investigation.
Finding that variable is key to reducing flakiness in pipeline
builds.
Flaky Tests Are Only Part of the Problem
An integration test may depend on a remote API that occasionally responds slowly. A build may download the latest version of a dependency instead of a pinned version. Two parallel tests may attempt to modify the same database record. A Docker container may start successfully but not be ready when the test suite begins. There could be multiple such examples.
These failures appear in different places in the Pipeline steps, but they have something in common: the pipeline does not fully control its execution environment.
1. Timing and Race Conditions
Modern continuous integration systems carry out workloads in parallel in order to shorten the time taken for builds. Tests are run in parallel, the services start asynchronously, and several jobs may access shared resources.
While this strategy does help to enhance performance, it also causes timing-related failures which are outside our control and are the primary source of flaky behavior.
Consider this simplified test:
start_service()
sleep(10)
run_integration_test()
The test simply assumes that the service will become available within ten seconds. This assumption of the service being ready in 10 seconds can be true most of the time. However, under heavy CI load, the service might require 20 seconds. The application is healthy, but the test fails because the pipeline made an incorrect timing assumption that didn’t work in some cases.
A better and logical approach would be to check whether the dependency is actually ready:
start_service()
wait_until_healthy(timeout=30, interval=1)
run_integration_test()
The difference between the earlier pseudo code and the updated one is important. The first implementation waits for time (10 seconds). While the second approach waits for state.
2. Shared Test State
Tests become difficult to trust when multiple jobs share mutable state.
Common examples include:
- Shared databases
- Common test accounts
- Reused directories
- Shared object storage buckets
- Fixed network ports
- Global environment variables
- Common queues or topics
Imagine two integration tests running concurrently. If both of the tests update the same user, then the final state depends on execution order.
Where practical, each pipeline execution should operate on isolated resources and clean them up afterward once the task is completed.
3. Uncontrolled Dependencies
A pipeline can become nondeterministic even when the application code does not change.
Suppose a build installs:
some-library >= 2.0
The pipeline succeeds today.
Tomorrow, a newer version 2.4 of the library is released. The same source commit now resolves to a different dependency and fails.
The source code did not change, but the build input did.
Lockfiles and pinned dependency versions reduce this uncertainty:
some-library == 2.3.1
This strategy applies beyond application libraries. Container images, operating system packages, build tools, runtimes, and infrastructure modules can all introduce unforeseen changes.
Reproducible builds require controlling as many of these inputs as reasonably possible to avoid such flaky behavior.
4. Caching Problems
Caching is one of the most effective ways to speed up CI pipelines. However, poorly designed caches can also introduce flaky behavior.
A common pattern is:
Build -> Restore Cache -> Compile/Test -> Update Cache
Problems arise when cache keys are too broad.
For example:
cache-key: dependencies
The above configuration may cause multiple branches or dependency configurations to reuse incompatible content.
A stronger cache key might include relevant inputs:
Cache-key: OS + runtime-version + lockfile-hash
The general rule should be: if changing an input can make cached content invalid, that input should influence cache validity.
Teams should also test pipelines from time to time without caches. If a build works only when a particular historical cache exists, then the pipeline may not be reproducible.
5. Test Order Dependencies
A reliable test suite should ideally produce the same result regardless of execution order.
Suppose two tests behave like this:
Test A:
create_user("demo")
Test B:
delete_user("demo")
If Test A executes first, then everything passes. However, if Test B executes first, then it may fail because the user does not exist.
Parallel execution often exposes these hidden dependencies because test ordering becomes less predictable.
Running tests in randomized order can actually be useful during the development phase because it intentionally exposes test coupling before it becomes a continuous CI problem.
A Practical Reliability Workflow
Teams do not need to fix every flaky component at once.
A more sustainable approach is:
Detect -> Classify -> Measure -> Prioritize -> Fix -> Verify
Start by collecting failure data. Classify failures so test problems are separated from other problems. Then prioritize based on frequency and impact. A test that fails once every six months may be less important to fix than one that causes dozens of retries every day. After implementing a fix, continue measuring the failure pattern. Otherwise, it is difficult to know whether the change actually improved reliability or not.
Reliability Before Speed
Engineering teams naturally want faster pipelines. However, a five-minute pipeline that developers must run twice may be less productive than a seven-minute pipeline that succeeds consistently.
Pipeline optimization should therefore consider both dimensions.
The goal should be fast and trustworthy feedback.
Final Thoughts
A faulty CI pipeline is, in most cases, not due to a single technology or one poorly written test case, but rather the result of uncontrolled variability in the tests themselves. Dependencies, infrastructure, shared state, caching mechanisms, external services, and execution environments can all be sources of the problem.
Although retries can temporarily hide these problems, they will not get rid of them permanently.
A reliable pipeline is achieved only by not treating intermittent failures as random events and instead treating them as measurable engineering signals that need attention and should be fixed.
Most important of all is to keep the developers' trust in the pipeline.
A CI system provides the most value when developers believe its results. When a pipeline turns red, engineers should have a reason to investigate. When it turns green, they should have a reason to trust it. Without this trust, there is very little developer productivity.
Opinions expressed by DZone contributors are their own.