CI/CD Implementation: 5 Common Mistakes and How to Avoid Them
CI/CD is one of the most trending practices in software development, but pitfalls of its implementation can become a serious obstacle to its benefits.
Join the DZone community and get the full member experience.Join For Free
Working within the technology industry, you've probably noticed a significant shift in software development approaches to process automation and DevOps practices.
According to the DevOps Trends Survey 2020, 99% of companies using DevOps practices and implementing the CI/CD pipeline have noticed significant improvements, such as faster release cycles and improved software quality. However, according to the same report, 85% of teams have difficulty implementing DevOps practices early on.
We at Techstack started moving our teams to a CI/CD approach about four years ago. We have seen consistent improvements in both teamwork and business results, but at the start, we, like most teams, faced pitfalls and obstacles that made us doubt the need for CI/CD.
Today we have overcome those difficulties and successfully use CI/CD on most projects. Based on our experience, let’s explore the benefits of this approach, as well as the most common mistakes and ways to avoid them.
Why Should You Move to CI/CD?
The CI/CD (Continuous Integration and Continuous Delivery) approach is based on short and fast iterations to minimize errors, speed up the development process, and improve product quality.
The infographic below shows how the CI/CD pipeline works.
Moving to CI/CD, teams get the following benefits:
Increase in release flow reliability, since the human element is excluded, and the verification stages are automated.
The ability to release small pieces of work enables teams to release important things first.
Reduction of pressure on QA teams from frequent releases.
Reduction of the complexity of releases involving the work of multiple teams within the same project. Automation helps to avoid potential conflicts in the multiple teams’ work and provides tools when they arise.
Improvement of the security of rebuilding the release candidate if a ticket blocks the release and needs to be removed.
Improvement in the visibility of all release stops, such as a failed or non-working builds, because an automatic system notifies the right person about the problem at the right time.
But let's be honest: the pros of CI/CD are talked about more than the cons.
Let's figure out what problems an engineering team may face when implementing the CI/CD pipeline.
5 Mistakes in Working With CI/CD and How to Avoid Them
Despite its advantages, CI/CD is a rather complex, multi-step process. Each of these steps may present difficulties and obstacles. Here are the five major errors that teams migrating to CI/CD most often encounter.
1. Building CD on Unstable CI
To build a CD, you need a reliable CI flow that already exists on the project. This way, you will be sure that:
each release unit does not break the system performance;
the process of building the application is very automated and repeatable (that is rebuilding the same code will lead to exactly the same result).
If you are not sure about these points, be ready for the constant stoppage of CD flow and the need for engineers to find and fix problems, such as new errors and crashing builds.
How to avoid: Make sure that all stages of the CI flow are implemented, and the team has confidence in the results of the CI work. For example, code that once compiled and passed tests can be guaranteed to collect and pass tests in the future.
2. The Benefits of Automation Aren’t Worth The Potential Risks
When automating processes, it is critical to take into account the ratio of the cost of automation to the benefits that will be obtained.
Let's look at an example:
We have a project where updates are released every two weeks. To automate it, the QA team needs to spend two months writing autotests (with manual regression, which takes no more than four hours). It’s obvious that automating such a process may never pay off.
Conversely, if the team goal is to constantly work on reducing the time for code delivery, then a CD can significantly save the time of the QA team, as well as guarantee an increase in the application reliability.
How to avoid: Before starting the CI/CD implementation, clearly compare the benefits and costs of this automation.
Here are some questions to help you do this:
How often does this process repeat itself?
How long does it take?
How many people and resources are involved in it?
Will the lack of automation lead to errors in the process?
Why does this process need to be automated right now?
Based on the answers, decide how urgently this process needs to be automated, and whether it needs to be automated at all.
It is important to consider the possibility of iterative CI/CD implementation if the project doesn’t currently need a full CI/CD flow, but the automation of individual stages will help solve urgent problems. In addition, you can automate only part of the product: for example, implement CI/CD on the backend without touching the mobile application, which is the consumer of this backend.
3. Understanding Continuous Deployment as Continuous Delivery
Continuous Deployment is a specific practice, where any change in the codebase should get into production automatically and quickly.
On the one hand, most engineering teams are wary of this, because quick changes may confuse users. On the other hand, they can make sure not to make the CD part of the methodology by not launching changes into production instantly. In fact, they just understand Continuous Deployment as Continuous Delivery.
Continuous Deployment is a separate mechanism, an add-on to continuous delivery that doesn’t replace it. When implementing a CI/CD process, consider Continuous Deployment as a separate feature that may or may not be needed in each specific project.
How to avoid: If you decide to work with Continuous Deployment, prepare your project for this in advance to avoid damaging the user experience. Implement the Feature Flags approach when application updates are hidden from users but remain visible to the product team. This way, you can turn it on and off for certain groups of users at the right time.
4. Unreliable Testing Systems
Reliable tests are the foundations of the CI/CD process. They guarantee the code is working correctly and allow you to go further down the flow release. If there is no trust in automated tests, the team must either duplicate work by conducting manual tests (which devalues the work of writing automated tests) or face numerous errors in the production stage (which will directly harm the product).
How to avoid: Make sure the testing systems you work with are reliable enough. Сheck if they meet two critical requirements:
Tests guarantee sufficient coverage of all functionality: all application modules and all main flows are covered by tests.
The result of each individual test can be trusted: tests do not crash by themselves, and if the test passed, then this part of the functionality was really tested.
5. Lack of Meaningful Dashboards and Metrics
Sometimes, Agile teams receive a metrics list before they’ve even had time to understand what is really useful for them to track and what is not. As a result, important indicators are not tracked at all, and a lot of time is spent on recording metrics that don’t bring any benefit.
Although Agile and CI/CD have different goals, they should help each other. Both approaches are based on the Continuous improvements practice ─ constant analysis of the accomplished, revision, and advancement. The easiest way to implement it is to cover all processes well with metrics and dashboards. The team must be able to see the current state to understand where to go next.
Also, any problems should be identified and resolved at an early stage. This applies equally to Agile and CI/CD. For example, a team working on SCRUM needs to understand their performance and burn rate, as well as understand their Time To Delivery, to see possible bottlenecks during the release flow. If this understanding is not there, then the team won’t know which part of the system is ineffective, thus, won’t focus on improvements.
How to avoid: Make sure the team has metrics that show the success of the CI/CD process. Set up processes to enable the team to see any issues and proactively respond to them.
Outcomes of CI/CD
CI/CD is a reliable methodology that helps teams be more productive while improving product quality and speed of release. Still, the path to deployment at the mouse click will be effective only when the processes are built correctly. Not only can special tools help in this, but also the cultural changes of each team member.
Opinions expressed by DZone contributors are their own.