How To Build an Effective CI/CD Pipeline

This is an article from DZone's 2023 DevOps Trend Report.

For more:

Read the Report

Continuous integration/continuous delivery (CI/CD) pipelines have become an indispensable part of releasing software, but their purpose can often be misunderstood. In many cases, CI/CD pipelines are treated as the antidote to release woes, but in actuality, they are only as effective as the underlying release process that they represent. In this article, we will take a look at a few simple steps to create an effective CI/CD pipeline, including how to capture and streamline an existing release process, and how to transform that process into a lean pipeline. 

Capturing the Release Process

A CI/CD pipeline is not a magic solution to all of our release bottlenecks, and it will provide minimal improvement if the underlying release process is faulty. For software, the release process is the set of steps that a team uses to get code from source code files into a packaged product that can be delivered to a customer. The process will reflect the business needs of each product and the team creating the product. 

While the specifics of a release process will vary — some may require certain security checks while others may need approvals from third parties — nearly all software release processes share a common purpose to: 

• Build and package the source code into a set of artifacts
• Test the artifacts with various levels of scrutiny, including unit, integration, and end-to-end (E2E) tests
• Test the critical workflows of the product from an end user's perspective
• Deploy the artifacts into a production-like environment to smoke test the deployment

Every team that delivers a product to a customer has some release process. This process can vary from "send the artifacts to Jim in an email so he can test them" to very rigid and formalized processes where teams or managers must sign off on the completion of each step in the process. 

Putting It Down on Paper

Despite this variation, the first, most-critical step in developing an effective CI/CD pipeline is capturing the release process. The simplest way to do this is by drawing a set of boxes to capture the steps in the release process and drawing arrows from one step to another to show how the completion of one step initiates the start of another. This drawing does not have to be overly formal; it can be done on a sheet of paper, so long as the currently practiced process is captured. Figure 1 illustrates a simple release process that is common for many products: 

Figure 1: A basic release process - Capturing the steps of the current release process is the first step to creating a pipeline

Speaking the Same Language

Once the current release process has been captured, the next step is to formalize the process. When speaking about a release process, and eventually a CI/CD pipeline, it is important to use a common vernacular or domain language. 

For pipelines, the basic lexicon is: 

• Step – A single action, such as Build, Unit Tests, or Staging, in the release process (i.e., the boxes).
• Stage – A single phase in the release process, containing one or more steps. Generally, stages can be thought of as the sequential columns in a pipeline. For example, Build is contained in the first stage, Unit Test in the second stage, and User Tests and Staging in the fifth stage. When there is only one step in a stage, the terms step and stage are often used synonymously.
• Pipeline – A set of ordered steps.
• Trigger – An event, such as a check-in or commit, that starts a single execution of the pipeline.
• Gate – A manual step that must be completed before all subsequent steps may start. For example, a team or manager may need to sign off on the completion of testing before the product can be deployed. 

A CI/CD pipeline is simply an automated implementation of a formalized release process. Therefore, if we wish to create an effective CI/CD pipeline, it's essential that we optimize our release process first.

Optimizing the Release Process

Since our CI/CD pipelines are a reflection of our release process, one of the best ways to create an effective pipeline is to optimize the release process itself before deriving a pipeline from it. There are three critical optimizations that we can make to a release process that pay dividends toward an effective pipeline: 

1. Streamline the process– We should minimize any bottlenecks or artificial steps that slow down our release process.
   • Remove any unnecessary steps.
   • Minimize the number of steps while fulfilling business needs.
   • Simplify any complex steps.
   • Remove or distribute steps that require a single point-of-contact.
   • Accelerate long-running steps and run them in parallel with other steps.
2. Automate everything– The ideal release process has no manual steps. While this is not always possible, we should automate every step possible.
   • Consider tools and frameworks such as JUnit, Cucumber, Selenium, Docker, and Kubernetes.
   • Capture the process for running each step in a script — i.e., running the build should be as easy as executing build.sh. This ensures there are no magic commands and allows us to run each step on-demand when troubleshooting or replicating the release process.
   • Create portable scripts that can be run anywhere the release process is run. Do not use commands that will only work on specific, special-purpose environments.
   • Version control the scripts, preferably in the same repository as the source code.
3. Shorten the release cycle – We should release our product as often as possible. Even if the end deliverable is not shipped to the customer or user (e.g., we build every day but only release the product to the customer once a week), we should be frequently running our release process. If we currently execute the release process once a day, we should strive to complete it on every commit.

Optimizing the release process ensures that we are building our CI/CD pipeline from a lean and efficient foundation. Any bloat in the release process will be reflected in our pipeline. Optimizing our release process will be iterative and will take continuous effort to ensure that we maintain a lean release process as more steps are added and existing steps become larger and more comprehensive. 

Building the Pipeline

Once we have an optimized release process, we can implement our pipeline. There are three important pieces of advice that we should follow in order to create an effective CI/CD pipeline: 

1. Don't follow fads – There are countless gimmicks and fads fighting for our attention, but it is our professional responsibility to select our tools and technologies based on what is the most effective for our needs. Ubiquity and popularity do not guarantee effectiveness. Currently, options for CI/CD pipeline tools include GitHub Actions, GitLab CI/CD, and Jenkins. This is not a comprehensive list, but it does provide a stable starting point.
2. Maintain simplicity– Each step should ideally run one script with no hard-coded commands in the pipeline configuration. The pipeline configuration should be thought of as glue and should contain as little logic as possible. For example, an ideal GitLab CI/CD configuration (.gitlab-ci.yml) for the release process in Figure 1 would resemble: 
    

   build:
     stage: building
     script:
       - /bin/bash build.sh
   unit-tests:
     stage: unit-testing
     script:
       - /bin/bash run-unit-tests.sh
   integration-tests:
     stage: integration-testing
     script:
       - /bin/bash run-integration-tests.sh
   ...
   deploy:
     stage: deployment
     script:
       - /bin/bash deploy.sh --env production:443 --key ${SOME_KEY}

   This ideal is not always possible, but this should be the goal we strive toward.
3. Gather feedback– Our pipelines should not only produce artifacts, but they should also produce reports. These reports should include:
   • Test reports that show the total, passed, and failed test case counts
   • Reports that gauge the performance of our product under test
   • Reports that show how long the pipeline took to execute — overall and per step
   • Traceability reports that show which commits landed in a build and which tickets — such as Jira or GitHub tickets — are associated with a build  

This feedback allows us to optimize not only our product, but the pipeline that builds it as well. 

By following these tips, we can build an effective pipeline that meets our business needs and provides our users and customers with the greatest value and least amount of friction. 

Conclusion

CI/CD pipelines are not a magic answer to all of our release problems. While they are important tools that can dramatically improve the release of our software, they are only as effective as our underlying release processes. To create effective pipelines, we need to streamline our release processes and be vigilant so that our pipelines stay as simple and as automated as possible. 

Further Reading:

• Continuous Delivery by Jez Humble & David Farley
• Continuous Delivery Patterns and Anti-Patterns by Nicolas Giron & Hicham Bouissoumer
• Continuous Delivery Guide by Martin Fowler
• Continuous Delivery Pipeline 101 by Juni Mukherjee
• ContinuousDelivery.com

This is an article from DZone's 2023 DevOps Trend Report.

For more:

Read the Report