You Can't Do CI/CD Without Automated Testing

If you ask most DevOps experts what goes into a Continuous Integration or Continuous Delivery chain, they’ll mention components like CI servers and code repositories. They’re less likely to discuss automated testing tools, despite the fact that automated testing is just as crucial in order to achieve complete CI/CD.

Below, I explain just how important automated testing is for a CI/CD workflow and discuss examples of release automation toolchains that incorporate automated testing effectively.

Automated Testing Gets Short Shrift

Automated testing does not always feature prominently in discussions about CI/CD chains. DevOps engineers tend instead to focus on the integration and deployment parts of the process.

Perhaps this is because DevOps itself gives short shrift to the role of testing and quality assurance. The very term DevOps emphasizes development and IT Ops, and leaves QA out of the picture.

Or maybe it’s because DevOps engineers assume that manual or ad hoc testing is still sufficient. You can’t fault them too much for that. Manual testing was sufficient in most cases before the advent of fully continuous delivery chains.

Why Automated Testing is Essential for CI/CD

The fact is that you can’t do CI or CD today without automated testing.

Why? Consider the following reasons why automated testing is key to enabling successful CI/CD:

• Automated tests mean faster tests. Automation leads to faster results than a human could produce manually. Automation also makes it possible to test in parallel, which speeds the testing process even more. Speed and the avoidance of delays are essential in order to deliver software continuously.
• Automated testing is more consistent. An effective CI/CD pipeline should include as few variables and eccentricities as possible. Consistency is the key to ensuring that software behaves as expected at all stages of the delivery chain. Test automation helps to achieve consistency by mitigating the chance of human error when performing tests.
• Automated testing enables agility. In a healthy CI/CD pipeline, it should be possible to adjust tools and frameworks quickly when needs change or better technology appears. This is part of what it means to be agile. With manual testing, being agile is difficult because test suites have to be rewritten or reconfigured whenever the CI/CD toolchain is updated. For example, if you switch to a new CI server platform, you have to update your manual tests. When you have automated tests, however, most of the configuration is done automatically. As a result, you can migrate to new technologies easily.
• Automated testing helps you get the most out of your team. Your DevOps team’s time is valuable (not to mention expensive—these people don’t work for free, after all). You should make sure that they can spend their time and energy focused on doing things that only humans can do, rather than performing tedious tasks that could be automated, like testing.
• Automated testing works well with small-scale changes. It often makes sense to perform certain types of manual testing (such as usability testing) periodically when a major update is made to an application (such as the release of a new interface). But in a CI/CD pipeline, most updates are small. Manually testing every small-scale update entails a great deal of tedium. Automated testing removes that tedium and allows your team to push out small changes on a continuous basis.

Bricking Automated Testing into CI/CD Toolsets

I noted above that automated testing has not traditionally been a major part of the conversation surrounding CI/CD best practices. There are signs that that is now changing, however. A look at some of the latest commercial CI/CD toolsets and release automation products shows that vendors are now building automated testing into the workflows that they provide to users.

Examples include:

• Bluemix Continuous Delivery, which became generally available late last year.
• Accenture Touchless Testing Platform, which was released in April 2017.
• CD Director, CA Technologies’ CI/CD delivery platform.

For DevOps teams, products like these are a healthy sign. They show that automated testing is now being integrated into enterprise-grade CI/CD tool sets. Vendors are finally realizing that DevOps is about more than just development and deployment. They’re recognizing automated testing as the glue that holds the rest of the DevOps workflow together.