Continuous testing is a key element in successfully implementing a DevOps pipeline. Continuous testing, which is often called shift-left testing, is an approach to software and system testing in which testing is performed earlier in the software lifecycle, with the goal of increasing quality, shortening long test cycles and reducing the possibility of software defects making their way into production code. The following V-model (Verification and Validation model) is useful in showing the relationships between each phase of the software development life cycle and its associated phase of testing.
One way that DevOps teams do Continuous Testing is by shifting integration testing to the left of its usual position in the delivery pipeline so that it occurs as close as possible to the build process. Since integration testing is where many disruptive, significant defects are often detected, this allows teams to receive feedback on code quality faster, with more accurate results.
Figure 1: Levels of software testing (V-Model) (Source: Software Testing Lecture).
The goal of most DevOps projects is to automate as many manual processes in the software development process as possible. Some roadblocks in a DevOps pipeline that lead to slow deployment include error-prone manual processes such as handoffs from a development group to a QA group, including ones that require signatures or bureaucratic approval. These kinds of handoffs mean there is a lack of shared ownership of the end product, which is contrary to the basic Agile testing and development methodology that says all members of a cross-functional Agile team are equally responsible for the quality of the product or the success of the project. Because of this, testing on an agile project is done by the whole team, not just designated testers or quality assurance professionals, including team members whose primary expertise may be in programming, business analysis, database or system operation.
Continuous Testing is one of several continuous activities that should take place simultaneously in a DevOps pipeline, including
- Continuous Build.
- Continuous Integration (CI).
- Continuous Delivery (CD).
- Continuous Deployment.
Continuous build or build automation is the first stage in implementing a Continuous Delivery DevOps pipeline. If your developers are practicing test-driven development (TDD), they'll write unit tests for each piece of code they write, even before the code itself is written. An important part of the agile methodology, TDD helps developers think through the desired behavior of each unit of software they're building, including inputs, outputs, and error conditions. New features implemented by developers are then checked into a central code base prior to the software build, which compiles the source code into binary code.
Continuous Integration is a practice where members of a software development team use a version control system and integrate their work frequently to the same location, such as a master branch. Each change is built and verified by means of tests and other verifications in order to detect any integration errors as quickly as possible. With build automation, the software build happens automatically, using tools such as Makefiles or Ant, rather than when a developer manually invokes the complier.
Figure 2: A Continuous Delivery pipeline includes a final release approval step (Image Source).
In the last stage of a Continuous Delivery pipeline, once an application passes all the required tests, it's then released into production. For all intents and purposes, this means releasing every good build to users. A fully automated CD pipeline requires the ability to deploy and release any version of a software application to any environment. Doing this effectively requires infrastructure automation, where environments (machines, network devices, operating systems, middleware, etc.) can be configured and specified in a fully automatable format. This means using cloud resources and virtual infrastructure to setup and manage your deployment process.
Infrastructure as Code (IaC) has become increasingly widespread with the adoption of cloud computing and Infrastructure as a Service (IaaS), which relies on virtual machines and other resources that cloud providers like Amazon Web Services and Microsoft Azure supply on-demand from their large pools of equipment installed in data centers. IaC lets DevOps teams automatically manage and provision IaaS resources using high-level programming languages. In practice, this means developers have the ability to control most or all IaaS resources via API calls, in order to do things like start a server, load balance, or start a Hadoop cluster. Many DevOps shops also let developers write or modify an IaC template to provision and deploy new applications instead of relying on operators or system administrators to manage the operational aspect of their DevOps environment.
Infrastructure as code is a powerful tool to help build an effective Continuous Delivery pipeline. However, it can also lead to deployment chaos if your organization doesn't have an established DevOps culture in place that puts a high value on teamwork and trust, as well as real-time test management software that can instantly pinpoint software defects. Continuous deployment is often confused with continuous delivery, but deployment and delivery in a DevOps context mean different things.
Continuous deployment means that every software change is automatically built, tested, and deployed to production. Continuous delivery means that a DevOps team is capable of deploying and releasing any version of a software application to any environment but may choose not to do it, most likely for business reasons. For example, if your organization is using DevOps to deploy mobile apps to a large number of mobile platforms and you discover that one or more of the platforms is no longer widely used by your customers, internal employees, or business partners, then either a person, an automated test, or a business rule in your CD pipeline can decide in that instance not to do a final release.
The central idea of continuous delivery is to deliver small software changes on an incremental or continuous basis to not only deliver improved software quickly but also in order to reduce the risk associated with introducing new software changes. To reduce the risk that the newly released software that may not be production-ready, Agile teams practicing CD can take advantage of a couple of different deployment strategies, “canary releasing,” or “blue-green deployment” to achieve zero downtime.
The "canary releasing" strategy involves releasing the next version of your software into production, but only exposing it to a small percentage of your user base. After it passes a number of environmental tests, you then release it to more servers in your infrastructure and route more users to it.
The blue-green deployment strategy involves setting up the latest version of your application on an identical clone of your production application stack and then switching traffic from the current production stack to the new one as soon as the application passes all the manual and automated tests in your delivery pipeline. This approach works especially well if you're using cloud resources and virtual infrastructure to setup and manage your CD pipeline.
Using IaC code mentioned earlier, DevOps teams can automate both the deployment and rollback processes to improve on-demand deployments. IaC code also makes it easier to do extensive testing, which promotes better quality control. Among the types of automated tests that can be performed on both canary and blue-green software releases are performance, load, spike and soak tests.
A performance test is a test of the response time of a software system running under a normal load, that is, with the expected number of users. The goal of performance testing is to get a baseline and figure out how an application will behave under normal conditions. For example, does your mobile app meet a required response time such as below 1000 ms (one second)?
Figure 3: Automated tests include performance, load, spike, and soak tests (Image Source).
Load testing involves scaling up the load on the system and then measuring performance as it scales up. Among the kinds of things that a load test measures are response times, throughput rates and resource-utilization levels. A peak load test is designed to measure performance when an application is subjected to more than the usual number of users, like during the busiest time period such as an end of month payroll cut-off.
There is more than one type of load test. One type is a stress test, where you escalate the amount of load over time until you find the limits of the system or breakpoint. This is the point where, for example, if you keep increasing the load on a client-server app, the server cannot handle any more requests and it will crash.
A spike test involves going over and above the maximum design capacity, in order to see how the application can deal with a large spiking load. A spike test is a good one to consider in cases where you are load testing new database changes for a database app. In the event that the spike test tells you the database needs major changes, or even switching databases systems entirely, a DevOps team can then proactively write IaC code to automate the process of setting up and configuring new cloud-based database resources to handle unexpected demand from end users.
Soak testing is done by running a set of tests on a system over an extended period of time – such as a 24 hour period – to see how the system performs. In soak testing, you're often looking for memory leaks that may not show up under functional, performance or stress testing over shorter periods of time. Soak tests are tests that a DevOps team can use to monitor battery usage by a mobile app since it’s important to keep track of battery consumption while running apps on mobile devices.
DevTestOps = DevOps + Continuous Testing
Patrick Dubois, who coined the term DevOps, said in a recent interview (How Dev vs. Ops Became DevOps") that he was inspired to promote increased collaboration between developers and operations because he felt, as a developer, "if you’re really passionate about what you're building...you do feel responsible about how your application is doing in production." In the past decade, DevOps has lead to a cultural shift in the way many organizations deliver software applications.
These days, however, most organizations have realized that DevOps without continuous testing leads to the quicker deployment of buggy software into the production environment, which is why the term DevTestOps is gaining currency as a way to describe the culture of collaboration needed among all of the teams involved in software development (developers, operations, quality assurance, business analysts, management, etc.) in order to speed up the delivery of high-quality software.
Building a successful Continuous Delivery pipeline means reducing the cost, time, and risk of delivering software changes by allowing for more incremental updates to applications in production. As your CD pipeline becomes automated, the obvious next step is to incorporate more quality control steps into the pipeline process, either through test-first approaches — such as test-driven development (TDD), acceptance test-driven development (ATDD), and behavior-driven development (BDD) — or by continually running automated scripts to test the new features.
As your software releases increase in frequency, the suite of automated test scripts — including those that do the performance, load, spike and soak tests described above — can be stored and re-used in future release cycles. These scripts can also be incorporated into test automation frameworks and used to reassure project stakeholders that thorough and adequate testing has been successfully completed.