[DZone Research] Automated Testing and DevOps Processes

This article is part of the Key Research Findings from the 2018 DZone Guide to DevOps: Culture and Process.

Introduction

For this year's DZone Guide to DevOps, we surveyed 549 software professionals to find out their thoughts on various aspects of DevOps. In this article, we focus on the latest trends in DevOps processes, paying close attention to the practice of automated testing. 

Automated Testing

Asking respondents to let us know which tests in their pipeline are currently automated, we saw a slight (mostly insignificant) decline in the results for every single test. Notable drops were in BDD framework tests (21% to 15%) and integration tests (54% to 49%); the decline in automated integration testing is particularly interesting as integration tests are (and have been for our last 3 surveys) the most popular tests to automate. And while year-over-year changes between the 2017 and 2018 surveys may be rather minimal, there are many tests that have become significantly less automated (or perhaps even used) since 2016, including story-level tests (-8%), component/unit tests (-9%), UI tests (-5%), and post-deployment tests (-4%).

However, the benefits of automation also show themselves in our survey results. DevOps practices such as automatic checks to proceed, automated performance testing, automated feature validation, and automated validation of database code correlate with around a 13% increase over average for a 2-hour-or-less mean time to recovery (MTTR) — or 14%, 10%, 16%, and 11%, respectively. The number of tests an organization is able to automate, though, is in part related to the size of the organization: larger organizations are able to automate more tests. While companies under 100 employees showed an average of 2.06 tests automated based on our survey results, companies over 10,000 employees averaged 2.80 tests. While this difference may seem minor, it represents the largest companies being able to automate 6% more of their testing than the smallest, based on our questions on the automation of 11 different possible tests. Taking the top 5 most popular automated tests overall brings this to an 8% increase between small (under 100 employees) companies and large (10,000+ employees) companies.

DevOps Processes

While the adoption of certain DevOps practices and processes has remained flat, the impact of what has been adopted shows pretty clearly. First, a few areas in this discussion did see some growth from last year; security issue detection increased by 8% (29%-37%), supporting the concept of DevSecOps; the practice of creating thorough audit trails increased 6% (16-22%); and the percentage of respondents saying their pipeline involves manual checks to proceed decreased 7% (56%-49%), indicating a shift towards automated alternatives to manual tasks. The benefits gained through these DevOps processes are in line with the benefits from automated testing: performance issue testing, overall visibility to anyone in the organization, thorough audit trails, code quality checks, code coverage checks, and code reviews all showed an MTTR greater than 10% of what respondents without these practices in their SDLC showed, on top of the automated tests and validations mentioned earlier. While none of these processes are inextricably linked to respondents’ beliefs that their organization has achieved Continuous Delivery, respondents using any one of these were, on average, 14% more likely to say that their organization had completely achieved CD.

Conclusion

As in our last article, it appears that the adoption of many DevOps processes has remained stagnant, year-over-year. Of those processes that did grow, such as DevSecOps, this growth could have been spurred on by interests outside of DevOps. The automation of repetitive tasks and increasing the security of software are two processes that will never lack for support in the software community at large. 

This article is part of the Key Research Findings from the 2018 DZone Guide to DevOps: Culture and Process.