We're super excited to release our first ever DevTestOps Landscape report! This report is based on a survey of the software testing community designed to understand their adoption of modern development practices, and its impact on testing.

You might be wondering: “But how does this survey differ from other DevOps Landscape surveys out there?” Good question.

Unlike other DevOps surveys, this one is focused on the evolving role of the tester. As more organizations make the shift to DevOps and adopt release processes like CI/CD, testing is impacted in so many different ways that we don't fully understand yet. 

So we mapped out the landscape of testing in DevOps to answer questions like:

• How has the rise of DevOps and Continuous Delivery affected testing?

• How much testing is being automated today? Does it matter?

• Can the least and most effective testing practices in DevOps be identified, to benefit teams moving to DevOps?

One of our questions the survey answered was whether or not DevOps is paying off. Here’s a hint: It’s not! We collected a few of our favorite findings below. Read the rest of the report for more findings that demonstrate how DevOps is failing.

Respondents on DevOps Teams Feel the Most Stressed

We expected that the more often teams shipped, the less stress they would feel. It’s reasonable to assume that if releases are incremental, there should be fewer surprises in production, hence, less stress. We ran a cross-tabulation against stress levels and different types of software teams. 

Interestingly, this analysis told a different story. In fact, 21% more modern teams felt higher levels of stress than traditional teams, and 23% more than transitioning teams.

Customer Happiness Levels Aren't Directly Affected by DevOps Processes

The customer happiness levels across all types of teams turned out to be about the same. Though about 10% more modern teams have "amazing" customer happiness levels than traditional teams, they still fall 9% behind in the "pretty good" happiness level range, so the results still remain pretty flat.

Based on our respondent demographic, we know that some are successfully "doing" DevOps (labeled as "modern teams"), but very few are reaching the potential of testing in DevOps — what we call DevTestOps. The difference here is that DevOps puts an emphasis on speed, while DevTestOps puts priority on software quality and customers, at speed.

What criteria are needed to be DevTestOps? Automate all the things, right? Actually, no. Automation is definitely a key to achieving DevTestOps, but automation alone will undo all the benefits it can offer. 

Teams that Automate Most or All of Their Tests Are More Dissatisfied With Their Testing Process

Grouping respondents into "little to no automation" and "most or all tests are automated" revealed that testers are more dissatisfied with their testing process the more they automate. The "Automate all the things!" mentality really doesn't translate well to testing. 

There's so much more that we pulled from the survey, so check out the report and let us know what results surprised you, and which results resonate with you. Leave a comment below or tweet us @mablhq with your thoughts!