The Reality of Testing in DevOps

DevOps has shaken the application development world, having been adopted by countless organizations in pursuit of continuous delivery of valuable software. With continuous delivery demanding continuous vigilance to maintain the delicate balance of delivering quality at speed, it was bound to be disruptive, and testing is very high on the list of most affected divisions. In fact, testing is routinely cited as the number one source of DevOps delays — and defects are still a major issue for 48% of DevOps teams.

Since its inception over the last 13 years, DevOps has bred a wealth of Dev Ops experts. We recently hosted 7 of them on our DevOps Unbound series to discuss the reality of testing in DevOps as part of the monthly roundtable:

The roundtable was standout in the range of perspectives shared. In this blog, we present the key parallels between their contributions, which remain pretty consistent despite the panelists' varied backgrounds. Take Adam operating Agile at scale at Dell, agreeing adamantly on the quality-first principle that Hilary Weaver swears by — even though Hilary works at the other end of the scale spectrum as a Senior Software Engineer in a small team of 8.

DevOps Demands a Quality-First Mindset

Our experts all agree on the fact that successful DevOps adoptions focus on quality delivery from the beginning of the software delivery cycle rather than tack "testing" on at the tail end. This shift works wonders to ensure that testing is no longer the 'whipping boy of DevOps', as Alan puts it.

Hilary urges organizations to focus on quality as a primary goal, recognizing that many leaders slip up by making it a secondary focus and "ship first and worry about quality later." And once you recognize that quality should take the top spot, be prepared to put your money where your mouth is, asserts Abel. Microsoft was prepared to pay the price for quality before few people even knew what a unit test was, putting them in great stead for their consequent digital transformations — including cloud migration across their product suite. Push out a bug into the cloud and it's there for all to see, Adam points out, where quality (or lack thereof) has never been more exposed.

"You Can't Automate Subjectivity"

Whilst DevOps' push for speed of delivery has undoubtedly triggered a wave in test automation, Hilary and Miriam firstly advise against going all-out in automating all of your tests. Don't forget that exploratory and usability testing is crucial in catching unexpected curveballs on the fly; manual testers absolutely still have their place. For the time being at least, your human resources are smarter than your systems for most aspects related to user experience.

Adam echoes Hilary and Miriam by advocating for manual testing to really stress your systems, and catch elements that make sense on paper but not in practice. His powerful reminder that software engineering is not a race creates breathing space to focus on testing that brings value, not merely speed.

Abel shares the reality that "the very sad fact is automated UI tests can be crazy fragile, especially when you start dealing with modern architectures like microservices. The more layers of dependencies you have, the flakier your tests become…to the point where we no longer run full end to end, automated UI tests." In Abel's 7+ years at Microsoft, he's never seen all of their automated UI tests turn green — firm evidence to implement them strategically.

Don't Skip Unit and Functional Testing

Where test automation is implemented, it's unanimous from our experts — their unit and functional testing has increased confidence dramatically. Adam and Abel both use a system of ranking their tests based on their layers of dependency, and Abel shares that their rate of pure unit tests without dependency at Microsoft stands at 28,000 just to check code in.

Adam points out that by shifting left, Dell's DevOps approach has allowed them to try different types of testing early on, instilling faith that they are bettering their test mix with every release. And sure enough, "nominally unit testing and functional testing takes up a significant amount of overall automated testing."

You Have to Look Beyond the Tools

Every successful DevOps program shares effective collaboration and communal ownership, with every person being allocated an appropriate role and taking accountability for their stake in the outcome. Clint at Tricentis observes:

"A lot of the time there's a heavy focus on 'what tool are we going to use to solve this problem in testing?' However, once we understand who we have, what their strengths are, then we can decide what tools or solutions we need to augment our processes."

Adam couldn't agree more, stating that it's all about people and interactions over processes and tools. Not only are the conversations between testers and developers absolutely critical in the DevOps paradigm, but they also need to be with the right people in the teams. Hilary emphasizes the importance of having testing as the sole focus of at least one specialist, speaking as the testing subject matter expert on her team. Don't fall into the trap of having hiring just developers who dedicate, say, 20% of their time to testing, because they won't be sufficiently equipped to solve critical fixes or quality issues.

We're all familiar with the term product roadmap, but have you considered implementing a team roadmap? Mitch suggests having just that to assess how you plan to mature and evolve as a team, working in parallel with your test strategy.

ML and AI in DevOps Tools: We've Only Scratched the Surface

Although AI and test automation are still in their infancy, some of our experts have dipped their toes and are excited about its capabilities. Abel uses Applitools (which integrates with Tricentis Tosca and qTest), and after his initial skepticism now thinks it's "freaking awesome," with rave reviews from Hillary too. They both agree that its visual regression capabilities are intelligent, where pixel-by-pixel comparison tools have routinely fallen short.

Clint remarks that what we're seeing is the evolution of making testing as easy as possible, one of the key incentives behind Tricentis AI and ML solutions such as Vision AI and LiveCompare.

Side note: Dave Colwell, VP of AI and ML at Tricentis, echoes this, recently sharing on the Transformation in 10 podcast that what underpins future releases is the goal to make it easier for testers to do their jobs.

Hilary appreciates that all 7 experts are very lucky in that they are all coming from organizations that have truly invested time and effort in test automation, which makes this discussion quite remarkable in its array of standpoints of the reality of testing on DevOps. They share their struggles as well as their successes, so it's worth checking out if only not to make the same — often very expensive — mistakes.