The Place of Manual Testing in DevOps

With the growing pressure of digital disruption, enterprises find themselves facing the need for speed-and change. DevOps has emerged to enable this change and help speed up time to market and cross-team collaboration. As a result, DevOps has transformed how we test and our daily testing routines.

But what does this mean for manual testing? Have you made the decision to use it or lose it? In his recent webinar, "Reality Check: The Role of Manual Testing in DevOps," Ingo Philipp addressed what the future holds for the role of manual testing and UI test automation, and how teams can achieve their required level of testing without slowing down the acceleration initiative.

The webinar (which you can watch on-demand here!), sparked several follow up questions on the applications and nuances of manual checking and exploratory testing in DevOps. In response, Ingo has gathered the best of the questions and provided his candid answers here.

Is Manual Checking Unacceptable in DevOps?

No, I would never say that it is unacceptable. I am not divorced from the real world! I do believe, however, that we need to reduce the effort of human checking to an absolute minimum in high-speed environments. We need to reduce the effort it takes us (testers) to repeatedly push data through our products and check things manually since it simply slows down the entire delivery process dramatically.

Reduction is the keyword, not disposal - and a natural consequence of that goal must be to automate as many (relevant) checks as fast as possible. Relevant checks are the checks that contribute most to the overall business risk coverage.

Risk-based testing becomes increasingly more important the faster you need to deliver your product since the only decision you can make is how much left-over risk you are willing to accept when you release your software. Without deciding on an acceptable level of risk, you will never finish testing. Never! The time needed for testing is infinitely larger than the time available.

Automating all your checks shouldn't be your goal - you should automate what matters most to your business, and this is usually a much smaller amount of checks than you might think. Keep in mind as well, that some of those relevant checks simply make more sense as manual tasks. There will always be manual checks that will have to be done regularly.

Is DevOps Killing Testing?

There have been a number of publications, talks, and threads on social media platforms lately about the possibility that testing is a "dying profession." Testing is not a dying profession. It's a fantastic question though because every tester should be prepared to have an answer ready (especially these days).

This discussion has been around for ages, simply because testing is accessible to everyone - from a manager, developer, chief executive officer to a young child, who can spot a problem when it occurs. Everybody can test.

This doesn't imply, however, that everybody can also do good testing. It's like saying that everyone who can cook is a master chef, or everyone who can drive a car is a racecar driver, or everyone who can write is a poet. That's obviously not true.

What bothers me most is that the discipline of testing is often reduced to just the ability to spot simple problems, and this then creates the impression that testing is not a skilled activity. Testing is so much more than that, in the same way that developing is so much more than just writing code. Michael Bolton (I believe) put it this way: professional testers reveal problems in a structured, effective and efficient way based on different heuristics, identifying potential risks based on models we create, analyzing actual risks based on experiments we do, and communicating these risks in a helpful way.

Is Exploratory Testing Like Manual Ad-Hoc Testing?

I don't really have a use for the term "ad-hoc," so I don't use it. Ad-hoc testing (nowadays) is often defined as chaotic, random, unstructured, beyond any rules, unbounded, having no goal, plan, or target, and therefore is unmanageable. Some push it even further and state that the limits of ad-hoc testing are that of space and time. These statements might sound impressive at first, but they are of zero practical significance. In this sense, exploratory testing and ad-hoc testing are two very different things.

What Are the Traits or Skills a Tester Should Have or Develop to Succeed in Today's Fast-Paced Environment?

This was nicely summarized by James Bach and Jerry Weinberg in one of their blog posts. They said that a tester needs to be cautious, curious and critical. This is what makes a great tester, and this is what makes a tester hard to fool - and that's crucial since the software product is constantly trying to fool us.

Great exploratory testers need to be able to pose useful questions, describe and understand what they observe, and to interpret and communicate what they find. I think that the ability to communicate your test results and your test approach in a clear way is often undervalued - but it's so important. It really allows others to read your work, and to value your work. As a tester, you should also be able to think critically about what you know, and you should know you will never know everything. That's my main mantra, and I always reflect on that before I start testing.

Another thing you should be able to do is to target specific issues without losing focus, and when you focus on something, you should mainly focus on things you don't know. That sounds hard, and it is, but that is what testing is all about.

When Would You Call a Test an Exploratory Test?

When the learning, the test design, the test execution, and the interpretation of the test results are carried out by the same person at the same time. These actions should be done simultaneously, they shouldn't be separated in time.

But, most importantly, a test becomes more exploratory when the test you do is influenced by the last test you did. Everything that you have learned so far, including the result of the last test, informs you about the next test or provides choices for how to carry on with the next test.

You become more exploratory when you focus on revealing new information, rather than confirming existing knowledge. That is what exploratory means: it's the search for discovery - and this is what distinguishes an exploratory test from a formal (scripted) check.