Q: When do we prefer manual testing over automated testing?

A: I have a problem providing a simple answer to this question…

And not just because it depends, but also because that’s not the model I use, so I don’t frame the question like that. I don’t think in terms of “manual testing” and “automated testing.”

I think in terms of “testing,” “interacting directly with the system,” “automating predefined tasks, actions, and paths,” “using tools,” etc.

Testing

If I’m testing, then I ask myself the following questions:

• Do tools already exist to help?
• Do I have abstractions that cover those areas?
• Do I have to create tools?
• Do libraries already exist?

I might automate heavily if I already have abstractions. My testing might involve lots of automating if tools already exist. I might not automate if I have to create. When and if I create, the testing has to wait.

Don’t Automate Under These Circumstances

So, what about scenarios where you might expect “don’t automate” as my default answer? The following are situations in which I wouldn't suggest automating.

• It's something that we rarely need to do.
• It's something that has to do with usability in terms of look and feel.
• It has to do with experimental features.
• It's something that's still under development and that changes frequently.

Many people conduct stress and performance testing rarely. They generally automate. Usability and “look and feel” have tools to detect standards non-compliance. I might very well use those. So, I might automate portions of this type of testing early.

Experimental features might fit into an existing process, so we might already have a lot of the scaffolding that we need to support minimal and speedy automating, in which case I might automate those.

I have automated features that were changing frequently because we were testing the same path through the application many, many times (every time it changed) and we often found problems (every single release). The effort of testing it by interacting with it was a waste. So, we didn’t interact with it until the automated execution could pass reliably. However, that also depends on what in the implementation changes and how that change impacts the approach chosen to automate it.

What’s the Real Question?

If I have to create stuff to help me automate as part of my process, then I’m likely to weigh up the cost of automating versus just getting on and interacting with it.

Are the assumptions behind the scenarios true?

• Scenario: It's something that we rarely need to do.
• Assumption: We'd spend a lot of time automating something that we would rarely use.

Perhaps the feature is so experimental that the easiest way to interact with it is by automating it. Otherwise, we have to spend a lot of time building the scaffolding required (GUI) for a user to interact with it.

• Scenario: It's something that has to do with usability in terms of look and feel. 
• Assumption: This is subjective.

Perhaps we rarely execute it because of the time and energy involved in setting up the environment and data rather than the tooling. Perhaps we need to automate more to allow us to test it more frequently.

• Scenario: It has to do with experimental features.
• Assumption: We write code to automate it that we throw away.

Perhaps standards exist to guide us, and if we ignore those and all the existing tooling, then we create something so subjective that no one else can use it.

• Scenario: It's something that's still under development and that changes frequently.
• Assumption: Automating a scenario always takes longer than interacting with it and maintenance in the face of change takes too long.

Perhaps, the GUI changes infrequently and the change takes place in the business rules behind the GUI. So, really, we are dealing with a single path, but lots of data. Additionally, perhaps we can randomize the data and implement some generic model-based rules to check results.

Scenarios Have More Nuances in the Real World

Very often, we discuss these questions as hypothetical scenarios. If we do, we need to drill into that scenario in more detail in order to come up with our answer. One of the main benefits of the exercise of "hypothetically…" is the asking of questions and fleshing out of a model to better explore the scenario.

Sometimes, conducting a contrary experiment will provide you with examples that make it harder for you to answer these scenarios without extensive model building.