I have a simple rule: “I might not automate if I have to create.” However, I have done the opposite. I have automated when others would not, and I have waded in when others would have automated. To answer this question, we need to question our situation and build a deeper model of our actual aims, options, and capabilities.
A: I have a problem providing a simple answer to this question…
And not just because it depends, but also because that’s not the model I use, so I don’t frame the question like that. I don’t think in terms of “manual testing” and “automated testing.”
I think in terms of “testing,” “interacting directly with the system,” “automating predefined tasks, actions, and paths,” “using tools,” etc.
TestingIf I’m testing, then I ask myself the following questions:
- Do tools already exist to help?
- Do I have abstractions that cover those areas?
- Do I have to create tools?
- Do libraries already exist?
Don’t Automate Under These CircumstancesSo, what about scenarios where you might expect “don’t automate” as my default answer? The following are situations in which I wouldn't suggest automating.
- It's something that we rarely need to do.
- It's something that has to do with usability in terms of look and feel.
- It has to do with experimental features.
- It's something that's still under development and that changes frequently.
Experimental features might fit into an existing process, so we might already have a lot of the scaffolding that we need to support minimal and speedy automating, in which case I might automate those.
I have automated features that were changing frequently because we were testing the same path through the application many, many times (every time it changed) and we often found problems (every single release). The effort of testing it by interacting with it was a waste. So, we didn’t interact with it until the automated execution could pass reliably. However, that also depends on what in the implementation changes and how that change impacts the approach chosen to automate it.
What’s the Real Question?If I have to create stuff to help me automate as part of my process, then I’m likely to weigh up the cost of automating versus just getting on and interacting with it.
Are the assumptions behind the scenarios true?
- Scenario: It's something that we rarely need to do.
- Assumption: We'd spend a lot of time automating something that we would rarely use.
Perhaps we rarely execute it because of the time and energy involved in setting up the environment and data rather than the tooling. Perhaps we need to automate more to allow us to test it more frequently.
- Scenario: It has to do with experimental features.
- Assumption: We write code to automate it that we throw away.
- Scenario: It's something that's still under development and that changes frequently.
- Assumption: Automating a scenario always takes longer than interacting with it and maintenance in the face of change takes too long.
Scenarios Have More Nuances in the Real WorldVery often, we discuss these questions as hypothetical scenarios. If we do, we need to drill into that scenario in more detail in order to come up with our answer. One of the main benefits of the exercise of "hypothetically…" is the asking of questions and fleshing out of a model to better explore the scenario.
I have automated when other people said I should not, and it saved time. I have interacted manually using tools when others said we should automate, and we found problems we would never have found if we had taken the automation path so early. I have also manually engaged in stress and performance testing.