Critically Evaluating Pros and Cons: ''But What If?''
Critically Evaluating Pros and Cons: ''But What If?''
Any time you see a list of pros and cons on the internet and believe its merits, ask yourself why.
Join the DZone community and get the full member experience.Join For Free
I see a lot of "pro and con" posts and questions on the internet. They often lack any context behind why the pros and cons are identified as such, so I find it hard to use them directly.
Pro/Con Articles Are Easy to Write and Plentiful
Pro/con articles are plentiful. And some companies make a business out of selling you reports filled with pros and cons, or approaches and tools to help you identify the "best" option, without undue effort on your part.
- You might see some benefits that you didn’t know about.
- You might be more interested in a tool/approach/technique that you had previously ignored.
The "pros" and "cons" tend to be based on unstated assumption about the environment that may not apply to you. They are based on someone else’s experience, and potentially a different environment.
- You might not have that experience.
- You might not work in that environment.
As such, they might be misleading.
Question: do you read them with this critical view in mind?
An Exercise I Conducted: What Are the Pros and Cons of Test Automation?
As an exercise, I executed a web search for "pros and cons of test automation" and evaluated the results. I’m not mentioning which search engine I used or which sites I paraphrased the "pro" below from.
Pro: Test automation is useful for repetitive tasks that are boring or are susceptible to human error.
I ask myself, what are the assumptions?
- Assumption: repetitive tasks are boring.
- Assumption: repetitive tasks are susceptible to human error.
And then challenge the assumptions.
But What If?
But what if it wasn’t boring?
- Perhaps you should make the task non-boring first before leaping to automating?
- What makes the task boring?
- Are you repeating the task with exactly the same data? Perhaps you should vary the data?
- What is the output of the task: data or information? Does it always show the same result? Or do you have to work to interpret the output?
- Do you care about the output? If not, perhaps you should stop?
- Does it always provide the same output? E.g., if it is a "test," has it ever failed? Perhaps the risk it is testing for now has a very low likelihood?
But what if it wasn’t susceptible to human error?
- Does that mean automation isn’t useful?
- How will automation make it less susceptible to human error? Won’t that make things harder to automate?
But what makes it susceptible to human error?
- Perhaps you should make it not susceptible to human error, and identify what makes it error-prone.
- Perhaps you could automate some of the task, rather than all of the task? E.g., data setup.
- Is it the task that is error-prone, or is it the variation in the task that exposes an error in the system — such that when it is automated and executed consistently, the error goes away?
- E.g., people type at different speeds. When a slow typist uses this, the system times out. This makes it susceptible to human error. Automating it removes the human error and hides the fact that the system is unusable to slow typists.
- Perhaps the system needs a fix?
- Perhaps you should train your staff to avoid errors?
- Perhaps the system is hard to use?
- Perhaps you need to rotate staff to different teams so that things are not "boring" but are "new?"
There are environmental nuances that the pro does not elaborate on. If you experience the symptoms that are part of those nuances, perhaps you should fix the cause.
And I could go on:
- Identify "But what if?" scenarios that invalidate, challenge, or question the pro or con.
- Find more pros and cons.
But I won’t, because then this would be too long. It would become a post about “Pros and Cons of Test Automation,” and I intended to explore the “Pros and Cons of Pros and Cons.”
I think it is a useful exercise.
Critical Evaluation of Pros/Cons Lists
Points I want to draw out:
- Their environment is likely differs from yours, but their pros and cons may not help you identify the differences.
- You might not have the skills, approach, or environment to make their "pros" work.
- You might be able to find a better solution than theirs that better matches your environment.
- Treat any list of pros and cons as "possibilites" rather than "absolutes."
- Ask questions of the person creating the pros and cons to see if the underlying environment matches yours.
Exercise: perform a Google search for “pros and cons of test automation” and evaluate the content of the results.
Or choose your own search term. I don’t mind. I’ve done the exercise.
Does It Need to Be “But... What If?”
I don’t just have to ask “But… what if?”
That would be overly prescriptive, and it wouldn't take into account the actual wording or context of the pros and cons.
The point is to challenge them.
- When would this pro/con be valid/invalid?
- What needs to be true for this to be valid/invalid?
- What assumptions are encoded here?
- Is there an agenda behind these pros and cons?
- Am I seeing a description of the only experience this person has?
- i.e., are they recommending a tool because it is the tool they use?
- Did they actually try other tools and approaches?
- Did they have the skills and experience to make those other tools/approaches work?
PACES: Pro And Con Evaluation System
I might summarize this as:
- Identify assumptions.
- Identify bias.
- Challenge the assumptions.
- Challenge the statements made.
I might refer to this as “Pro And Con Evaluation System,” i.e., PACES.
But I won’t. And neither should you.
Or should you? Challenge that assumption.
Published at DZone with permission of Alan Richardson , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.