Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Do You Need to Calculate ROI for Test Automation?

DZone's Guide to

Do You Need to Calculate ROI for Test Automation?

Rather than learning different ways to calculate ROI, consider other factors of test automation and how you identify value.

· DevOps Zone ·
Free Resource

Is the concept of adopting a continuous everything model a daunting task for your fast moving business? Read this whitepaper to break down and understand one of the key pillars of this model in Continuous Governance: The Guardrails for Continuous Everything.

I am often asked about ROI, and I wrote a parody answer. In this post, I explore initial beliefs and then questions to explore and expand my model of ROI.

"How do you calculate ROI of X?"

(Where X is "automation," "this tool," "exploratory testing," "blah blah blah.")

The answer is that I don't.

I wrote a parody blog post "How to Invest in Testing," and one section of that post basically makes fun of ROI calculations. I don't work with tools or decisions that justify an ROI. I don't see a lot of value in it, so for me, it's an easy parody target.

I think people overuse ROI. And I'm cynical in the sense that I think people use it without exploring alternatives.

Bureaucracy loves ROI: process, numbers, calculation, busywork.

Consultancies love ROI: calculators, processes. You can't make a decision.

Check out this web search: Google - Test Automation ROI Calculator. How many test consultancies come up?

ROI is often used for a sales process. Why is that?

First Principles and Beliefs

Rather than getting sucked into ROI, let's look at first principles.

I don't use ROI. Why?

  • I see no value in it for the work I do.
  • I don't have to.
  • I have other ways of making decisions.

We could investigate that.

I See No Value in It for the Work I Do

The costs I work with are almost negligible compared with the costs of the business or the projects. I think ROI works well when deciding if projects are a go/no-go, rather than decisions on projects.

I Don't Have To

This is probably the most important answer. If you are working in an environment and you don't have to do an ROI, then don't. I'm willing to bet that the ROI calculation does not have a numeric representation of all the factors in your environment that are important (see the HS2 ROI Report).

The HS2 ROI Report requires selling the infrastructure to reap a 50% return on investment— not a great return, and it ignores a lot of costs. This post was written in 2011 when the estimated cost was 14bn; in 2018, the current estimated cost is between 60 and 100 billion according to independent news (I can't tell if the ROI has changed in response to the estimated costs).

In the formula you use for ROI, what is your number for:

  • Job satisfaction?
  • Opportunity cost of delay due to using an ROI calculation?
  • Communication mismatch with other team members?
  • etc.

The formula doesn't cover everything that might be important.

If you are working in an environment that forces you to create an ROI, then you'll probably end up having to game the ROI calculation to get the result you want anyway. You'll need to have other ways of making decisions.

I'm not a consultancy that tries to sell big services based on an ROI calculation.

I Have Other Ways of Making Decisions

  1. Make the decision.
  2. Weigh the options, make the decision.
  3. Decide on something, get the budget to implement it.

An evaluation criterion I use is "Can I justify this decision?"

Things that help:

  • Tossing a coin
  • Cost-Benefit/Feature Analysis
  • Comparative Options Analysis

Why toss a coin? If you get a result you don't like, you feel it, and don't always go with what the coin says.

Some people equate Cost-Benefit analysis with ROI. I think they are different.

But rather than try and argue and engage on the level of numbers and ROI, why don't we consider our own positions on ROI and what they are based on.

I'm going to give you an exercise, and you can do this if you are Pro or Against ROI. I've done the exercise, I found it very useful. I'm not going to cover my answer here as it would be too long, but I'll create a blog post and a Patreon video covering it.

Exercise: Ask Yourself Questions About ROI

ROI is often a default reaction. I encourage you to think it through for yourself.

  • Why are you using ROI?
  • What is your definition of ROI? Does that match the rest of the world's?
  • What do you gain by using ROI?
  • What do you lose by using ROI?
  • What would happen if you did not use ROI?
  • How could you make a decision without using ROI?
  • How does using ROI help you make a decision?
  • Do you always use ROI? What was different for the situations when you didn't?
  • What other questions could you ask yourself about ROI?

And if you don't use ROI, try these:

  • Why are you not using ROI?
  • What is your definition of ROI? Does that match the rest of the world's?
  • What do you gain by not using ROI?
  • What do you lose by not using ROI?
  • What would happen if you did use ROI?
  • How could you make a decision using ROI?
  • Do you never use ROI? What was different for the situations when you did?
  • What other questions could you ask yourself about ROI?

I worked through this exercise and found it useful. I will expand on my answers on Patreon, but I hope you perform the exercise yourself, and I'd be interested in what you find out when you do.

Are you looking for greater insight into your software development value stream? Check out this whitepaper: DevOps Performance: The Importance of Measuring Throughput and Stability to see how CloudBees DevOptics can give you the visibility to improve your continuous delivery process.

Topics:
devops ,roi ,automated testing ,test automation ,software testing

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}