Hard Facts About the ROI of Software Testing

“How much money will this make?” seems, on it’s face, to be a reasonable question.

Take your project ideas, figure out the return on investment (ROI), sort from high to low, and you’ve got a start on the priorities list for the company. Plus the the formula for ROI is simple, with just two variables, Value (V) and Cost (C):

(V – C)

———————

That’s it. A 6% per year bond has an ROI of 6% per year. The time-horizon matters – if a project saves us a dollar a day for the next hundred years, the ROI will look different than and ROI for the next twelve months, and there are ways to discount the value of future money. Most companies use a planning window, typically three to five years, to look at ROI, with slower to change companies using a wider window and startups thinking in terms of terms of months to perhaps one year.

So how do you calculate the ROI of testing?

Let’s tackle that from a different angle, looking at ROI in another hard to calculate industry.

The ROI of Insurance

Like testing, insurance is a product that you purchase to cover risk then hope you don’t really need, or at least not much. To cover the risk, the insurance company pools the money, takes a bit of a management fee out, then passes funds out to people based on need.

The administration fee means the average ROI – the ROI if you had a big enough group – is negative.

And yet people still buy insurance.

That’s because for any individual person, the risk is simply too great. The very chance of value makes the decision rational, even if the ROI does not. Think about that for a bit.

As much as this makes me reluctant to talk about the return on investment for testing, I also have to admit, that opening question about value seems reasonable. Allow me to suggest a different way to look at it – Economic Value Add, or EVA.

Economic Value Add

In his book Principles of Product Development Flow, Don Reinsertsen suggested that software has a sale price that can be calculated. This might be easier to do with commercial software, but still, we can come up with some approximation of value for the software. Once we know that, we can look at the sale price of the software before, then after, each step to come up with economic value add.

EVA = ValueAfterStep - (ValueBeforeStep+Costs)

If the first version of the software handed to your testers is a total mess that can’t possible ship – if you get 404 errors and login errors all over the place, then congratulations. The test/fix process is adding some huge percentage of value to the process, in the 90-95% range. You could plug those numbers in to the formula find the ROI of testing. I’d advise caution, due to the insurance problem, but you could.

The problem with this kind of thinking is all the extra time embedded in those test/fix/retest cycles. It’s artificial; it doesn’t need to be there. It also adds costs. Most of the teams I work with today are trying to drive first-time quality – trying to get code to work better the first time. This drives EVA down — unless you get testing involved in the whole process, in asking questions and providing insight, to reduce those problems. That doesn’t decrease the EVA as much as it spreads it out.

The Value of Information

Testing can provide a great deal more than bug reports; it can provide insight and information about the software. That means new feature ideas, usability ideas, ways to combine features to add power for the users. Testing can act as a second line of support, sure, but in our role as advisors and experts we can also provide insights that are fed back into the product and drive down support costs.

All of that may be hard to measure, but it is the tip of the iceberg on how testing adds value.

Calculating the ROI of testing might be hard, but finding the value?

I don’t think so.