How to Calculate ROI for Test Automation

Test Automation is attractive, necessary, and expensive too. The decision in front of the test manager is whether to invest in test automation tools or continue with manual testing. In this article, we will look at a couple of ways of analyzing returns on test automation.

The most basic form of ROI calculation is Simple ROI calculation; however, in this article, we will discuss slightly advanced techniques–Efficiency ROI calculation and Risk Reduction ROI calculation.

Efficiency ROI Calculation

The calculation needs to be done in terms of days since automated tests can be run for 24 hours continuously; as opposed to 8 hours that is considered for manual testing. However, 18 hours seems more reasonable since sometimes the test cases are interrupted and do not run for 24 hours.

The formula, which forms the basis of ROI calculation, is:

(a) Automated test script development time = (Hourly automation time per test * Number of automated test cases) / 8

(b) Automated test script execution time = (Automated test execution time per test * Number of automated test cases*Period of ROI) / 18

(c)Automated test analysis time = (Test Analysis time * Period of ROI) / 8

(d) Automated test maintenance time = (Maintenance time * Period of ROI) / 8

(e) Manual Execution Time = (Manual test execution time * Number of manual test cases * Period of ROI) / 8

Notes: Period of ROI is the number of weeks for which ROI is to be calculated. Divided by 8 is done wherever manual effort is needed. Divided by 18 is done wherever automation is done.

In this method, time investment gains are considered rather than monetary gains. Once gains and investment costs are calculated, we can insert the values of these variables into the formula and get the total Efficiency ROI.

In this method, the focus is on actual efficiency of automation and not just the money factor. This method does not require any sensitive information such as the hourly rate of tester, etc. However, this method makes several assumptions such as automated test cases fully replace the manual test cases (but, in reality, some cases automated test cases may need manual intervention) and that the manual testing counts only one manual tester.

Risk Reduction ROI

In this method, automation benefits are independently calculated thus addressing the ROI concerns that were not addressed in the earlier approach. When automation is adopted, resources can be freed to do more productive tasks such as time in-depth analysis of application, exclusive random and negative testing, test design/development etc.

In all, when automation is adopted increased coverage can be attained. However, when manual testing is solely relied upon, it can lead to undiscovered application bugs and post delivery bug fixes. Because of this, the quality of testing and the product is reduced, thus putting the organization at a risk of having to pay a penalty or undergo some loss. This loss needs to be considered in the formula so as to calculate the gain in terms of money if a bug were discovered post delivery/implementation.

Thus, investment cost remains the same, but gain is the monetary loss that the company may have to face if automation is not implemented. These values are then to inserted in the same formula which will then give the ROI.The advantage of this method is that the problems in the above method are addressed and it deals with the positive effect of test coverage. However, there is some subjectivity in this method and high level of risk analysis is required along with difficulty in calculating the loss of not doing test automation.

Conclusion

Since there are several factors that affect the ROI calculation, it is advisable to introduce improvements to the overall automation testing by introducing efficient tools, investing more time in the initial phase itself, using better reporting mechanisms, etc.

Also, there is no single accurate mechanism to calculate ROI. It is up to the tester’s/management’s discretion to use the method that suits their needs and situation the best.