How to Assess the Value of a Code Review Culture

In manufacturing, getting to an ROI is pretty cut and dry. What are the cost of the materials, how many products will that yield, and what can we sell it for? Calculating the value of a collaborative culture and a strong peer review practice isn’t quite as intuitive.

When bugs slip through the cracks, it isn’t just expensive; it’s a risk to your entire business. The stakes are high, to say the least. Organizations clearly save money if their development teams can identify and fix bugs before they reach the customer’s hands. We can all agree to that, but how much do they save? Do you have to wait for a disaster to get an understanding of the cost of inadequate reviewing? Before getting to an explicit value, let’s first consider the scope.

1. The Review Scope: Aligning on Requirements Early

Before we can assess the value of the review process, it’s worth looking at where it starts and ends. If we say that peer reviews only need to start after there are lines of code to go over, then we are leaving a lot of room for miscommunication. Bad things can happen when a team doesn’t thoroughly discuss the particular requirements and specifications of a project at the start.

Steve Partridge, Director of Solutions Management at PTC, mentioned in our recent webinar:

“As one customer example from our side that we’ve seen, they had one of the worst warranty issues in their industry. They spent a couple of years going through a Six Sigma project. Really detailed analysis of ‘Why are our warranty issues are so bad?’ At the end of it all, it all came back to missed and misunderstood requirements. Just by saving, or by fixing that problem or addressing that problem, they were going to save 20 something million dollars a year.”

From the start, you need collaboration across functions to ensure that everyone is on the same page. Otherwise, you could have a business analyst or requirements author write something that is not testable or too ambiguous. If you let the peer review process drive an Agile, cross-functional approach, then your team culture will naturally become more collaborative. In the past, we’ve written about how peer code review aligns closely and supports Agile tenets. When evaluating the review process, it’s worth treating it like a discipline rather than a task.

2. The Cost of Finding and Fixing Defects

While every type of defect could have a different cost, tying the cost per defect to labor hours required to fix is a great place to start. Once you have that number, the volume of customer-reported defects can easily be inputted into an ROI calculation, multiplied by the cost per defect. For example, Aimetis, a global leader in intelligent video technology, switched from the practice of ad hoc, over-the-shoulder reviews to working with Collaborator as the team’s primary code review tool. As a result of this emphasis on collaborative software improvement, the number of customer-reported defects dropped by 90%. So was it worth it to make this change? Most people would assume yes, but Finance departments don’t run on assumptions. When you can bring a well-reasoned ROI into the discussion, the value of code review can speak for itself.

3. Factoring for Faster Reviews

After you’ve identified the cost per defect and can track changes to the volume of defects that are customer-reported, then the question is, how can you expedite the overall review process to maximize your ROI? The most effective and efficient way to conduct code reviews is through the use of a tool that allows the team to contribute frequently and remotely.

When we surveyed 550 developers earlier this year for The State of Code Review 2017, 60.4% of respondents said that their teams were executing tool-based code reviews. We also found that developers who use a code review tool are four times more likely to review on a daily basis than those using a meeting-based approach. When Heart Test Laboratories, a company developing a low-cost heart disease screening test, adopted Collaborator, they reduced their code review and test review timeline by 70%. If your current code review method is over the shoulder or meeting-based, it is worth evaluating the impact that adopting a tool-based approach could have on your review ROI.

4. Putting it All Together

When individuals have gone to measure the ROI for adopting the Agile methodology, the general framework is actually fairly similar. Cycle time, defect trends, and cost avoidance are the leading factors for both equations. These components all work together in one comprehensive calculation to illustrate the impact of code review on the bottom line. If you think your organization could benefit from tying a financial number to peer reviews, you can try out the ROI calculator that we use for Collaborator.