Enhancing Testing Efficiency: Transitioning From Traditional Code Coverage to Code Change Coverage

In software development, maintaining high code quality and reliability is crucial for building robust applications. A key metric for gauging testing effectiveness is code coverage, which measures the percentage of code executed during automated tests. While traditional code coverage offers valuable insights, it has limitations. Code change coverage or code diff coverage addresses these challenges by focusing testing efforts on recent changes in the codebase. This targeted approach not only optimizes testing but also enhances the reliability and quality of software.

Challenges of Traditional Code Coverage

• Quantity over quality: Traditional code coverage often prioritizes achieving high percentages across the entire codebase, potentially overlooking critical functionalities and edge cases.
• Maintenance overhead: Maintaining high coverage requires continuous effort in writing, updating, and maintaining tests, which can be overwhelming in rapidly evolving projects.
• False security: High coverage can give a false sense of security, masking under-tested areas where critical bugs may lurk.
• Legacy code: Achieving high code coverage in legacy systems is challenging due to their complexity and lack of modern testing infrastructure.
• Requirement coverage: Ensuring that newly introduced tests cover all aspects of new requirements adequately is difficult.

How Code Change Coverage Helps

Code change or code diff coverage addresses these challenges by targeting recently modified or added code. This approach enhances efficiency and ensures thorough validation before deployment:

• Focused testing: Prioritizes testing where it’s most needed, minimizing redundant tests and optimizing resource use.
• Early issue detection: Promotes proactive bug detection, reducing post-deployment issues.
• CI/CD integration: Seamlessly integrates into pipelines for rigorous testing pre-deployment.
• Requirement coverage: Helps identify coverage gaps related to specific requirements, enabling targeted test additions to enhance overall quality.

Implementing Code Change Coverage

To integrate code change or code diff coverage effectively, you need to transform traditional code coverage reports into code change coverage reports and build a tool incorporating the following steps:

Step 1: Calculate Differences Between Git Branches

Calculate code changes by comparing the target branch with its parent branch, identifying modified line numbers that constitute actual code changes.

Example using the jgit library:

DiffFormatter diffFormatter = new DiffFormatter(out);
diffFormatter.setRepository(repository);
diffFormatter.setContext(0);
List<DiffEntry> entries=null;
if(branchMode)
{
   	entries=getBranchDifference(repository, diffFormatter, branchForComparison);
}
else
{
   	entries=getCommitDifference(repository, diffFormatter);
}
for (DiffEntry entry : entries) {
    diffFormatter.format(diffFormatter.toFileHeader(entry));
    String diffs[] = out.toString().split("\\n");
    String fileName = "";
    LinkedHashMap<String, String> lines = new LinkedHashMap<>();
    for (int i = 0; i < diffs.length; i++) {
        String s = diffs[i];
        if (s.startsWith("+++")) {
            fileName = s.replace("+++", "").replace("b/", "").trim();
        } else if (s.startsWith("@@") && s.endsWith("@@") && s.indexOf("+") > -1) {
            String index = s.substring(s.indexOf("+") + 1).replace("@@", "").trim();
            String ind[] = index.split(",", 2);
            int n = 1;
            if (ind.length == 2) {
                n = Integer.parseInt(ind[1]);
            }
            if (n == 0) {
                continue;
            }
            int startLine = Integer.parseInt(ind[0]);
            for (int j = i + 1; j < diffs.length; j++) {
                s = diffs[j];
                if (s.startsWith("@@") && s.endsWith("@@")) {
                    i = j - 1;
                    break;
                }
                if (s.startsWith("+")) {
                    String t = s.replaceFirst("\\+", "");
                    lines.put(String.valueOf(startLine), t);
                    startLine++;
                }
            }
        }
    }
    
    differences.put(fileName.replace("src/", ""), lines);
    ((ByteArrayOutputStream) out).reset();
}

An additional example for calculating differences between Git branches can be found on Stack Overflow.

Step 2: Process Output of Traditional Code Coverage

Utilize XML or JSON reports from tools like JaCoCo, Cobertura, or pytest-cov. These reports contain covered and uncovered line numbers. Implement a mapping of changed line numbers to determine which lines are covered or not covered.

Example using JaCoCo XML report:

List<Element> packages = xml.getChilds();
for (Element p : packages) {
    if (p instanceof Element && p.getName().equalsIgnoreCase("package")) {
        String packageName = p.getAttributeValue("name");
        List<Element> sourceFiles = xml.getChilds(p);
        for (Element c : sourceFiles) {
            if (c instanceof Element && c.getName().equalsIgnoreCase("sourceFile")) {
                String className = c.getAttributeValue("name");
                List<Element> lines = xml.getChilds(c);
                ArrayList<String> missed = new ArrayList<>();
                ArrayList<Branch> branchList = new ArrayList<>();
                for (Element l : lines) {
                    if (l instanceof Element && l.getName().equalsIgnoreCase("line")) {
                        if (!l.getAttributeValue("mi").equalsIgnoreCase("0")) {
                            missed.add(l.getAttributeValue("nr"));
                        }
                        if (!l.getAttributeValue("mb").equalsIgnoreCase("0") || !l.getAttributeValue("cb").equalsIgnoreCase("0")) {
                            Branch branch = new Branch();
                            branch.lineNumber = l.getAttributeValue("nr");
                            branch.coveredBranches = Integer.parseInt(l.getAttributeValue("cb"));
                            branch.missedBranches = Integer.parseInt(l.getAttributeValue("mb"));
                            if (branch.missedBranches > 0) {
                                branch.missed = true;
                            }
                            branchList.add(branch);
                        }
                    }
                }
                missedLines.put(packageName + "/" + className, missed);
                branchSummary.put(packageName + "/" + className, branchList);
            }
        }
    }
}

Step 3: Create a Detailed Report

Aggregate and analyze data by mapping the Git diff from step 1 and the coverage data from step 2 to generate a human-readable report that accurately reflects code change coverage percentages.

Conclusion

Code change or code diff coverage focuses on validating new or modified code, ensuring high quality and reliability. While traditional coverage is beneficial, code change coverage improves by validating critical changes promptly. Additionally, the tool provides insights into coverage related to specific requirements, epics, or entire releases, aiding in assessing release quality effectively.

In conclusion, adopting code change coverage is a strategic move toward enhancing software quality in a dynamic development environment. By focusing on recent changes, teams can ensure that new features and modifications are thoroughly tested before deployment. The integration of code change coverage into CI/CD pipelines not only improves efficiency but also provides deeper insights into the quality of specific releases. As the software development landscape continues to evolve, embracing such innovative approaches will be key to maintaining a competitive edge and delivering high-quality software solutions.