5 Essential Test Metrics Every Company Can Use
Speed, impact, and flakiness are only a few of the software testing metrics you should be using. See what the rest are and why you should use them.
Join the DZone community and get the full member experience.Join For Free
Test metrics are essential in creating reports that accurately measure the success of your testing initiatives. With 23% or more of IT budgets being spent on testing, it is critical to know that your test suites are giving you what you think they are. Because of this, most companies have put some sort of test tracking in place, however, most companies struggle to find the right test metrics to ensure executive buy-in.
Before diving into what test metrics organizations should consider tracking to benchmark their testing efforts, it’s best to give thought to the overall test reporting strategy. Some basic questions to be answered should include:
- Who is the audience for test metrics? It is essential to target the metrics and level of detail based on business line, function (i.e. technical vs. business), and seniority.
- What is the best tool for reporting test metrics? Take into account the audience you are reaching and the intended action from the reports generated.
- How often and when should I send out test metrics? Ideally, information is available to be viewed any time, but assuming work needs to be done, how often and when should reports be distributed?
You may also enjoy: 5 Questions to Help You Determine If You Need Better Software Test Metrics
Once you’ve determined the overall test reporting strategy, the next step is to define the metrics to be covered in your regular reports. While there are seemingly unlimited options, here are some of our favorites.
- What is it? How long your automated test takes to execute.
- Why do you need to track it? Faster running tests allows for more frequent testing and correlates with higher quality code.
- How to measure it? Simple: with a timer that starts at the beginning of each test run and finishes as soon as the last test completes.
- How to improve it? Smaller, more atomic tests that can run in parallel, and the help of a test parallelization solution like Selenium Grid (self-hosted) or Sauce Labs (cloud).
- What is it? How many tests change in status (pass to fail, or vice versa) from one execution to the next, without any change to the functionality that is being tested.
- Why do you need to track it? Flaky tests = more test maintenance and less confidence in automated test results, both of which can kill automated testing initiatives.
- How to measure it? By tracking test history, so you can single out tests with a changing status from one test run to the next.
- How to improve it? Several strategies – from automating test set up, to formalizing and stabilizing test environments, using more robust test selectors, and more.
- What is it? How much of the application (functions, usage, code, etc.) is covered by tests in the test suite.
- Why do you need to track it? To ensure that new application functionality is properly covered by the test suite, so new tests are created to cover appropriate areas.
- How to measure it? This can be the most difficult thing to measure – requiring linking tests to requirements, or tests to code branches.
- How to improve it? By making sure to build new tests which cover untested areas of the application, and that there are enough testers to keep pace with new functionality.
- What is it? The ratio of how many defects are new vs. recurrence of known or existing defects.
- Why do you need to track it? Make sure those filing defects search an existing repository of defects before logging them, and consolidate defects into one set of linked issues.
- How to measure it? How many defects found are new (not linked to an existing defect) vs. all defects found in a test cycle/sprint.
- How to improve it? Through a focus on robust tests which highlight new errors, and expanded focus on ancillary exploratory testing to find new potential issues.
- What is it? The importance of a certain defect regarding the business process, implying its criticality (even if it’s “severity” might only be cosmetic).
- Why do you need to track it? To make sure that defects that have the most impact to conducting business as usual are prioritized and resolved quickly.
- How to measure it? How many tests are linked to a certain defect, also by using things like Google Analytics (where possible) to track usage of the various application components.
- How to improve it? Impact is difficult to control; critical bugs will always arise and they must be dealt with swiftly. But teams should aim to improve the accuracy of their impact measurements over time, as they get more data around their application.
Planning your test metrics can seem daunting, but it can be simplified by starting with the basics. We’ve found it is always best to start small and to grow and adapt the number of test metrics tracked as time progresses. It’s always easy to add a new test metric to a report – but can be tough to take one out. So, start small and measure what matters!
Opinions expressed by DZone contributors are their own.