The bug reports that come out of the different stages of testing throughout any development lifecycle are one of the best assets you have for analyzing not only the quality of the product your release, but also how well your development and testing processes work.
If you are using a good test management suite like HP Quality Center, you probably ask developers, testers, and other people involved in various testing phases like members of the user community, to create your bug reports using a standard form; this makes the process easy to understand and allows for consistent reports you can easily run analyses on. However, it can pay to put some consideration into the customizations you want to make to these forms early in the project, or, if you are a software company, in your corporate test strategy. While it may seem like the path of least resistance is to ask for only the truly essential details you need to investigate and fix something – that is the severity of a bug, the test it relates to, and the steps to replicate it – if you do this you are actually losing out on a great opportunity to gather more intelligence about your product, and the way your teams are working.
One metric many teams fail to ask for in their bug reporting is the 'expected test phase' for each bug or issue. This is a shame, because this is data that can actually be extremely useful and telling if you want to improve your overall processes, and also work better with any external development or testing teams working on the project.
What is the 'Expected Test Phase' for a Bug?
Using an expected test phase is actually quite intuitive – all it refers to is the test phase you would expect a bug to have been found in. If you have a coding bug that stops some functionality working, then you would expect this to be found before the software or iteration goes into functional testing or any other form of black box testing, for example. If unexpected paths through the system cause the bug, you would expect this to be found in functional testing, rather than user acceptance testing.
Why Is It so Helpful?
Knowing the expected test phase doesn't help you fix the bug, but it does give test and development managers a clear picture of whether iterations or full releases (depending on your development approach) are moving between test phases in the right way. A lot of bugs found in a later test phase than expected means that software is either being pushed through the pipeline too quickly, or there are problems in the testing at a given phase.
If you are working with third party developers then it can be even more useful. If your testers are finding a lot of bugs you believe the developers should have found and fixed before handing over the build to your test team, then this data gives you a way to quantify and prove that, making it much easier to ask them to tighten up their testing.
Adding Expected Test Phase to Your Required Bug Report Fields Without Causing Confusion
If you are using Quality Center or similar, the expected test phase is already an optional field on bug reports, so really you just need to add in to your policies and project test strategies that you will require it. It may seem complicated to explain to the people who will be reporting bugs at each phase, but for professional testers and developers it is easy to work with, and test managers or other development team members who are working with other stakeholders or users who are less familiar with testing terminology can always add this data to those bug reports on their behalf when they are reviewing the bugs.
The expected test phase is an often overlooked piece of data about each bug that really is a lot more important that it seems. By adding it to your general testing and bug reporting policies, you'll find you have a new way to analyze the performance of your project, and a way to more effectively hold third parties accountable for the work done on the builds they deliver.