Assessing Team Performance and Software Quality for Improved Test Management

Nearly every software company that has ever existed has tried to get a feel for quality and team performance with a metrics program at least once. Managers and directors review test management reports that describe lines of code, bug counts, pass and fail rates, and sprint burn-down charts and make guesses about what the numbers mean. These types of numbers and metrics hint at the story of development, but they don’t tell the whole truth; that half-truth can get people into trouble.

So, how can managers get better information about software quality and team performance, and make more informed test management decisions?

The first place to start when looking for a deeper answer is to ask what problem the team is trying to solve:

• Do releases run over schedule?
• Do software features run late?
• Are customers complaining about quality problems?
• Does one team feel slower than all the others?

Identifying the right questions will help guide where the magnifying glass of measurement needs to be focused.

Team Performance

If the problem is related to performance or efficiency, test managers can explore some ideas around Lean fFow, the area of study that helps people figure out exactly how long it takes to go from idea to delivered product.

Let’s look at an example of how Lean Flow can help improve team performance.

Think about a run of the mill software development team. At the beginning of a sprint, the team commits to completing a set of features in the span of two weeks. At the end of that two weeks, only 7 out of the 10 committed features are ready to go to production. But how can that be, why was team late?

The superficial answer might be that estimates were off, and that might be true, but there is something deeper that is making this team late. Every person on that team was working on more than one feature, jumping back and forth and context switching. The workflow looks like this:

• start writing code
• go to a meeting for the other feature that will be worked on later
• submit some code to the repository
• get bug reports and ask some questions
• review design document for other feature
• and on and on…

In Lean Flow, there is a measure called touch time. This is the exact amount of time a piece of value, in software that is usually a feature, is being worked on in a week. This number is usually strikingly low. Guess why.

Work in progress (WIP), is the devil. The more tasks a person is jumping between in a week, the less that person can get completed. That’s just math.

Most people don’t realize how much work is going on at any one time. Trying to shrink that will show performance improvements. Take each feature a team is working on at one time, and write them down on post-it notes, one per note. Stick these notes up on a wall and stand back. That list is probably more than a handful. If the list is more than one note per team of developers, the list is too long. One piece flow, working on one feature at a time till it is ready to ship to production is the top of the line in efficiency. Working on one feature at a time shrinks context switching and causes touch time to skyrocket.

Quality

The other prong of the measurement problem is needing to get a feel for quality. The traditional way to do this is to look at bug counts for each release, maybe organized by severity, and when that number gets too high saying ‘Why is quality so bad?!’. This usually doesn’t do more than drive team moral down for a few weeks. Looking at bug counts (in context) and how they change over time might be interesting, though.

For example, if the last release had 5 bugs reported by the customer, and the newest had 50 reported bugs, something is not right. Maybe the client upgraded their browser, or maybe development was down a person and under a lot of pressure. Regardless, there is something there to study and learn from. Changes in reported problems can be a good place to investigate, but they don’t point directly to changes in quality.

Quality is value usually to a paying customer or someone with enough power to get people working on a change. To understand product quality, someone has to talk with that person and figure out how they feel about the product. While one user may have problems with the product, another may be completely in love with it. Characterizing how everyone feels with one number is questionable at best.

There are lots of different ways to measure team performance or product quality. Some of them are useful, some (DRE or % of tests automated for example) of them are meaningless. To get started, find a question in the process and measure it for a couple of weeks. Try an experiment and then move on. Most importantly, recognize that numbers are only a place to start investigating. A person has to put the story together.

Download our free guide to understanding agile testing methodologies.