Beware of Misleading Metrics

Yvette Francino spent years as a software developer within IT organizations at IBM and Diners Club, International before moving on to IT management roles at companies including Sun Microsystems, TechTarget, and McKesson. She is an Agile coach, based in Boulder, Colorado.

Is Agile Really “Changing the World”? Will We Still Be Talking About It In Five Years?

YF: We’ll still be talking about it.  Every year it keeps trending up. Now, Agile is being used in all kinds of areas. I just heard of a wedding planner who is using Scrum for project planning. I think a lot of the Agile terms are going to end up in dictionaries. Agile is more about project management and leadership techniques than software development. It’s not a magic bullet. And it has been around for a while. The concept is making small changes, inspect and adapt, grow and improve. The magic is in putting all this together.

What Are Some Effective Ways to Measure the Results of Agile?

YF: In software, metrics is a hot topic and it can be controversial. One classic Agile metric is velocity. My favorite metric comes from customer satisfaction, and there are many different ways to measure that. There are lots of metrics around quality. You need to be careful with defect counts, because there is so many types of defects and metrics can be misleading. My team once got an award for no defects reported after the release. Yet we found out later that we had no customers for the product, which is a worse problem. What you decide to measure should depend upon the product, your objectives and the customer needs. It mostly comes down to determining if customers are getting what they want.

In a Recent Article, You Wrote That Unit Testing Should Comprise Most of the Automated Testing Activity. Why?

YF: In unit testing you are testing a very small piece of code, so it is the easiest to automate. It is always automated because it’s programmatic. The further out that you go toward the UI, it gets more complicated due to the interdependencies in the code. If you have great coverage at the unit level you don’t need to go through every different code path later. Certain types of testing such as usability testing or exploratory testing cannot be automated. Save the manual testing for that type of creativity, but not for tedious logic testing which can be automated.

What Are the Challenges for Developers and Testers to Transition From Manual to Automated Testing? Is It Difficult to Write Automated Tests?

YF: There are a lot of challenges with automation. Testers without coding experience or any knowledge of scripting languages will probably struggle. Automated testing is programmatic, and is essentially a form of coding. While some tools don’t require actual coding, you can use the English language, you still need to put in the right keywords which translate into code. Understanding those logical patterns of development are important. Testers don’t need the same skill sets as developers but since they are part of the same team in Agile, understanding what developers are doing, and having a common language is important.  Putting an automation test framework in place and building up the skillset of the technologies used can be a big challenge for teams.

Does Manual Testing Still Have a Place?

YF: Some people have the opinion that you can automate everything. Yet my feeling is that you still need manual testers. There’s also the option of using beta customers as the testers. So the question may become, who does the manual testing? I have found that at enterprise companies at least, there is usually a combination of manual and automated testing. Manual tests are necessary, again, for exploratory tests and other tests that require human feedback, such as usability tests.