The 5 Best Metrics You've Never Met

Technology companies working to stay relevant in their market all seem to be waist deep doing some kind of transformation. Agile transformations, Digital transformations, and DevOps transformations are ubiquitous as companies attempt to change the way they work in hopes of improving business outcomes.

Measurement and metrics are a key part of any transformation. When it comes to assessing a transformation, to see if the needle is moving in the right direction, performance metrics have come under intense scrutiny. Traditional performance metrics, such as counting the number of lines of code and the number of software bugs should be used with caution because there are bugs that are not worth fixing and code that is not worth maintaining. These old-school performance metrics represent activities, not outcomes. Activity metrics tell organizations very little about the true impact on business goals.

Activity metrics focus on busyness, but busyness does not equate to business value delivered. People can be remarkably busy all day long dashing from meeting to meeting with no increase in business revenue or reduction in costs. Traditional metrics may be free and easy to measure in existing tools, but are they beneficial? Think about the behaviors that occur in your organization by how metrics incentivize people. What we measure impacts people because people value what is measured. We need to find better ways to measure outcomes.

So what to measure? Consider Flow metrics. Flow metrics are performance metrics that reveal trends of desirable business outcomes, such as faster time-to-market, responsiveness to customers and predictable release timeframes. These business outcomes play an essential role in successful transformation efforts as the bar to remain relevant in the future keeps rising. Flow metrics correlate with the generation and/or protection of business value. Allow me to introduce you to five powerful Flow metrics.

Flow Time

Flow time is a measure of how long something takes to complete from beginning to end. You might be thinking, "Wait, that's cycle time." And you might be right. It depends on the context as to which definition you use. Depending on whom you ask, "cycle time" has different meanings and the clock may start or stop in different places. Just know that cycle time is an ambiguous term and that's why I prefer to use Flow time when discussing speed metrics. Because Flow time is an unfamiliar term to most, it provides an opportunity to clearly define the meaning. Flow is value pulled through a system smoothly and predictably and is the first of the three foundational principles underpinning DevOps

To determine where to start the clock for your context, consider the Flow time illustration below. The clock starts ticking when the request is approved and ends when the change is up and running in production.

In comparison, the clock for Lead time starts with the initial customer request. But, similar to bugs that are not worth fixing, some customer requests are not worth doing. Take Apple for example. With popular products, the number of changes requested is so high that it is not possible to triage them all. Popular open source projects have similar problems.

Flow time has a start time and an end time. That's all. Flow time doesn't stop the clock just because the weekend rolls around. What flow time does do is help quantify the probability of completing x percent of work in so many days.

Collecting historical Flow times that show, for example, that 90% of a certain type of work gets delivered within 30 days allows us to say that 9 out of 10 times, we deliver these kinds of requests within 30 days. We know then that there is a 10% probability that some work will take longer. This is important because it helps us become more predictable with our customers.

Flow Efficiency

Good metrics help others to see a clearer picture and help set more accurate expectations when it comes to questions like, "When will it be done?" Due dates rarely take wait time into consideration. The problem is usually not in the work time — it's in the wait time.

Think about delays from dependencies on other people — wait time matters more when it comes to how long things take than the actual size of the work. You are better off estimating the wait time than the work time. Wait time often consumes 85% or more of flow time.

The WIP Report

A training team that concentrates together on producing training materials progresses faster on the training collateral during weeks when they are not also traveling to customer sites and speaking at conferences. A marketing team progresses faster when they work on 7 initiatives at one time instead of 13. College students finish their homework sooner when they take two classes instead of three classes. One can argue that it depends on the complexity of the work. The homework for three freshman-level classes may take less time to complete than the homework for two graduate-level classes. And this is why it's important to break up work into smaller bits that can be completed and delivered quickly. The quicker the delivery, the faster the feedback.

Too much Work-In-Progress (WIP) (referred to in the Flow Framework as "Flow load"), opens the door for more dependencies, more conflicting priorities, more unplanned work to creep in, which causes delays. Capturing WIP trends and comparing them to Flow time results can help you see the relationship between WIP and speed in your organization.

The Aging Report

Aging reports reveal how long work has been sitting in the pipeline not getting done. Looking at all the work that's been in the system for more than 30 days (or 60 or 120 days) shines a valuable light how much waste is in the system. This example compares the average duration of work items and highlights the ones that are taking longer than the average.

Flow Distribution

Categorizing work into different work types supports changing work priorities and report data filtering., Flow Distribution shows the targeted (and the historical) proportion of work item types, bringing visibility to planned work allocation. When work is categorized, you can filter on work type for reports, such as the WIP reports, which in turn can help you improve your WIP allocations and your predictability.

Depending on context, allocations may need to change. If you've just released a new feature, tackling defects or debt may take priority over introducing more features. If you continue to do more feature work, it will steal capacity away from other work, like fixing problems related to tech debt. Categorizing and measuring work type distribution helps you prioritize accordingly.

Mapping Metrics to Outcomes

Keeping pace with the future requires change. When it comes to transforming the way teams work, consider mapping metrics to desirable outcomes to improve decisions during transformations - or during other times filled with uncertainty.

If time-to-market is one of your desirable outcomes, (because people complain about how long things take), measure Flow time to help others see just how long things actually take.

If efficiency is one of your desirable outcomes, (because people are blocked waiting on specialists or events), measure Flow efficiency to see where bottlenecks exist, so you can focus on areas that will improve flow. When it comes to flow, It doesn't do much good to optimize an area that is not the bottleneck..

If teams are dealing with unplanned work and/or conflicting priorities, measure the amount of WIP to expose overallocated teams. When it comes to efficiency, time is wasted when there is too much focus on resource efficiency overflow efficiency.

If important unfinished work (such as fixing security vulnerabilities) is neglected, measure the age of partially completed work to expose risks. Like a bridge under construction, zero value is realized until it's finished.

If certain types of important work (such as fixing technical debt) are not prioritized accordingly, measure work type distribution to bring visibility to problems related to allocations.

Final Tips

Beware of falling into the trap of optimizing for a single metric. A hyper-focus on speed might not make you more predictable. It's okay to ease off in one area to benefit the whole system.

Flow metrics look at the trends over time — instead of viewing single data points in isolation, ask — are we moving in the right direction?

If letting go of existing unhelpful metrics is too big of a change for your organization and meets with resistance, consider capturing Flow metrics in addition to your current metrics and compare the outcomes. Experiments are a great way to test new ways of working.

Quantitative measures are usually more accurate than personal perceptions and experiences. When you are knee deep in transforming, Flow metrics can help you make better decisions.