Reimagining DORA Metrics and Leveraging Feature Flags

Does the emergence of feature flags affect the interpretation and utility of DORA metrics?

On this week’s episode of Dev Interrupted, host Dan Lines and Ariel Perez, VP of Engineering at Split.io, discuss the state of DORA metrics and whether they need reimaging in a world of feature flags. Listen as Ariel explains why he believes feature flags are more than a tool and have begun to reshape our understanding of software development and the metrics we use to measure it.

Dan and Ariel also touch on how feature flags can drastically reduce lead time and mean time to recover and conclude their chat with an intriguing look at the granular nature of control in the modern software engineering landscape, where the unit of control has shifted from the application as a whole to individual features.

“ The reality is that about 70% of everything you put in production has either little or negative value.

So only 30% of what you're doing actually has value for the customer. It's unbelievable, right? So how do you think about that idea of value? The reimagined world of DORA metrics with feature flags and that granularity help you ship faster.”

Episode Highlights

• (2:05) Introduction
• (7:15) Pursuing productivity at the expense of effectiveness
• (15:50) Project allocation discussion
• (24:55) Reimagining DORA metrics
• (31:30) Radical ideas for DORA
• (41:25) How teams should invest in shipping faster

Episode Excerpt

Ariel Perez: But the first one is this idea. DORA Metrics came in a world where we were just trying to ship faster, right?

And in a world where deployment and release were the exact same thing, in order for you to ship a change to a customer, it was a release and a deployment. Same exact thing; I had to put the software in production, and the world has changed a lot since the DORA metrics were published initially. And again, DORA metrics are amazing at trying to figure out which teams are very efficient at shipping software that doesn't break shipping software that's high quality.

How good is an engineering organization at building stuff and shipping it? The key thing is what happened with the world changing over the last 10 to 15 years, but even more, the industry really even less so over the last 10 years, feature flags as a term, as a concept for building my applications right.

And that's not assuming that everyone knows exactly what a feature flag is, although it's a lot more ubiquitous now, it's a toggle, it's a switch, it's a feature flag. It's basically, at its core, an if statement around your code. If this feature flag is on this code path gets executed. Otherwise, it's off. Sounds very simple, but the radical aspect of it is now I can actually ship code off and, at a separate time, turn it on.

You have different ways to do it. So many, there's UIs, there are database implementations, and config implementations, but that's the key idea that I can just ship the code and keep it off. Now, why would I do that? I need the code to be on. There are so many things that you can test and validate while the code is off before any customers see that code.

You can imagine how many times you think something is perfect in staging, and then you put it in production, and it explodes. You miss some config. You miss some issue features. Flagging it off allows you to ship it out without it breaking. The other thing that features flagging really allows you is a trunk-based development branch by abstraction.

You can just commit your code and not wait for these long-lived branches that just carry risk because they're not being merged. So now, how does that change the DORA metrics? The idea then, now in, in a feature flag world, it's not enough to think about how often you're deploying. The idea is, how often are you releasing?

Cuz you can deploy over and over again with the code off. One thing that the features flag has helped you do is to deploy more often. Yes. That's great because if you're feature flagged off, you can merge more off, then you can merge to the trunk and just ship the code. So your deployment frequency skyrockets, but you're not delivering anything to users yet.

So with feature flags now, you might turn off and turn on different features at different times. So then, how do you define deployment frequency now? Is it change frequency? Is it change enablement frequency? So it's something to think about there. If I ship one piece of code, And then I flip a flag next week, and I flip another flag the following week, and I flip another flag.

The last week, did it have one deployment, or did it have five deployments? Is it five changes? Is it changing frequency now? So that's one thing that comes to mind. Yeah. The other thing that comes to mind is I think lead time drastically goes down when I can feature, flag everything off and merge it.

But the other major impact is meantime to recover. For elite engineering teams, the mean time to recover is primarily constrained by how quickly I can ship a fix. But in a feature flag world, it's how quickly I can flip a flag and turn it off; your meantime to recover can go down to seconds in the best feature flagging platforms.

So that's another massive impact on that DORA metric.

Read the full episode transcript here.