How Data Science Can Drive DevOps Growth
Engineering analytics platforms offer DevOps leaders the tools they need to be able to identify bottlenecks in their team's Software Development Lifecycle.
Join the DZone community and get the full member experience.Join For Free
To improve, you must measure. Without measuring, you are left reading tea leaves with no understanding of whether you're closer, or further away, from your ultimate goal. Data provides the essential feedback to truly understand how well we're doing—indeed, it often surprises us.
This concept is of fundamental importance for Engineering Managers and one I realized the importance of after reading The Goal by Eliyahu Goldratt. Indeed, as many successful organizations do, I had it as my main piece of compulsory reading when onboarding new starters. As The Goal elucidates, it is utterly essential to select the right goal and North Stars before you can drill down to start to improve things.
Engineering Managers using erroneous measures of performance (for example, placing excess importance on lines of code changed) have fundamentally misunderstood the importance of selecting the right metrics. Thanks to the work of Nicole Forsgren, Jez Humble, and Gene Kim through many years of working on the State of DevOps Report, we know how to measure a Software Developer Lifecycle in isolation. Summarised in the book, Accelerate, we understand that just 4 key metrics serve as indicators of software delivery performance. The authors remark that of these indicators, the “highest performers are twice as likely to meet or exceed their organizational performance goals”
These four metrics can then be broken down into more local indicators to understand how to improve the process. To cut Cycle Time (time from the first commit to that being merged), many factors can become important; such as developer tooling, review time, test execution time, etc. Once an outlier local metric is found, it becomes necessary to drill in further to understand how to improve things further; for example, you might identify that Pull Request review time is slow specifically because of a slow time to first response. Both quantitative and qualitative data become important as you drill further in.
In other words; it's absolutely reasonable to be opinionated about what data you analyze and how you drill in to identify where bottlenecks are. Presenting erroneous data has the prospect to cause more harm than good (I would highly recommend Hello World by Hannah Fry for an excellent read on what happens when data science is abused). Don't just question how the data itself, but question why that data matters.
I fundamentally believe that software engineering organizations are lagging in their ability to use data to understand bottlenecks. We lack the data which those managing customer support or marketing teams have been able to take for granted. For this reason, I've been particularly excited to begin supporting Haystack Analytics in building out its engineering analytics platform. I recognize this isn't enough though, tooling alone cannot drive improvements to productivity—instead, we must fundamentally see data as integral to the DevOps process.
By simply being able to measure, we can tell whether we're improving or not. Not all decisions are black and white though, sometimes it can be tough to understand if a new process change or approach will make things better or worse (particularly when they are human-centric policies). Likewise, there can sometimes be trade-offs between our North Star metrics (for example; a particular approach may reduce Cycle Time but do so at the cost of a higher Change Failure Rate).
“You need to give your company the freedom to experiment. For instance Jeff Bezos’ “two-pizza principle” – that you should never have teams involved who can’t be fed on two pizzas. That’s how you get that type of experimentation.”
- James Anderson, Scottish Mortgage Investment Trust
It is essential to have both the platform and the culture to be able to take contentious decisions using evidence. Relying on empirical data ultimately reduces the space for disagreement. We should be able to experiment with changes to the engineering process using scientific study designs, like Randomised Control Trials (RCTs). RCTs are seen as the gold standard of scientific research due to minimizing bias and confounding factors. Nevertheless, in order to be able to study, you first must be able to measure—and this is the capability most engineering organizations currently lack.
Opinions expressed by DZone contributors are their own.