I came across an interesting article on DevOps.com by Payal Chakravarty of IBM discussing what they learned from their Agile to DevOps transition. At the outset of a major change, it is critical to pinpoint the aims of the project and how to translate those goals into tangible metrics with which to track advancement. In fact Payal’s team at IBM questioned from the beginning how they were going to qualify the project as a success. This led to creating over time, eight metrics which they still use to keep track of their success and progress.

1. Frequency of deployments – The number should either remain steady or go up week to week.
2. Volume of changes – Measure the volume and complexity of user stories and new lines of code deployed.
3. Amount of time from development to deployment – The lead time from when the code begins development till it’s deployed to production. The amount of time should decline as the team develops – and is a central gauge of how efficient the process is and where it must be improved.
4. What is the ratio of unsuccessful deployments? – How often do deployments fail or even cause outages? As DevOps is implemented, the quality of the deployments should go up and the percentage of failed deployments should go down.
5. Recovery Time – This is perhaps the greatest test of the quality of the team – how long does it take to recover when a failure does occur. Although the time should generally trend downwards, teams should not be discouraged by the occasional spike as they run into issues for the first time.
6. Customer Tickets – The goal of DevOps is to increase deployments without causing failures. By reviewing the number of customer tickets you’ll have a good idea how well you’re doing.
7. Increase in user volume – By tracking how many users are signing up you can make sure the organization can handle the new requests.
8. Response Time – This number should remain steady no matter the percentages of change in user volume as the product should be functioning in predetermined thresholds.