In our Enterprise Continuous Delivery Maturity Model we looked at the idea of continuous deployments to production and flagged the process as “Extreme”. It’s just too much for too many teams. A decade ago, I might have said the same thing about continuous integration.
As DevOps and continuous everything have taken hold, we see more and more teams start to contemplate deploying each change to production. These are the rare teams with really great tests and little hassle from auditors.
While still the minority, the growth in these groups is definitely noticeable. Most teams don’t get all the way to production though – even if they want to. They move only to the last pre-prod environment. It’s hard to convince the business or their colleagues that fully automating the release cycle is safe.
Paul Kipp recently tackled this fear on his blog. I love his take on the topic and encourage you to read his article in full. Paul highlights that the discrepancy between how meaningful consistent success in pre-prod is (pretty meaningful) and the amount of confidence we usually gain from that success (not so much). Paul suggests a big sign indicating the number of days since we were really glad we didn’t deploy live. When that number gets large, you can have a talk about why you are leaving value on the shelf rather than getting it to customers more quickly.
Great idea. But I think that many on the business side will see the sign as analogous to a circus posting “160 days since last tight-rope walker fall.” While fewer falls is better than more, broken necks are always bad. Compliment this broadcast of good MTBF (mean time between failure) with better recovery metrics (MTTR).
Prove that you can rollback quickly. Or better yet, build for resiliency, with a cluster immune system. When you get to the point where you can’t do much damage to production even when you try, you’ll have an easy sell.