The first time a developer approached me about continuous delivery, I was a bit surprised. I was the Agile coach on the project, and I was not involved with the day to day workings of the delivery mechanism. I was the project’s expert on Scrum: the world of sprints, stories and estimation. How could I help with deployment processes?
“Isn’t delivery more your area of expertise?” I asked him.
“Well, yes perhaps, but we need your help to make deployments more frequent and more disciplined.” He said.
“That’s what you mean by continuous delivery?” I asked finally.
“Yes. We want to cut down deployment time from 2 weeks to one day” he explained. “And we want deployments to be less stressful”.
“Is that even possible? You mean you didn’t love it when we had to work till 11 PM during the last launch.”
“No, we hated it. And we want to make things easier. It benefits scrum too. More predictable deliveries, happier clients,” he said.
Well, that got my attention. I wanted to know more before I could be of any help, however. The team and some research helped me understand what he was talking about.
Continuous delivery is the implementation of a pipeline (workflow) where code is frequently checked in, tests are automated, and deployments are frequent (note that you do not have to make the code live; you just need to be in a state so that it is ready to go at any moment.)
A continuous delivery pipeline has many benefits for the team, business and the users:
- Ensures that most major issues and conflicts are identified early in the development process when it is cheaper to fix these.
Less stress during deployments and releases - happier developers!
Increased predictability in workflow since the process is iterative
Transparency: everyone is aware of all the needed steps and all or portions of the pipeline can be managed by anyone in the team.
When a step in the pipeline become obsolete or a modification is needed, the team can implement changes easily or roll back if needed.
Allows team to collect peer, business and user feedback more rapidly
Cleaner code and stable environments, lower risks with deployments
Increased ability to react to market and user needs; nimble workforce
I met with the full development team so I could get their thoughts on adopting a continuous delivery pipeline before we made anything final. I saw all the benefits, but I was cautious by nature, so I dug into what the challenges were; what were the barriers to entry?
“Before we can deploy daily, we have to clean house.” one pragmatic developer chimed in. “Right now, we check in code whenever and cut branches by the dozens."
He was talking about the issue of source control management. Code is usually worked on collaboratively using software like Git which allows individual developers to clone the source repository (aka cutting a branch) and work on slices of code in their local environment; this ensures that developers do not corrupt the main source code (mainline) or cause conflicts with others who may be working on the same piece of code.
“By cleaning house, you mean we can’t have 2 thousand branches in Git, right?” I said. I’d nagged them about cleaning up the branches on other occasions, but I did not have a compelling enough reason until now.
[Grunts and agreements from developers]
“And even after you clean all that up, you have to stay cleaned up. You can’t cut branches willy-nilly. We should only check in to release branches,” the Pragmatist said.
Release branches are branches which may be released into production if all tests are passed.
“This can only work with continuous integration in place,” said one of the test analysts. “We are using Jenkins for integration like we should, but not all our testing is automated.”
Continuous integration is kind of the older brother to continuous delivery. Continuous integration allows team members to safely merge their changes into the source code.
“Listen, before we get a new pipeline, let’s talk about the environments. It’s all lunky right now. All the apps are tied together like christmas lights, and if you deploy one small change, the rest will light up too - and not in a good way,” said the Architect. “Every time you change the color of a button, I don’t want have to deploy for 2 hours because it has to run through everything.”
“Yeah, that would really slow everyone down” someone else added.
“Okay, okay,” I said, “let’s just get all the feedback on the whiteboard so I can get a grip!”
What We Wanted
1 day deployments using an automated delivery pipeline.
What We Needed to Do to Get There:
Checks-ins have to be frequent — at least once a day
Check-ins made to release or other designated branches only
Have to automate as much as possible so manual testing is minimal or none
Deep dive into creating a more flexible and modular infrastructure so that tiny changes do not slow down the whole system
Make sure the team follows ONE delivery pipeline so that you don’t have multiple processes causing conflicts
Encourage deployments in progressively more production-like environments, and if possible, directly to production
Codify the rules of the pipeline so that everyone knows what to do, and deployments are not the responsibility of one or two people
Create feedback loops so that we can make the pipeline better over time
Create a culture which prizes clean code and stable environments over adding new features without careful thought
When we reviewed our list, a few solutions seemed pretty obvious from the get go. I said that we could codify deployment rules using the Definition of Done (DoD). The team looked at me like that was really obvious.
"Fine, you already knew that!” I said. “And you probably know that I am going to suggest retrospectives for making the pipeline better over time.”
Yep, they already knew that.
“The things that we can’t use retrospectives or the DoD for,” the newest developer on the team asked, “what do we do about those things?”
“Ah! Well, we can create rules of engagement for encouraging code cleanliness.” I said. “But all this will not be a quick fix. Building the type of culture we’re talking about is not trivial by any stretch of the imagination. And changing up the infrastructure and/ or getting all the tests automated are going to be large projects on their own. You’ll need real elbow grease for that. Those changes have to be done gradually.”
“Incrementally?” one of the devs said looking very happy with himself. “Perhaps even iteratively?”
“Yes, precisely," I said. "Scrum can help us bake in some time every sprint so we get a chance to clean up the code, and modify infrastructure. We'll have improve all this non-functional stuff even as we get new features out the door. We’ll have to make sure to take all this into account during sprint planning.”
“Yeah, we do. And we need you to make sure that the Product Owner is educated about all this, especially the impact to velocity. We need you to explain to stakeholders why these changes will help velocity in the long run,” the Architect said.
They all nodded.
“Yeah, I figured I should do those things,” I said with a smile. “Your humble Scrum Master, at your service!”