Doing Big Deployments That Don’t Suck
I’ve been doing more than my fair share of deployments lately.
These deployments aren’t the one-click-then-beer variety you find on Heroku or in some seriously agile shops. These deployments are arduous day-long journeys, caravanning eight unique technologies, traversing multiple connected systems both up- and down-stream. And bad things do live in the water.
Seriously, it has been a lot of hard work and some difficult times reliving past mistakes. So here’s my list of freshened lessons that make big deployments suck less:
1. One group chat and one conference line. Don’t use email for the deployment team. It’s slow. terrible at threading conversations. And, somehow, people always get dropped off the address list – or appended to the cc field 37 times. A dozen individual chats and phone calls are terrible for anything but proving why Europe mothballed the semiphore back in the 19th century.
2. A deploy with fewer people is better. An all-hands-on-deck deploy may sound comforting. In reality. that’s just more people in the way, arguing over the best corrective path, and slowly burning out while standing at the ready. Deploys with fewer people – with clear responsibilities and fully prepared for the tasks of the day – beat mob deploys any day of the week.
3. Have your support people at the ready. Just because your core deploy team is fully briefed and prepared doesn’t mean you have all the expertise you’ll ever need. Know who else you’ll need if the fit hits the shan. Make sure that each member of this auxiliary group is ready to drop in at a phone call’s notice. Collect and distribute the best contact information for that auxiliary group to each member of your core group prior to deploy day.
4. Clear communication channel. Who owns outward communication and in what direction? People who need deploy status updates could include stakeholders, upper management, cooperate support teams, and auxiliary team members. They will not all want or need the same status reports. (Hint: you don’t want to send a 100-line deploy plan every hour to your stakeholders.) You need to decide in advance who owns which channel, the frequency of status updates, and the content of those updates.
5. Clear decision-making. When your app server and the database stop talking the last thing you need is a team that descends into bickering over which server to kick first. A deploy will involve a variety of experts, but there should be a single person in charge who – once all voices have been heard – calls the shots.
6. Clear accountability. The entire deploy team should know who is responsible for what well before the deploy begins. For me, that means one owner for each discrete part of the deploy. This separation may be by system, technology or product subteam – whatever makes sense for the deploy. Finally, only core deploy team members own deploy tasks. Sometimes a non-team-member is responsible for completing a task, but we still want a core deploy team member accountable to see that it gets done.