How to Run Post-Release Reviews: Learning from a Release Event

A release is a highly orchestrated set of changes performed on a complex system, and, as with any change to a software system, it can introduce risks to availability and system stability. These risks are offset by the rewards of shipping updated software to customers, but you should view every release event as a trade-off – your organization is shipping software and the price is increased exposure to downtime and error.

Releases are risk, and the best way to minimize this risk is to analyze each release event. Your goal as a release manager is to mitigate these risks and to understand how to take advantage of every opportunity to learn from your activities.

If you are performing regular releases, you should be having “after-action” summaries and identifying opportunities for improvement. In this post, I’m going to discuss some of the mechanics of a post-release review session. How do you run them? When do you schedule them? Who should be involved?

First, Do You Need to Conduct an After-Action Meeting?

If your releases involve a release playbook that people have to follow; in other words, if you are not 100% automated with your release process you should schedule a meeting after every single release. If you have releases that span departments and which involve a high amount of risk you will certainly benefit from an all-hands-on-deck review meeting.

On the other hand, if you release to production every single day using highly automated systems your release process may be a non-event that people fail to recognize as a release. In these situations, you don’t need to schedule an after-action review, but you should be scheduling periodic sessions to review efficiency and your automation playbook. This fully automated release reality is rare, but more companies are moving toward full automation.

When Do You Schedule the After-Action Meeting?

Schedule your review meeting as soon after the completion of your release playbook as possible, but not too soon. Some release processes are very calm and controlled, but other release processes can stretch long into the early morning as changes must be made off-hours. If your releases are especially difficult to give your team some time to reflect on what worked well and what didn’t.

Make sure you schedule your after-action release reviews soon enough for people to remember the details of issues encountered during a release. The pace in many organizations is hectic and waiting a week is often too long as key actors will lose track of the details.

Who Should Be Involved in an After-Action Review?

The following people should be involved in an after-action review:

Release managers responsible for all affected systems. This includes both application managers for individual applications and enterprise release managers who were coordinating cross-project release activities.

Developers involved with the release process. You may have hundreds of software engineers involved in creating software for a particular release, but you don’t need to involve all of them. The senior developers involved in a release process are the ones supporting knowledge transfer to support groups and other high-risk release activities such as database changes.

System administrators involved in release activities. Anyone who had to support changes to production-facing systems from an operations perspective should provide feedback.

Project management. Project managers responsible for project scheduling and communicating with the business should be included in the meeting to provide feedback on planning and visibility.

Anyone with responsibility during a release process needs to have input into the review process. If they are not able to attend the meeting, the release managers responsible for the review should ask for written feedback.

Questions to Answer During a Release Review

Here are some of the questions you want to ask during a release review to drive continuous process improvement:

“What were the communication pain points during this release process?”

Whether you are dealing with a QA team that doesn’t feel like they had enough lead-time to prepare testing environments or a production control team that didn’t get enough information. If you are running a highly orchestrated release process that involves more than 50 people there’s a good chance that most of your follow-up activities are going to be focused on communication and notification.

Discuss communication surrounding the release announcement. Ask if there were any cases where communication was missing for release governance decisions or release scheduling. Since a release is a constantly changing process with a lot of opportunities for delay, it is important to make sure that release management has a “megaphone” to communicate with all affected participants. As you conduct multiple release reviews you’ll develop an end-to-end model of how you communicate release plans and release status. Make sure to use these meetings as an opportunity to adapt this model to organizational changes such as new teams or new team members.

Using Plutora, you can assess communication channels and look at RACI matrices to understand who was informed and who should have been informed. A good follow-up task is to assess every RACI and understand if it was accurate and how it needs to be changed going forward.

“What should we automate next?”

It is 2016 and it is time to start automating your release process. If you have tasks that require people to wait while someone runs a script or performs a change to a database look into approaches that allow you to orchestrate and automate. When a highly complex release process relies too much on manual playbooks you run the risk of human error.

During your meeting make sure to ask all participants what they think can be automated. Release engineering teams tend to be defensive when people bring up automation. It is often viewed as a threat to these teams so make sure to be sensitive. You may need to solicit this feedback outside of the meeting and then follow up offline to avoid conflict. Again, it’s 2016, and just about everything that can be automated needs to be, but there are some valid reasons not to if a process is being retired or if a process requires on-the-fly decision making that is tough to model with a script.

Using Plutora, you can track all release activities and understand how long each took and what impact it had on the overall release timeline. Use the feedback from your after-action review and Plutora to identify the best opportunities for automation.

“What were the risks in this release and how do we avoid them going forward?”

Releases are pure risk, and a release process contains steps that are riskier than others. Ask participants to think through the process and offer up suggestions to make your releases more predictable and less risky.

This could involve making sure that there is a rollback plan for a critical change or adding in an extra validation step to avoid release-related downtime. It’s impossible to have a software change without some risks to availability, but you can reduce your exposure by understanding what release process steps need attention.

With Plutora, you can assess how a step impacts an entire release’s timeline, and you can find the trouble spots. Use Plutora as a tool to help you visualize the “internals” of a particular release, and use it to identify critical cross-project dependencies that often lead to “hold your breath” moments during a high profile release process.

“Does anyone have any suggestions?”

Open up the floor for suggestions and take people up on their offers to help improve a release process. As a release manager, you’ve probably realized by now that you can’t fix every problem yourself. Use these after-action review meetings as an opportunity to encourage more participation and engagement. If you practice these review meetings regularly, you’ll have a team that feels like they can affect the tools and processes they use.

A team that feels like they can own and influence the release process is a team that won’t settle for the status-quo. These after-action release reviews are your chance to encourage a culture of continuous improvement and honest, data-driven decision making that is shared across multiple teams.