Release Management: Is Your Product Ready for Success?

You never get a second chance to make a first impression. So how do you make sure it's a good one? This matter is constantly in the mind of release managers and product owners because they know that once a product is set loose into the world, all bets are off.

Although releasing a product or new feature may be tempting as soon as it is functional, users expect (and deserve) quality and polish. Therefore, before even reaching the beta test stage, much non-technical work must be done: documentation must be written, security must be assessed, and business objectives must be established. These are some things we have learned about software releases at Semaphore.

Project Stages

From idea to release, we can break a software project into roughly three stages:

1. Design and Development

Comprises everything from idea to MVP or working feature. We've discussed design patterns in other blog posts, so that I won't spend too much time on that now. However, there is no shortage of practices to help us uncover good designs, like writing acceptance criteria and following Test-Driven Development, Behavior-Driven Development, or Domain-Driven Development methodologies.

2. Tech Preview

The sharp blade of the reality check. We bring a few users to get their early feedback so we can make improvements and reduce uncertainty at the release.

3. Release 

Also known as General Availability (GA). In this last stage, we make the release available to all users.

The stages are separated by checkpoints. We have two kinds of Definitions of Ready and the Definition of Done:

1. Definition of Ready (DoR): These tasks are required before users can access the system or experience the feature.
2. Definition of Done (DoD): consists of all the follow-up tasks closing a stage, like gathering user feedback, completing outstanding tickets, or doing a retrospective.

We cannot enter a new project stage or consider it done until all the tasks in DoR and DoD are ticked off.

The ready checkpoint (DoR) dictates if users can be allowed into our system. Thus, the technical preview can begin only once we meet the DoR. It does not end until we've completed all the tasks in the DoD. The same thing happens during GA release.

The Technical Preview

The technical preview is an opportunity to get the user's feedback before the final release. Before evaluating whether a feature or product works, we must bring them into the mix. During the preview stage, we deploy the latest release in production and use feature flags to control who can see the previewed feature.

We can deploy the latest release in production and use feature flags to select which users access the previewed feature.

Most users understand that they may find some kinks and will be happy to help us sort them out. Therefore, we can start the tech preview even if the application is not perfect (hint: it never will be).

That being said, there are some precautions we need to take before any user is let into our systems. Here's where the DoR for the technical release comes into focus.

Definition of Ready for the Technical Preview

Let's say we're shipping a new feature in our product. Early feedback from real users is essential for its success. We can reach out to users who may benefit from the feature or be interested in testing it out.

How do we know when we can invite users to check it out? First, we define a list of tasks and conditions that must happen before we can define our system as ready. These are tasks that are required to be completed for every feature before it can be shown to the users:

• Performance Metrics: every new component shipped should be tracked and added to the monitoring dashboard to observe its performance and stability. Semaphore, for instance, monitors the four golden signals (latency, traffic, errors, and saturation).
• Utilization Metrics: we need to ensure the new features are being used and how.
• Audit Logs: shipping anything to production that's not covered in audit logs is a security issue. Every event should be logged to have full audit compliance.
• Security Assessment: we must ensure the new components don't have any critical vulnerabilities. This is important for two reasons. First, to protect users from security exploits. And second, to comply with any security standards the company follows, such as ISO 27001.
• Deployment to Production: the release should be deployed to production. This can be the primary production environment or a staging copy as close as possible to the real thing.
• Preliminary Documentation: the users will need instructions about what changed and how to use it. It doesn't need to be full-blown documentation yet.

The list can be expanded to include more items as needed.

Once we have our DoR defined and fulfilled, we're ready to start the technical preview. After that, the feature passes to the "in-review" state, and we can begin the feedback-gathering process that may last weeks or months.

Note: Instead of technical previews, the engineering team at Semaphore used to have two kinds of betas: public and private, which had different processes and requirements. Having two sets of similar-but-not-quite-the-same DoRs and DoDs ended up being unnecessarily complicated. So, the processes were streamlined into one. Users are invited to the technical preview based on needs and interest in a given feature.

Definition of Done for the Technical Preview

The technical preview ends when we've gathered enough feedback and refined the system to the point we consider it ready for release.

Before we can close the technical preview, there are some tasks to complete, which will depend on the DoD for our project.

The items in the DoD bucket vary, as they will change from team to team or from company to company. For Semaphore, the list includes:

• Gathering Feedback: the whole point of this exercise is to gather actionable feedback from the users. Feedback lets us better incorporate the voice of the customer into the system. In addition, it allows us to find bugs and improve usability.
• They Were Cleaning Up Any Remaining Tasks: addressing and closing outstanding tickets. Ensure all the problems and feedback obtained during the technical preview have either been fixed or have mitigation plans in motion.
• Communicating Internally: everyone involved in supporting production should be notified about any problems found during the technical preview (and their solutions).
• Setting Up Alerts: we need to be aware of a problem in production as soon as it happens. This includes configuring Slack/Teams alerts, setting up pagers, and ensuring the on-call personnel is on the same page.

General Availability Release

The moment of truth we've prepared for so hard is close at hand. Users will not be as forgiving during a general availability (GA) release as during a tech preview. Problems might disrupt their work. That's why we went to such lengths to minimize the chance of the issues in the release. However, no amount of preparation will guarantee a successful release. So, we need to prepare for any problems that might arise.

Definition of Ready for Release

The software might be ready to release, but that doesn't mean we are prepared for the release. So, we must define and fulfill the GA release DoR checklist before going into the next step.

As always, every project has different needs. However, as a guideline, here're are some of the tasks that are considered necessary at Semaphore:

• Writing the Public Docs: public docs must be updated with the details of the new feature. In addition, it should be clear which plans have support for it.
• Skilling Up Internally: everyone in the company, especially support and customer-facing staff, should be briefed on the upcoming changes.
• Setting KPIs: based on technical preview utilization, we can set business objectives and key performance indicators (KPI) for the feature in question. This may include new revenue expected, utilization metrics, or feature adoption.
• Writing a Change Log: the change log should include everything that has changed since the last release.
• Setting Up Alerts: we need to be aware of any problem with the new features shipped as soon as it happens. This includes Slack/Teams alerts and pager notifications for the on-call personnel.
• Planning a Marketing Strategy: the marketing strategy communicates the changes and generates user interest. A good campaign can bring new customers or convert free users into paid ones.

Definition of Done for GA

With some luck, the release went smoothly, and users are happy with the new features. What now? Can we move to the next project? Not yet. This project is not done until all items in the DoD are checked off.

What's typically left at this point is doing some cleanup work and tidying up loose ends:

• Clean Leftover Tickets: after shipping the new features to GA, there might be some remaining low-priority tasks. They should be addressed or scheduled to be worked on later.
• CLI and API Support: provide support to the new feature via public APIs or the CLI tools because this makes it easier for users to access resources from the terminal and integrate them into their custom tooling. If this support is integral to the new feature, we can move this item to the DoR checklist.
• Do the Project Retrospective: The retrospective is one of the most critical phases of a project. It's a space for the team to discuss what went well and what can be improved for the next project. Also, it allows the team to identify potential pitfalls early and resolve conflict areas.

Conclusion

It's a mistake to think that coding is the hard part of software development. The real challenge comes when you open the floodgates and users pour in. Unless everything is prepared, issues can compound and overtake any engineering team. A formal definition of what constitutes being ready and done serves us to avoid rushing through a release.

Thanks for reading, and happy releasing!