Writing Better Production Readiness Checklists
When considering reliability tools, we may overlook checklists; however, these simple to-dos can aid your business in their efficiency and ensure a successful product launch.
Join the DZone community and get the full member experience.Join For Free
When we think of reliability tools, we may overlook the humble checklist. While tools like SLOs represent the cutting edge of SRE, checklists have been recommended in many industries such as surgery and aviation for almost a century. But checklists owe this long and widespread adoption to their usefulness.
Checklists can help limit errors when deploying code to production. In this blog post, we’ll cover:
- How to make a production checklist
- Why production checklists are helpful
- Keeping your checklist up to date
How to Make a Production Checklist
Production checklists should be holistic. They should cover everything from launch logistics to contingency plans for failure. Let’s break down what you’ll need for a thorough checklist.
1. Determine the Service Level of What You Are Launching
To determine how thorough your checklist should be, consider what level of reliability your customers need. You may be tempted to be as comprehensive as possible with every checklist, but that costs time and may be unnecessary. At Mercari, the service level is determined based on the service’s SLO. Services that are critical to business success are scrutinized more than niche services.
2. Map Out All the Checklist Areas
List all major components of your service. These components may be under the ownership of various teams. For example, you’ll likely need to consult server management teams, testing teams, and many others. It’s important to know as soon as possible whom you’ll need to consult. Some areas to consider include:
- Server-Side: What machines will this service run on? If you’re cloud-based, will your plan cover the new service’s load?
- Client-Side: Is your service usable for all potential clients?
- Monitoring: Do you have ways of collecting data from your new service?
- Growth: Do you have a roadmap for how you will maintain or improve the service going forward? What if usage increases? What if you need to expand functionality?
- Dependencies: What other in-house and third-party services does your service depend on? Will they integrate smoothly?
- Testing: Has the new service been tested in an environment mirroring production?
- Security: Will your new service pass your security audits?
- Reliability: What level of reliability will your users expect? Do you have a plan for when you are unable to meet these expectations?
- Incident response: What will you do if an incident causes service interruption or degradation? Do you have runbooks to cover these incidents?
- Legal: Do you have an SLA that guarantees availability? Does this service deal with personal information that must be kept secure?
- Logistics: What is the launch schedule? What resources will you need?
3. Prepare the Checklist Items
Each of these areas contains many issues and requires data to answer. Your checklist should ask for each piece of data. Here’s an example of how certain sections could be broken down:
You may also want to include information on who to consult to check off each item, and the timeframe for being able to check it. Build your checklist and check items off as development progresses. Double-check to ensure that items are ready to go, and right before launch, do a final check through the whole list, just in case.
Keeping the Checklist in Check
As you develop, you’ll likely find more areas you want to vet prior to launch. To keep your checklist from becoming too long, you’ll need a system to make sure new additions are helpful. At Google, teams have two criteria for adding an item to the checklist:
- “Every question’s importance must be substantiated, ideally by a previous launch disaster.”
- “Every instruction must be concrete, practical, and reasonable for developers to accomplish.”
You can determine criteria based on the service level you have assigned. It’s better to have an unnecessary item than to lack one that is essential. It’s okay to start with a big checklist and remove items after each launch as needed
Why Are Production Checklists Helpful?
Production checklists can seemingly add overhead to engineers’ jobs. However, the upfront work can save teams from future problems and ensure a successful launch. Production checklists help:
- Remove the cognitive toil of having to remember everything
- Identify possible problems ahead of time
- Prepare resources ahead of time
- Motivate development to complete necessary items
- Prioritize key requirements over unnecessary additions
- Ensure contingency planning, improving reliability
- Keep everyone in the loop throughout development as a centralized progress meter
How to Keep Your Production Checklist Up to Date
You will need to review and revise your checklists periodically to keep them useful. Be sure to revisit them at these times:
When Development on a New Service Starts
When mapping out a new service, consider which production checklist to use when it launches. Based on the type of service and service level, find the closest checklist you have. Review it to make sure it follows the processes and architecture you currently use. Add any service-specific requirements as you develop.
After a Launch
Take a look at the production checklist after you launch the new service. Were there any problems with the launch? Could they have been checked for beforehand? Look for checklist items that were misunderstood and filled out incorrectly. Revise these items to ensure the checklist lines up with the reality of development.
After an Incident
If an incident impacts the new service, see if any of the contributing factors could have been addressed with the checklist. If so, try to capture those items on future checklists. This task can be incorporated into your incident retrospectives.
As Part of Regular Review Cycles
Set a schedule to review tools like runbooks and production checklists. Make sure to invite all team members who will be required to use these runbooks or checklists. Each of these people can provide insight on what to improve moving forward.
Published at DZone with permission of Emily Arnott. See the original article here.
Opinions expressed by DZone contributors are their own.