What Developers Can Learn About Ops From a Month in the Trenches [Video]

[This article was written by Tori Wieldt]

Who goes into a deployment hoping something will go wrong? Cornelia Davis, director of platform engineering in the Cloud Foundry team at Pivotal Software,says she did exactly that, because, well, “That’s how you learn stuff!”

Cornelia was a seasoned developer who knew a little about software ops. Very little, as she puts it. Familiar with well-established ops practices such as using jumpboxes and checklists, Cornelia felt she needed to learn more, so she spent a month with Pivotal’s production team to get up to speed. She shared what she learned in a Cloud Foundry Summit session last month called “A Developer’s Perspective on Cloud Foundry Operations: One Month in the Trenches.”

You can watch the entire presentation and check out her slides below, but here are some highlights I took away from attending her talk:

1. Do deployments during work hours

Contrary to the longstanding practice of updating software in the middle of the night or on weekends, modern ops teams are increasingly opting to update their software during (local) business hours. Using Software-as-a-Service (SaaS) architectures makes it much easier to roll back to a safe state (think minutes, not hours … or hours, not days). Also, if ops needs them, the developers who wrote the code are much more accessible during the day.

2. Categorize deployments by type

Different types of releases have different requirements and execution times. Pivotal divides releases into New Releases, Stemcell Upgrades (just the OS), and Manifest Only (change in topology). New Releases and Stemcell Upgrades can take hours, so the team starts them first thing in the morning. Manifest Only releases take just minutes, giving the ops team more flexibility on when to deploy them.

3. Continuously reduce deployment time

The Pivotal ops team is always optimizing for the shortest deployment times possible. The ops team saw that, during their deployments, compiling the packages took a long time. To speed things up, the team arranged its pipelines so that they compiled packages in advance. Instead of being compiled and then pushed, the already-compiled packages are shared using a shared package cache between staging and production. This technique can significantly reduce deployment time and should work for anyone using Cloud Foundry.

4. Use Checklists

Pivotal has checklists for pre-deployment, deployment, and post-deployment. Cornelia also suggested creating checklists for each type of release. (The last item on your checklist should be “Update Checklist.”) Finally, to make sure nothing gets lost, your code, documentation, and checklists should all be in one place. Pivotal, for example, checks everything into Github.

Before her time in the dojo with the ops team, Cornelia says she was “gleefully ignorant” about what ops does and the challenges it faces. Now she knows better. “It was an awesome month!” she told the Cloud Foundry Summit crowd.

You can access Cornelia’s slides here, and watch her entire presentation below: