DevOps and Keeping the Boring Things Boring

I, For One, Welcome Our New Robot Overlords was the title of Mykel Alvis (@mykelalvis)'s session at the 2016 All Day DevOps Conference. He wasn’t trying to curry favor with the new robot rulers, ala Kent Brockman, but, instead, was evangelizing on the importance of precision in DevOps.

Mykel is the DevOps Coach at Cotiviti Labs, the development arm of Cotiviti. Every day, they handle healthcare data, which as you know happens to be some of the most regulated and protected data around. As Mykel said, “We don’t do cat pictures on the Internet.” A mistake, breach, security flaw, etc. could mean the exposure of privacy and financial data for millions of people. However, they are a large organization with exposure that needs to be managed. They have 200+ repositories, three to four deployments per day, 15 developers, and zero operators (sort of). Without the right discipline and rigor, costly mistakes could happen.

Hence, Mykel’s need for the robot overlords. As he noted during the conference session, robots have “lower utility, but greater predictability.” They do exactly what they are told, and they do it the same way, every time. By contrast, humans are superior to responding to unpredictable events. Mykel’s goal is to leverage the strengths of each. Cotiviti Labs has systems, policies, procedures, and a culture in place that supports the automation of tasks as much as possible to ensure mistakes are kept to a minimum and the humans can focus on the unpredictable. They are striving to make the boring things stay boring. They call it “The Labs Way” and here is how they do it.

• Treat everything as code. Everything. Infrastructure, data, everything. As an example, they perform pull requests to get status updates for executive reports.
• Test the code. 
• Perform formal releases. They build their code in very specific ways, with extremely rigid gatekeeping processes.
• Releases produce immutable artifacts. Once something has been released, they will never change it. They could, but they don’t.
• Keep everything. Not in a huge, unorganized file system. In an organized repository.
• Failing tests mean a failed build.
• Design your systems to be automated. They don’t do things that can’t be done automatically.
• Deal with those systems only through automation. This is probably the most challenging thing we do since it is extremely hard to convince developers to write their code to do this.
• Operations is developing. Operations is not a bucket for development. Operations is a development task. You treat it like it is real code.
• Because everything is code. Everything.
• All defects are defects of code.

To achieve this, they standardize, evaluate, test, discard the useless, and simply own it.

None of this comes easy. It challenges norms. It pushes boundaries. It mandates discipline. Your team has to understand that you have a way to do things. There is a reason, and it will reduce risks and make sure everything works. It will make the boring stay boring.

You can watch Mykel’s full presentation from the All Day DevOps conference here (30 minutes). The other 56 presentations from the All Day DevOps Conference are available online, free-of-charge here.