7 DevOps Tips From Gene Kim

Over the past few weeks, New Relic has held a series of DevOps roadshows in cities around the country, headlined by DevOps expertGene Kim, co-author of The Phoenix Project, and founder of Tripwire. Gene told attendees about why DevOps matters so much, how DevOps is finding a home in even the largest enterprise, and the importance of the ops part of DevOps. And he shared his own hard-earned tips on how to successfully implement DevOps in all kinds of organizations.

DevOps Tip #1: Deploy smaller changes more frequently

According to Gene, practicing fast, easy, multiple deploys “frees us from learned behaviors that deploys are complex and painful. Our common experience shows us that the larger the change we promote into production, the larger the crater it makes when it lands.” To make smaller deployments work, it’s important to

• Decouple feature releases from code deployments.
• Deploy features in a disabled state, using feature flags.
• Require all developers check code into trunk daily (at least).
• Practice deploying smaller changes, which dramatically reduces risk and improvesMTTR.

DevOps Tip #2: Fearlessly enable testing in production

The conventional wisdom about testing in production is “that’s what devs do to ops because they hate us.” But Gene says it’s actually a game changer:

• When we can continuously deploy disabled code into production, we can test under production-like loads long before the feature release.
• When we build our code and environments with resilient design patterns, we can rehearse and simulate large-scale failures so we’re ready (à la Chaos Monkey)

As Adrian Cockcroft famously said, “Rehearsals are critical, because otherwise, people don’t know what to do. Yes, fire drills are annoying when you’re standing in the parking lot, waiting for it to be over. But when the building is actually on fire, it feels good when you’re standing in the parking lot and everyone is there with you, with no one trapped in a smoke-filled room.”

DevOps Tip #3: Build a one-step environment-creation process

• Make environments available early in the development process.
• Make sure dev builds the code and environment at the same time.
• Create a common dev, QA, and production environment-creation process.

DevOps Tip #4: Synchronize the schedules of dev and ops

According to Gene, it’s important to schedule updates in the middle of the day, when both groups are working. Not nights, weekends, and holidays when the devs are out of the office and ops folks are bearing the entire burden. 

DevOps Tip #5: Change the way sprints work

“If I had a magic wand,” Gene says, “I’d change the agile sprints and definition of ‘done.’ At the end of each sprint, we must have working and shippable code, demonstrated in an environment that resembles production.”

DevOps Tip #6: Version control is more important for ops than dev

“Ops using version control is more important to performance than devs using version control,” Gene says. That’s because there are more configuration settings in the environment than there are in the code, and “most failures have to do with environment errors, not code errors.”

DevOps Tip #7: Allocate 20% of cycles to technical debt reduction

According to Gene, if you don’t pay your 20% debt-reduction tax, you’ll end up spending 100% to fix the accumulated issues. While management at time-stressed companies may balk at what they see as overhead, you don’t just need a product, you need a product that works and doesn’t crash all the time. Make sure the bean counters know how long it will take to fix these problems later and how it could hurt the business … you never want to be in a position where you can’t ship products. As Gene explains, “It’s the price you pay to get great dev and ops.”