DevOps: Flexible configuration management? Not so fast!

you’re in charge of establishing a department-wide deployment automation capability. your fellow developers are excited about it, and their managers are too. there is no shortage of ideas on how it might work:

• “let us create our own workflows!”
• “we should be able to configure our own servers.”
• “it should be able to deploy from nexus, artifactory, s3, or whatever we choose.”
• “we can finally use the app versioning scheme my team likes.”
• “my team should get to do parallel installs if we want”
• “we should have open apis so anybody can execute their own deployment solution.”
• “each team should be able to configure the middleware for their application’s needs.”

developers hate being told how to do things, so there is a general consensus that if you can make this deployment tool as flexible as possible, you’ll be able to build the best deployment automation system the world has ever seen.

sounds great, except that it’s totally wrong.

flexibility kills quality

drift is evil.  drift causes downtime and rollbacks.  flexibility creates drift.

i’ve been involved in data center migration projects where almost every server in a production farm was configured differently.  it’s amazing the application even worked!  on many other occasions we have rolled back code because the qa and prod configurations were so different that our testing failed to uncover critical bugs.  although these environments sound ridiculous, i’m confident that it describes a common scenario across enterprise environments.  i will also state that we had talented systems administrators managing the environments, unfortunately each one had the flexibility to manage the systems to their liking.

our initial investment in deployment automation (and what became devops) was largely driven by a need to eliminate drift and increase availability.  we knew automated deployments should be driven by data, and server instance data would be sourced from a cmdb.  however, we quickly realized that our cmdb schema allowed for configuration drift. this led to one of our first devops principles: don’t manage problems that you can eliminate .

eliminate drift with inflexible schema data .  tools from operations teams tend to be server or device centric and we wanted our deployment automation to be app and farm centric.  in other words, we wanted to deploy apps to a farm entity, where the server instances are attributes of the farm.  however, we found traditional schema for configuration data was very flexible.  the diagram below shows a typical farm with multiple instances, and each instance has an os version.  since the os version can be independently selected for each instance, the schema allows the ability to represent drift across the farm.  while architecting our app deployment cmdb (interestingly named deathburrito), we specifically did not want to manage farm configuration consistency.  we simply wanted a guarantee that our farm deployments did not have drift.

to achieve this we made a simple change to the schema that did just that – prevented the data from representing farm drift (picture below).  although you can incorrectly represent farm attributes, the data driven deployment is either 100% right or 100% wrong.

gratuitous flexibility and useful flexibility

eliminating schema flexibility to control drift is not that controversial since most people get it — and support it.  when you start limiting personal preference, man look out, people get really passionate over stupid things.  so we started communicating another one of our devops principles: flexibility is not always a good thing.

your deployment automation should start with inflexibility and provide flexibility as needed.  don’t get me wrong, we absolutely support innovation and the ability to empower our department with tools that enable creativity.  i often confuse the hell out of people by saying weird stuff like, “by limiting your flexibility, i can offer you more flexibility.”  and i actually mean it — because we focus on the flexibility that is actually valuable.  the objective is to distinguish between value-added flexibility and gratuitous flexibility, and eliminate the gratuitous junk.

• value-added flexibility can be represented by a middleware option between tomcat, jboss and glassfish.  each solution provides different features to the development team and they should have the ability to choose the best match (within reason) for developing to application requirements.  easy enough, there is value to the options.

• gratuitous flexibility can be represented by allowing multiple install directory variations for each tomcat app.  sysadmins usually have a preference and sometimes make it a very passionate preference.  although the configuration matters, it should support automation and security, not personal preference.  there is no inherent value gained by allowing your environment to have different install directories such as /opt , /app , /u01 .  in fact, allowing options creates complexity for install scripts, logging, permissions, service accounts, monitoring etc. pick one and restrict the rest.
  
  

one of the great things about automation is the ability to make the deployment platform deliver what you want, and fail what you don’t want.  it’s a platform that gives the devops team enforcement power in the it department that is rearly available.  like most organizations, you probably have many awesome design standards that are drafted, but in effect are just glorious shelfware documents.   automation empowers your ability to eliminate drift, control flexibility and operationalize the shelfware.

so back to my statement about limiting flexibility to offer more flexibility?  i will argue that by eliminating all the gratuitous variations, you can simplify environment complexity and eliminate the associated busy work and waste.  i also believe that eliminating the gratuitous variations will allow your devops teams to focus on delivering the value of predictable self-service deployments… real flexibility is the ability to provide your developer and test teams self-service deployments at any time, over weekends and around the clock.

source: http://skydingo.com/blog/?p=56