I recently came across this blog post by Mark Needham that highlights a common problem we hear from our users: even when they use an automated deployment tool like Puppet or Chef, they still run into environment and configuration drift problems. Mark does a great job explaining how this could happen if you’re using Puppet so I’m not going to repeat it here. But suffice to say that when drift does happen, finding it is a slow and painful process and it has the potential to trigger disastrous results:
“…a version got bumped and something elsewhere stopped working and it took us quite a while to work out what had changed.”
While configuration management tools like Chef or Puppet are a critical component of the DevOps Toolchain for building scalable infrastructure, they are only one piece of the automation puzzle. These tools do a great job helping to prevent drift but drift inevitably finds a way to sneak in. Sometimes it’s for the reasons that Mark talked about in his post. Sometimes it’s because humans are still a part of the IT process and where there are humans, chaos (in this case changes that go undocumented or unnoticed!) will ensue .
To ensure that the actual state of your environment is what you think it is, you need to verify it regularly by checking for any deviations from its previous or intended state. This step allows you to close the feedback loop of your automation system.
Just like any good security system which requires firewalls for prevention as well as intrusion detection to catch any sneaky viruses that manage to get in, a robust automation system should include both an automated deployment platform and a drift detection solution . Only then can you be confident that you know the actual state of your environment.