Evolving Enterprise Infrastructure Using Chef
People consider Chef as a configuration management tool. You specify the state using the infrastructure DSL that Chef provides. You just apply yoru configurations, and you get reproducible infrastructure. Here are the main topics:
- Chef is a modular infrastructure management toolkit. There are several way to look at chef: At its core chef is an API layer over day to day system administration. All other Chef components leverage this API.
- chef-client and chef-solo are two binaries that execute Chef scripts in client/server or serverless chef setups.
- Knife is a command line interface to chef-managed infrastructure.
All these things together give you traditional configuration management style workflows via a Chef managed infrastructure.
Now, there are ample one-off tasks that happen only once in a server's life time. Incident management, audits are another example. There are configurations related to third party hosted solutions. Enterprise's own service deployments. Most of these workflows don't model well in conventional configuration management style infrastructure management.
For such workflows chef can be easily extended. Chef provides multiple avenues to extend its capabilities:
- Chef internally distributed as a set of different libraries.
At its core chef has Mixlib::ShellOut , which is a separate rubygem. This is the component that provides portable command execution. Since 90% of the time configuration management systems dispatch command, this is the component that can be called the heart of Chef. If you have lots of shell-script based tools you can easily port them with mixlib-shellout. I have rarely seen bash scripts that check the return value of every command it invokes. It's more difficult to maintain long shell scripts. With mixlin-shellout its much easy to express safe guard measures.
- Chef uses mixlib-json to convert JSON data. This is
also a separate gem, and you can independently use it. You might be
wondering why this is important. JSON is the principle marshalling
solution in chef and most other modern systems. mixlib-json facilitates
the inclusion of a json conversion library from the available ones. You
can use mixlib-json independently to grab raw data and serialize them
and feed into other programs in a relatively portable way.
- mixlib-cli and mixlib-config .
Mixlib::CLI lets you easily parse command line arguments. While
mixlib-config lets you read, access override configuration values. Again
both of them are distributed as separate gems. You can use these two
and build your custom command line tool chains rapidly.
- knife-plugins. Knife can be extended using plugins. It's a more formal way of extending knife via sub-commands. The best part of Chef is that it's pure ruby everywhere, so you can pretty much use any rubygem (fog, aws-sdk, openvz) to have any cloud based infrastructure automation tool. There are many cases where your configurations are not applied against a server (linux or windows etc). They can be hosted solutions. Like IAM entries inside AWS, or ultraDNS based resource records, pagerduty escalation policies, salesforce settings, campfire notifications etc. The knife-plugin ecosystem along with the other libraries provide an elegant way to develop such tooling.
You can use an organization that can start automating smaller chunks of work and then refactor them into portable recipes so that a conventions-configuration management style infrastrcuture is possible. Chef's philosophy emphasizes resources as the primary building block and idempotency is a desired criteria. It is desired because there is an implicit assumption of repetetive invocation of chef runs within a system. Once you have your infrastructure to this level, there is lot magic that can be leveraged from chef, and also a lot other tools that work on top of chef can be adopted.
- Chef is also available as a single monolithic operating system package. Its called omnibus
installers. They are present for most of the popular operating systems. For instance,
the linux installer bundles everything on top of glibc. So you have an
isolated, battle-tested working Chef binary that you can introduce to
your system without inflicting major drifts.
This is crucial, as its lets you minimize iterative migration of manual or
semi-automated infrastructure under full configuration management. Or,
if not traditional configuration management, still reproducible and
automatic infrastructure. Start with only the omnibus installer and then
iteratively automate the most repetitive and time consuming tasks.
Whatever it is.
- Once you have adopted chef significantly you'll realize the volume
of chef-scripts is large. This is a by-product of treating large
infrastructure as code. Code needs maintenance. And the maintenance can
be easy if appropriate software development practices are applied.
Having tests, keeping your chef scipts under a CI adds tremendous value.
But the entire paradigm depends upon how easy it is to write tests against
chef script. Last year has seen tremendous efforts on this aspect.
Now we have decently effective unit testing, functional testing, and
integration testing frameworks (chefspec, cucumber-chef, minitest-handler
are examples respectively). Chef allows you to treat infrastructure as
any other functional domain, like mobile or banking or health care. Any
developers can work with domain experts to develop neat chef scripts
(an operations guy is an example of a domain expert).
Together these things makes it much easier for an enterprise to iteratively improve and evolve its infrastructure to accommodate newer innovations (like cloud adoption, automatic failovers, disposable servers) with greater agility using Chef....