Over a million developers have joined DZone.

Configuration Management At Scale

Big environment, small team, no problem!

· DevOps Zone

Download “The DevOps Journey - From Waterfall to Continuous Delivery” to learn learn about the importance of integrating automated testing into the DevOps workflow, brought to you in partnership with Sauce Labs.

I had the opportunity to speak to Ben Abrams, Lead DevOps Engineer at Cloud Cruiser about configuration management.

Configuration management -- writing code to repeatedly and reliably configure systems the way you want them and that resolves divergence over time – has been around since long before the birth of modern CM tools, such as Chef, Puppet, Saltstack, Ansible, and the like. Where you used to have to rely on home grown solutions using ssh scripts and database queries to get information, the sophistication of modern CM tools makes it far easier for anyone needing or wanting to manage many systems.

This capability is especially important for a small team trying to manage thousands of servers across multiple environments – a pretty common scenario for large enterprises, and even many smaller ones (like us). Fortunately, by automating the entire process of resource management across platforms and the software that lives on this hardware, it does not matter if we have 10, 100, or 1,000 machines -- it's not much additional effort to manage their configuration, even as the number grows.

At Cloud Cruiser, we make use of a popular CM solution called Chef.  By leveraging Chef, we manage all configuration on a system and the applications running on hundreds of servers with little more effort than managing tens of servers would be. We can make any needed changes to systems, applications, databases, etc., regardless of how many instances we have. Having a robust query mechanism for retrieving configurations, and allowing us to determine the items on a system we wish to configure separately from what those values are, is extremely powerful. We can write the code once with sane defaults, then overwrite them at various levels, such as environment, role, etc., outside of the codebase to configure it as needed. This is a powerful abstraction and allows for many different patterns of implementing infrastructure as code. You can find some examples here: http://blog.vialstudios.com/the-environment-cookbook-pattern.

How about a real-world example? Here I am going to walk you through a real change that I needed to make recently. The change happened in an internal wrapper cookbook: https://github.com/CloudCruiser/ops_chef-cc_netuitive which I recently open sourced to show users how you can consume the upstream community cookbook (of which I happen to be the original author and am an active maintainer): https://github.com/Netuitive/chef-netuitive. The change is simple: we want to enable a new Netuitive collector for MongoDB. Normally this would pass through our CI process with jenkins but for this demonstration I will bypass this.

Diff of things changed:

```

$ git diff 574228a15e62de2dba95b6bc4b38ec414037e7c0 981876a4c15ec682c9524f75abb283cd34de0fe5

diff --git a/Berksfile.lock b/Berksfile.lock

index 90ca8ce..3430134 100644

--- a/Berksfile.lock

+++ b/Berksfile.lock

@@ -9,7 +9,7 @@ DEPENDENCIES

 

 GRAPH

   apt (3.0.0)

-  cc_netuitive (0.3.0)

+  cc_netuitive (0.4.0)

     netuitive (>= 0.0.0)

   netuitive (0.5.0)

     apt (>= 0.0.0)

diff --git a/attributes/default.rb b/attributes/default.rb

index e69de29..c6b8e09 100644

--- a/attributes/default.rb

+++ b/attributes/default.rb

@@ -0,0 +1,10 @@

+if node.roles.include? 'mongodb_server'

+  default['netuitive']['custom_collectors'] = {

+    'MongoDBCollector' => {

+      'enabled' => 'True',

+      'host' => '127.0.0.1:27000',

+      'metrics_blacklist' => '.*databases.*|.*metrics.repl.executor.shuttingDown.*|.*storageEngine.*|.*writeBacksQueued.*|.*mem.supported.*|.*tcmallocformattedString.*',

+      'simple' => 'True'

+    }

+  }

+end

diff --git a/metadata.rb b/metadata.rb

index 4aff54c..18fdeb2 100644

--- a/metadata.rb

+++ b/metadata.rb

@@ -8,6 +8,6 @@ long_description 'Installs/Configures cc_netuitive'

 source_url 'https://github.com/CloudCruiser/ops_chef-cc_netuitive' if respond_to?(:source_url)

 issues_url 'https://github.com/CloudCruiser/ops_chef-cc_netuitive/issues' if respond_to?(:issues_url)

 

-version '0.3.0'

+version '0.4.0'

 

 depends 'netuitive'


```

We can then upload the cookbook to the chef server:

```

$ berks upload cc_netuitive
Uploaded cc_netuitive (0.4.0) to: 'https://<MY_CHEF_SERVER>:443/organizations/<MY_CHEF_ORG>'

```

At this point, we can wait for it to converge or we can force it with a `chef-client` run. To illustrate what the change looks like, I will manually converge on one machine:

```

sudo chef-client

```

Here is the relevant output of it changing on the node:

```

     --- /opt/netuitive-agent/conf/collectors/MongoDBCollector.conf            2016-05-20 20:24:23.624265967 +0000
      +++ /opt/netuitive-agent/conf/collectors/.chef-MongoDBCollector.conf20160520-26501-ugiota           2016-05-20 20:24:23.624265967 +0000
      @@ -1 +1,8 @@
      +################################################################################
      +# Config File is chef managed, local changes will be overwritten
      +################################################################################
      +enabled = True
      +host = 127.0.0.1:27000
      +metrics_blacklist = .*databases.*|.*metrics.repl.executor.shuttingDown.*|.*storageEngine.*|.*writeBacksQueued.*|.*mem.supported.*|.*tcmallocformattedString.*
      +simple = True

```


I waited to let the others automatically converge, but often you will use something like `knife ssh` or a push job server to push these changes immediately.

Verifying the change went to the expected nodes:

```

$ knife ssh 'role:mongodb_server' -a ipaddress 'ls -l /opt/netuitive-agent/conf/collectors/MongoDBCollector.conf'
10.55.152.176 -rwxr--r-- 1 root root 437 May 20 20:53 /opt/netuitive-agent/conf/collectors/MongoDBCollector.conf
10.55.151.84  -rwxr--r-- 1 root root 437 May 20 21:10 /opt/netuitive-agent/conf/collectors/MongoDBCollector.conf
10.55.150.95  -rwxr--r-- 1 root root 437 May 20 21:08 /opt/netuitive-agent/conf/collectors/MongoDBCollector.conf

```

By using the right tools, we were able to author a change that would work on multiple machines in a very short amount of time, which allowed us to get back to doing more interesting things than remoting into to a bunch of machines and editing config files. Config management is a must in any large environment, and I encourage anyone that feels like it's too scary to get started to contact me at bena@cloudcruiser.com and I will be more than happy to help you get going.





Discover how to optimize your DevOps workflows with our cloud-based automated testing infrastructure, brought to you in partnership with Sauce Labs

Topics:
configuration management ,config management ,devops ,continuous integration

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}