Over a million developers have joined DZone.

A Brief Look At Some DataSift Platform Automation Elements

DZone 's Guide to

A Brief Look At Some DataSift Platform Automation Elements

· DevOps Zone ·
Free Resource

In the DataSift Operations team we try to automate as much as possible to leave us with more time for concentrating on more important tasks. I’d like to share some of the interesting snippets from our Opscode Chef recipes that might help you save some time too.

DNS Reverse Delegation for Active IPs

Reverse DNS delegations allow applications to map to a domain name from an IP address. Reverse delegation is achieved by use of the special domain names in-addr.arpa (IPv4) and ip6.arpa (IPv6).

For all IP address blocks that IANA allocates to the RIPE NCC, they also delegate the corresponding reverse DNS zones within the centrally-administered ‘in-addr.arpa’ and ‘ip6.arpa’ zones.

The RIPE NCC also publishes ‘zone fragments’. These are the parts of zones managed by other parties – the other Regional Internet Registries (RIRs), who share zone management of early registration networks.

— RIPE NCC’s Page on reverse delegation

Since we have our own blocks of IPs from RIPE we need to maintain a DNS server that is authoritative for our little bit of the in-addr.arpa namespace, and since Chef has all that information as part of its OHAI data gathering we might as well let Chef handle this for us.

First we define a series of databags with some data:

  "id": "0-16-172-in-addr-arpa",
  "subnet": "172.16.0",
  "hosts": { 
  	"1":	"misc-172-16-0-1.networksaremadeofstring.net",
  	"2":	"misc-172-16-0-2.networksaremadeofstring.net",
  	"3":	"misc-172-16-0-3.networksaremadeofstring.net",
  	"252":	"misc-172-16-0-252.networksaremadeofstring.net",
  	"253":	"misc-172-16-0-253.networksaremadeofstring.net",
  	"254":	"misc-172-16-0-254.networksaremadeofstring.net"

The default values in the databag are then overwritten by the FQDN of any hosts in Chef that match the search.

#Do our reverse zones
reverse_zones.uniq.sort.each do |zone|
  records = data_bag_item("bind", zone.gsub(".","-"))
  # At some point do a search if the IP address of a zone matches 
  # a host then over the records array with the name of this host 
  search(:node, "ipaddress:#{records['subnet']}*").each do |node|
    records['hosts'][node[:ipaddress].split('.')[3]] = node[:fqdn]
  t = Time.now
  template  "/var/named/master/db.#{zone}" do
    source "master_reverse_zone.erb"
    owner "named"
    group "named"
    mode 0644
    variables (
      :records => records['hosts'],
      :arpaAddr => zone,
      :serial => t.strftime("%Y%m%d%H"))#%H%M%S
    notifies :restart, "service[named]"

The Chef template is quite simple:

@	IN	SOA	ns1.networksaremadeofstring.net. operations.networksaremadeofstring.com.	(
	<%= @serial %>	;serial
	1800           ;refresh
	600            ;retry
	604800         ;expire
	86400          ;minimum
<%= @arpaAddr %>.	IN      NS      ns1.networksaremadeofstring.net.
<%= @arpaAddr %>.	IN      NS      ns2.networksaremadeofstring.net.
0               IN      PTR     nwrk.networksaremadeofstring.net.
<% @records.sort.each do |octet, ptr| %>
<%= octet %>		IN		PTR		<%= ptr %>.
<% end %>
255				IN      PTR		bcst.networksaremadeofstring.net.

Likewise, our forward DNS zones are also populated by Chef, so within 30 minutes of bootstrapping a node it is in our forward and reverse DNS zones ready without any human interaction.

Zenoss Monitoring

We rely on Zenoss to keep an eye on things we get for free, such as disk space, CPU metrics, etc., but obviously we don’t want to have to add hosts ourselves. That’d be crazy.

Instead, we perform a search against what we term a "base" role that defines the data center provider, the data center itself, and then the cab. Each role contains a series of attributes so any given server will have as part of its attributes its exact location, UPS / generator feeds, etc.

These base roles are then added to Zenoss as Location organizers and nested appropriately.

  ServerList = []
  FacilityList = []
  RackList = []
  search(:role, "name:base_*").each do |role|
    if role.default_attributes.has_key?("location") && (role.default_attributes["location"].has_key?("building") || role.default_attributes["location"].has_key?("city"))
      Facility = Hash.new
      Facility[:building] =  role.default_attributes["location"]["building"]
      Facility[:city] =  role.default_attributes["location"]["city"]
      Facility[:country] =  role.default_attributes["location"]["country"]
      Facility[:gps] =  role.default_attributes["location"]["gps"]
      Facility[:description] =  role.description
      Facility[:name] =  role.name
      #Chef::Log.info "Found a facility: ( #{Facility[:name]} ) #{Facility[:building]}, #{Facility[:city]}, #{Facility[:country]}, #{Facility[:gps]}"
    if role.default_attributes.has_key?("location") && role.default_attributes["location"].has_key?("rack")
      Rack = Hash.new
      Rack[:id] = role.default_attributes["location"]["rack"]
      Rack[:role] = role.name
      Rack[:description] = role.description
      Rack[:facility] = role.run_list[0].name
      #Chef::Log.info "Found a rack: #{Rack[:id]} in #{Rack[:facility]}"
  FacilityList.each do |facility|
    Chef::Log.info "Adding Facility #{facility[:name]}, #{Facility[:building]}, #{Facility[:city]}, #{Facility[:country]} to Zenoss"
    Name = facility[:name].gsub('base_','/')
    zenoss_zendmd Name do
          location Name
          description facility[:description]
          address "#{facility[:building]}, #{facility[:city]}, #{facility[:country]}"
          action :location
    #Lets see what racks are in this facility
    RackList.each do |rack|
      if(rack[:facility] == facility[:name])
        Chef::Log.info "Found a Rack #{rack[:id]} in #{rack[:facility]}"
        RackName = "#{Name}/#{rack[:id]}"
        zenoss_zendmd RackName do
          location RackName
          description rack[:description]
          address "#{Facility[:building]}, #{Facility[:city]}, #{Facility[:country]}"
          action :location
        #Now we find the servers in this cab
        search(:node, "role:#{rack[:role]}").each do |server|
          Chef::Log.info "\t\t Found server #{server["fqdn"]} / #{server.ipaddress} in environment #{server.chef_environment}"
          Device = Hash.new
          Device[:fqdn] = server["fqdn"]
          Device[:ipaddress] = server.ipaddress
          Device[:environment] = server.chef_environment
          Device[:location] = RackName
          if(server.chef_environment == "production" || server.chef_environment == "staging")
  #Now we do the heavyweight loading into Zenoss
  zenoss_lightweight_batchload "zenbatchloading devices" do
      devices ServerList
      action :run

The batch loading provider simply creates a text file compatible with the Zenoss zenbatchload process but then performs a diff to ensure that only new hosts are batchloaded, which ensures a quick Chef run because thousands of hosts aren’t batchloaded each run:

  #write the content to a temp file
  file "/tmp/chefzenbatch.batch.new" do
    owner "zenoss"
    mode "0600"
    content batch
    action :create
  file "/tmp/chefzenbatch.batch" do
      owner "zenoss"
      mode "0600"
      content "/Server/Linux\n"
      action :create
  execute "check for diff" do
      command "sort -o /tmp/chefzenbatch.batch.new /tmp/chefzenbatch.batch.new && diff /tmp/chefzenbatch.batch.new /tmp/chefzenbatch.batch.old | grep \"<\" | awk '{print $2 \" \" $3 \" \" $4}' >> /tmp/chefzenbatch.batch"
  #run the command as the zenoss user
  execute "Set file for later diff" do
      command "mv /tmp/chefzenbatch.batch.new /tmp/chefzenbatch.batch.old"

Arista Zero Touch Provisioning

I’ve written a lot about our use of Arista switches and, even though I love them, writing all the config manually can get boring, so why not have Chef do it all for me?

As mentioned in the Zenoss section above, we use Chef roles to define a rack cabinet and as part of the attributes for that role there will be a network section that looks a bit like this:

"network" => {
	"routing" => {
    		"network" => "",
    		"mask" => "27"
  	"switch" => {
		"name" => "ACS-AS-39",
		"uplink1" => {
			"ip" => "",
			"mac" => "aa:bb:cc:dd:ee:ff"
		"uplink2" => {
			"ip" => "",
			"mac" => 11:22:33:44:55:66"
		"management" => {
			"ip" => "",
			"mac" => 77:88:99:10:10:aa"

The ZTP cookbook can then create the relevant files for pulling down by the switch as detailed here.

I have to admit that I’ve not yet gone as far as to bootstrap a switch with Chef-Client and use the various Arista cookbooks.

Here’s one I made earlier…

There’s plenty of other stuff involved in our data center automation but the end result is that we can have entire cabs of servers (and their top of rack switches) up and running, monitored and in forward/reverse DNS by creating just one new Chef role and then booting them up.

At our scale that’s one hell of a time saver!


Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}