Sensu Monitoring Handler Sets

In past articles we have covered some of basics of Sensu handlers. A nice feature we haven't touched on yet is handler "sets". Handler sets were added around v0.9.2 and can be quite useful for saving time when modifying your handler.

For example, consider you have a standard set of handlers that you assign to most of your checks — pagerduty, irc, campfire. Now, suppose you want to add the GELF (graylog2) handler to all of your monitors as well. If each of your checks is defined as such:

{
  "checks": {
    "all_disk_check": {
      "notification": "Diskspace Too Low",
      "command": "PATH=$PATH:/usr/lib64/nagios/plugins:/usr/lib/nagios/plugins  check_disk -w 25% -c 15% /",
      "subscribers": [ "all" ],
      "interval": 60,
      "handlers": ["pagerduty", "irc", "campfire"]
    }
  }
}

… you would need to modify every check's "handlers" attribute to include your new "gelf" handler. If you have a lot of checks this can be a little bit of a burden.

Handler sets save a lot of time in this situation. If you instead define your checks like this:

{
  "checks": {
    "all_disk_check": {
      "notification": "Diskspace Too Low",
      "command": "PATH=$PATH:/usr/lib64/nagios/plugins:/usr/lib/nagios/plugins  check_disk -w 25% -c 15% /",
      "subscribers": [ "all" ],
      "interval": 60,
      "handlers": ["default"]
    }
  }
}

And then define your handlers on your Sensu server like the following. Notice we have converted the 'default' handler to a set.

"handlers": {
      "default": {
        "type": "set",
        "handlers": ["pagerduty", "irc", "campfire", "gelf"]
      },
      "pagerduty": {
        "type": "pipe", 
        "command": "/etc/sensu/handlers/pagerduty"
      },
      "irc": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/irc"
      },
      "campfire": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/campfire"
      },
      "gelf": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/gelf.rb"
      }   
    },

You only need to modify the "default" handler set to add the new "gelf" handler and no need to change any of your check definitions.

It's a simple feature, but it can save a bunch of time. You could also setup a handler set specifically for metrics and another set for notifications. In the case of metrics, you could easily ship metrics to multiple systems – Graphite, Librato, Cube, etc. And adding a new system would be as simple as creating the handler and adding it to the handler set. Happy Sensu'ing.