Over a million developers have joined DZone.

Integrating etckeeper with Logentries & Chef

· Performance Zone

Discover 50 of the latest mobile performance statistics with the Ultimate Guide to Digital Experience Monitoring, brought to you in partnership with Catchpoint.

[This article was written by Jimmy Tang] 

When working within a team to maintain system infrastructure, properly documenting and communicating changes made to configuration files within /etc is fundamental to preventing knowledge gaps throughout your team.

integrating-etckeeper-and-chef-with-le (1)

While version control tools like git are helpful in tracking standard changes to a code base, git doesn’t capture metadata important to /etc like permissions of/etc/shadow. To address this need, we’ve been exploring etckeeper – a small the version control application developed by Joey Hess (of Debian fame) for recording packaging installed or removed from /etc. While working with etckeeper, it became apparent that tracking changes over time in context of other events occurring within our systems would be useful and easily accomplished with Logentries.

etckeeper has a number of execution stages and provides a mechanism for hooking into each stage via a set of simple scripts. Using the Logentries API, you can easily send etckeeper metadata at etckeeper’s post-install and commit phases. All you’ll need to start with is a token-based log in your Logentries account.

During the post-install phase we use a hook stored in/etc/etckeeper/post-install.d/40send-to-logentries. You’ll need to insert the correct token into the following script.

#!/bin/sh
set -e

pl="/var/cache/etckeeper/packagelist"
dirty="/var/cache/etckeeper/dirty"
host=`hostname`
inet=`ip -4 address show dev eth0 | grep inet | awk '{print $2}'`
appname=etckeeper
NC_OPTS="-w 3"

token="TOKEN_FROM_LOGENTRIES"
API=api.logentries.com
PORT=10000

# run this if /etc is not clean
if etckeeper unclean; then
    message="Logging changes in /etc after $HIGHLEVEL_PACKAGE_MANAGER run"

    echo "$token appname=$appname host=$host action=notification stage='post-install' message='$message' inet='$inet'" | nc $NC_OPTS $API $PORT
    # list changed files
    git diff --name-status > $dirty
    while read -r line
    do
        f=$line
        echo "$token appname=$appname host=$host action=config stage='post-install' change='$f' inet='$inet'" | nc $NC_OPTS $API $PORT
    done < "$dirty" fi if [ -e $dirty ]; then     rm -f $dirty fi # Installed/removed packages if [ -e $pl.pre-install ]; then     message="Installed or removed packages"     echo "$token appname=$appname host=$host action=notification stage='post-install' message='$message' inet='$inet'" | nc $NC_OPTS $API $PORT     (     etckeeper list-installed | diff -U0 $pl.pre-install - | tail -n+4 | egrep '^[-+]' || true     ) > $pl.diff
    while read -r line
    do
        name=$line
        echo "$token appname=$appname host=$host action=package stage='post-install' change='$name' inet='$inet'" | nc $NC_OPTS $API $PORT
    done < "$pl.diff"
fi

if [ -e $pl.diff ]; then
    rm -f $pl.diff
fi

During the commit phase we execute this hook in/etc/etckeeper/commit.d/10send-to-logentries. Again in the following script you will need to insert the correct token.

#!/bin/sh
set -e

dirty="/var/cache/etckeeper/dirty"
host=`hostname`
inet=`ip -4 address show dev eth0 | grep inet | awk '{print $2}'`
appname=etckeeper
NC_OPTS="-w 3"

token="TOKEN_FROM_LOGENTRIES"
API=api.logentries.com
PORT=10000

# run this if /etc is not clean
if etckeeper unclean; then
    message="Logging changes in /etc after $HIGHLEVEL_PACKAGE_MANAGER run"

    echo "$token appname=$appname host=$host action=notification stage='commit' message='$message' inet='$inet'" | nc $NC_OPTS $API $PORT
    # list changed files
    git diff --name-status > $dirty
    while read -r line
    do
        f=$line
        echo "$token appname=$appname host=$host action=config stage='commit' change='$f' inet='$inet'" | nc $NC_OPTS $API $PORT
    done < "$dirty"
fi

if [ -e $dirty ]; then
    rm -f $dirty
fi

Once the above hooks are deployed to your hosts and can access api.logentries.com, any state changes and package state changes in /etcwill be sent to Logentries in real-time.

...
appname=etckeeper host=x01 action=package change='-dstat 0.7.2-4' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=config change='Dbash_completion.d/darcs' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=package change='+dstat 0.7.2-4' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=package change='-dstat 0.7.2-4' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=package change='+socat 1.7.2.4-1' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=package change='-socat 1.7.2.4-1' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=notification message='Installed or removed packages' inet='192.168.1.70/24'
...

Once your etckeeper metadata is being sent to Logentries, there are several useful things you can do, including:

  • Building alerts to notify teams of new changes
  • Creating custom tags to easily spot specific changes
  • Build dashboards to visualize config changes over time, compared to systems behavior

If you use Chef, we developed an etckeeper_wrapper cookbook which extends the etckeeper-cookbook, providing a number useful features for deploying etckeeper and a handler for the chef-client to execute etckeeper at every chef-client run. This provides a small safety net for the ops team to revert changes if things go wrong. The etckeeper_wrapper cookbook can be found here athttps://github.com/jcftang/etckeeper_wrapper.

Is your APM strategy broken? This ebook explores the latest in Gartner research to help you learn how to close the end-user experience gap in APM, brought to you in partnership with Catchpoint.

Topics:
performance ,version control ,monitoring ,integration ,infrastructure ,logentries ,etckeeper

Published at DZone with permission of Trevor Parsons, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}