The Poor Man's Monitoring and Recovery

When you are not willing to pay for monitoring services like AppDynamics or UptimeMonitor, you still have a cheap Shell script-based solution.

Shells scripts and Slack are your friends.

Scenario

You have a Tomcat web server running. You want to take action if it dies and notify a group of people about the action.

Tools

Crontab, Shell script, and Slack.

1. Set Up Crontab

In your Linux server, add a new crontab crontab -e and edit the following check every minute:

*/1 * * * * /opt/scripts/check.sh

Now, restart your cron service service crond restart.

2. Create the Check

Create a new Shell script in /opt/scripts/check.sh with the following code:

#! /bin/sh
SERVICE=/etc/init.d/tomcat8

if $SERVICE status | grep -q "not running"; then
    $SERVICE start
    /opt/scripts/slackpost.sh "https://hooks.slack.com/services/your-hook" "#monitor" "Tomcat Automatic Restart"
fi
if $SERVICE status | grep -q "stopped"; then
    $SERVICE start
    /opt/scripts/slackpost.sh "https://hooks.slack.com/services/your-hook" "#monitor" "Tomcat Automatic Restart"
fi

This will find out if the service crashed or was not started, and it will start it. After staring the service, it will send a notification to your Slack channel (you can strip this piece if no slack desired).

3. Notify Slack

Create another script in /opt/scripts/slackpost.sh with the following content:

#!/bin/bash

# Usage: slackpost "" "" ""

webhook_url=$1
if [[ $webhook_url == "" ]]
then
        echo "No webhook_url specified"
        exit 1
fi

shift
channel=$1
if [[ $channel == "" ]]
then
        echo "No channel specified"
        exit 1
fi

shift

text=$*

if [[ $text == "" ]]
then
        echo "No text specified"
        exit 1
fi

escapedText=$(echo $text | sed 's/"/\"/g' | sed "s/'/\'/g" )
json="{\"channel\": \"$channel\", \"text\": \"$escapedText\"}"

curl -s -d "payload=$json" "$webhook_url"

Finally, use chmod +x /opt/scripts/*sh to make both scripts runnable. 

Note: This code can be found here. 

Happy coding!