Open Source Application Monitoring: Catching Exceptions

imagine if you will, you are working on a new critical application and you need to write the output of your process calculations to a file. simple stuff, a few lines of java later you have...

file file = new file("myfile.txt");
try {
  boolean filecreated = file.createnewfile();
  log.debug("filecreated = " + filecreated);
} catch (ioexception e) {
  log.error("could not create file",e);
}

you even remembered to log the exception to the logs just in case there was a problem in production. a few weeks later the code ships and works perfectly for weeks until one day the network mount disappears and the application starts to throw exceptions.

your applications logs then fill up with the exception message and stacktrace but no one realizes there is an issue until an angry customer rings up complaining that they never received their report.

a far worse scenario is that the exception occurs in production but the development staff decide that it is a “good exception” and that the best course of action is to ignore it. forever! well until the new guy starts and they have to explain that it is a “good exception”, and so are the following 600 exceptions.

i remember when i first heard the term “good exception”, i was working for startup in london over ten years ago. i was new to the company and the first phase of the application was already in production as part of a critical beta phase of the product. each morning a developer would have to be in the office ready to deal with any issues that may arise from 6am.

one cold december morning i was in the office and as part of the morning grind i was going through a checkout of the application. checkpoint number 27 was “check application logs”. no more detail, so i jumped on to the application server and started to tail the logs and to my dismay hundreds of exceptions were being logged in realtime.

i spent the next hour trying to work out what was wrong with the application and what had changed to cause such an exception storm in production. at around 8:00am one of the developers who had the longest tenure in the team arrived in and calmly pointed out “oh those, those are good exceptions, you can ignore them. they occur the day after a billing cycle due to a bug in one the core components”.

key lesson; exceptions should be exceptional, if you get an exception in production you need to deal with it.

exceptional workflow

exceptions are part of both the development process and the application monitoring process. an ideal flow is that once an alert is generated in production it is fed back into the development process as a potential bug fix or improvement. the key is to provide adequate monitoring of exceptions in production and to provide sufficient feedback into the development team.

how many of the applications you have worked on have had anything more than log level or log scrapping exception monitoring?

how many development processes have you seen that link production exception to bug fix and strive to fix as many exceptions as possible?

how many “good exceptions” were written to your logs in production since you started reading this post?

baking exception monitoring

personally i think one of the reasons for poor infrastructure in critical areas like this is down to the way different parts of the organizations are structured. in many large teams people are dedicated to different function of the application lifecycle. developers are generally focused on the application business requirements and have unforgiving deadlines. support teams have deadlines of a different type. they also tend to support many applications across a range of functions.

with the advancement of the devops movement these communities are starting to join forces and work on the infrastructure behind the applications. so one problem is certainly being addressed and will start to become more and more widespread in the next 2-3 years the other major factor is tool support. how many good modern tools are available for application monitoring that are quick to use and onboard? there are a number of interesting commercial startups in this space at the moment, airbrake for example is used by a number of corporation to add monitoring support to their application.

airbrake offers rich functionality but also supports almost all popular languages in it’s api arsenal. however it is hosted on their servers and this deployment configuration will not suit a large majority of application developers who build bespoke software for internal clients and are forbidden to publish information external regardless of content. interestingly enough there is an open source alternative to airbrake called errbit which is compatible with the airbrake api.

it’s a ruby on rails application that can be easily installed on your local server or for the purpose of this blog i put it up on heroku mainly for ease of use. once you have installed errbit you can quickly post exceptions and stacktraces to the server and it has some basic workflow for your support staff to monitor and deal with the exception. it also has integration with some of the most popular issue trackers however there is currently no jira support.

installing errbit

this was the first time i used heroku for anything even though i had heard great things. i had an account but it was unverified something that i over looked when i did my first installation. errbit needs mongodb and to use mongodb with heroku you need to verify your account with a credit card. this surprisingly stop my application working for a while and it took me ages to notice the small error message in the install script. you have been warned!

to install the application you need to follow the simple steps from the github page https://github.com/errbit/errbit (you will need git and ruby installed locally)

clone the repository

git clone http://github.com/errbit/errbit.git

create & configure for heroku

gem install heroku
heroku create example-errbit --stack cedar
heroku addons:add mongolab:starter
cp -f config/mongoid.mongolab.yml config/mongoid.yml
git add -f config/mongoid.yml
git commit -m "added mongoid config for mongolab"
heroku addons:add sendgrid:starter
heroku config:add heroku=true
heroku config:add errbit_host=some-hostname.example.com
heroku config:add errbit_email_from=example@example.com
git push heroku master

seed the db

heroku run rake db:seed

pretty quick, well once you have a validated heroku account. once completed simply type

heroku open

and your new errbit install should be running. my instance is at ebit.herokuapp.com and you can use anyone@anyone.com/password to login

once you have installed errbit you will need to configure your users and whatever applications you plan to monitor. again straightforward, clicking “add a new app” button will bring you to configuration screen and once you create the application record you will get the important application id you will need this later when publishing exceptions

publishing exceptions from java

as i mentioned earlier errbit is compatible with all the language apis that airbrake provide and luckily for me there is an actively developed api for java available at http://github.com/airbrake/airbrake-java . this will allow you to send exceptions from you java server appications, mobile applications and desktop clients. to start using it with maven add the following dependencies to your pom file

<project>
    <dependencies>
        <dependency>
        <groupid>io.airbrake</groupid>
        <artifactid>airbrake-java</artifactid>
        <version>2.2.0</version>
        </dependency>
    </dependencies>
</project>

once i imported the libraries i saw a slight problem in how to override the url for communicating with the backend server. in the airbrakenotifier class, which is responsible for calling the server side rest api, the url for airbrake is hardcoded whereas i needed to override it for errbit. a quick solution was to create a new errbitnotifier class which takes the base url a construction argument.

import airbrake.airbrakenotice;
import airbrake.noticexml;

import java.io.*;
import java.net.*;

public class errbitnotifier {

    private final string baseurl;

    public errbitnotifier(string baseurl) {
        this.baseurl = baseurl;
    }

    private void addingproperties(final httpurlconnection connection) throws protocolexception {
        connection.setdooutput(true);
        connection.setrequestproperty("content-type", "text/xml");
        connection.setrequestproperty("accept", "text/xml, application/xml");
        connection.setrequestmethod("post");
    }

    private httpurlconnection createconnection() throws ioexception {
        return (httpurlconnection) new url(string.format("http://%s/notifier_api/v2/notices", baseurl)).openconnection();
    }

    private void err(final airbrakenotice notice, final exception e) {
        e.printstacktrace();
    }

    public int notify(final airbrakenotice notice) {
        try {
            final httpurlconnection toairbrake = createconnection();
            addingproperties(toairbrake);
            string topost = new noticexml(notice).tostring();
            return send(topost, toairbrake);
        } catch (final exception e) {
            err(notice, e);
        }
        return 0;
    }

    private int send(final string yaml, final httpurlconnection connection) throws ioexception {
        int statuscode;
        final outputstreamwriter writer = new outputstreamwriter(connection.getoutputstream());
        writer.write(yaml);
        writer.close();
        statuscode = connection.getresponsecode();
        return statuscode;
    }

}

perhaps the airbrake api could potentially allow for custom configuration of the url in the next revision. once you have created a new errbitnotifier you can start publishing exceptions. going back to our previous example

import airbrake.airbrakenotice;
import airbrake.airbrakenoticebuilder;
import org.apache.log4j.logger;

import java.io.file;
import java.io.ioexception;

public class testexception {

    private static org.apache.log4j.logger log = logger
            .getlogger(testexception.class);

    public static void main(string[] args) {

        file file = new file("h://myfile.txt");
        try {
            boolean filecreated = file.createnewfile();
            system.out.println("filecreated = " + filecreated);
        } catch (ioexception e) {
            log.error("could not create file",e);
            airbrakenotice notice = new airbrakenoticebuilder("b4f7cb2020b2972bde2f21788105d645", e, "prod").newnotice();
            errbitnotifier notifier = new errbitnotifier("ebit.herokuapp.com");
            notifier.notify(notice);
        }

    }
}

this code will throw an ioexception (well at least on my computer, since i don’t have a h drive!) and the exception will be seen on the errbit console it has the ability to spot duplication of exceptions and you can set it up to email you when the exception is generated.

also the airbrake api has log4j appender support but it is tied to the airbrake public url and i have left it out of the post. however it can be turned on by the following log4j configuration example

log4j.rootlogger=info, stdout, airbrake

log4j.appender.stdout=org.apache.log4j.consoleappender
log4j.appender.stdout.layout=org.apache.log4j.patternlayout
log4j.appender.stdout.layout.conversionpattern=[%d,%p] [%c{1}.%m:%l] %m%n

log4j.appender.airbrake=airbrake.airbrakeappender
log4j.appender.airbrake.api_key=your_airbrake_api_key
#log4j.appender.airbrake.env=development
#log4j.appender.airbrake.env=production
log4j.appender.airbrake.env=test
log4j.appender.airbrake.enabled=true

conclusion

application exception monitoring is an important part of your application lifecycle.

exceptions should be easily visible to the support and development teams and your development process should look to address all exceptions in forthcoming sprints.

exceptions should be used for exceptional cases only, any exception that is not acted upon in production is noise and creates confusion.

tool support is important in this area and errbit looks like a great multi-language tool that can help support your exception management workflow.