How to Use Graylog for Software Monitoring

As every software system "talks" in logs, there a need to constantly manage and regularly examine them. What complicates the process is that the logs are often scattered across multiple servers, and, as the data volume grows, management of the logs becomes more and more time-consuming. To address these challenges in my software company, we opted for Graylog, an open-source solution for log management. We usually have it paired up with Grafana, an open-source dashboard for data visualization. These two tools allow us to keep an eye on performance and technical state of the software we develop.

What Is Graylog?

Graylog is a powerful platform that allows for easy log management of both structured and unstructured data along with debugging applications. It is based on Elasticsearch, MongoDB, and Scala. Graylog has a main server, which receives data from its clients installed on different servers, and a web interface, which visualizes the data and allows to work with logs aggregated by the main server.

We use Graylog primarily as the stash for the logs of the web applications we build. However, it is also effective when working with raw strings (i.e. syslog): the tool parses it into the structured data we need. It also allows advanced custom search in the logs using structured queries. In other words, when integrated properly with a web app, Graylog helps engineers to analyze the system behavior on almost per code line basis.

Graylog/Grafana dashboard example.

Graylog Use Cases

The main advantage of Graylog is that it provides a perfect single instance of log collection for the whole system. This comes in handy if the system infrastructure is large and complex. It could be distributed around multiple places, and not all team members could have immediate access to all its components. With Graylog, we tackle these issues and ensure our incident response time is rapid.

At Logicify, we use Graylog both for the applications under development and the ones already released publicly. In both cases, some modes of Graylog application are unique, while some intersect.

Use in Development, Testing, and Staging Phases

In non-production environments, Graylog is primarily used as logs depot. It stores structured logs of the applications, so it is easy to navigate in them and find any information a developer could be interested in, be it the time-stamp or duration of a web-request, exception or error stack trace.

As Graylog consistently stores all the logs of an application, it allows tracking of the system’s state for every specific moment of time. This gives developers an efficient mechanism to understand the context of any error. When this context is combined with QA reports, it is easy to replicate the issue locally and fix the bug. Thus, Graylog becomes a powerful helper for the developers to optimize the code and improve the application’s performance.

Graylog/Grafana dashboard example.

Use in the Production Phase

In software products that are already released for public use, Graylog is also applied for log storage. However, unlike the case with apps under development, in released apps, these logs are used primarily for the purposes of maintenance and incident response. This is made possible thanks to Graylog’s notifications configured for specific events in the web application, such as 5xx errors or performance counter hits. These notifications could be sent to a predefined list of emails every time an issue occurs. Support managers immediately learn what happened from the system itself and react proactively, even in case users do not report the issues. Dev people, in turn, could debug the application and fix it ad-hoc.

Graylog is instrumental for business decisions regarding specific features in a software product. The tool collects custom analytics on user behavior in the system and visualizes the data in the form of pie charts, time bar graphs, and other graphics. For instance, one could measure the average response time of an application's components, or determine the time interval when the maximum amount of orders was placed on your eCommerce platform. Based on the data received, product stakeholders could make the decisions on further scaling the application, adding or removing some pieces of functionality. We use Graylog as a collateral tool for data visualization as Grafana is more sophisticated in this regard.

Graylog/Grafana dashboard example.

Advantages of Graylog

There are a few advantages of Graylog we have noticed so far, and these are what make the tool perfectly fit into our workflow and delivery.

1. Graylog is a free, open-source software.
2. Its trigger actions or notifications immediately inform us when something needs attention, so we constantly keep abreast of the system performance.
3. With error stack traces received from Graylog, engineers understand the context of any issues in the source code. This saves time and efforts for debugging/troubleshooting and bug fixing.
4. The tool has a powerful search syntax, so it is easy to find exactly what you are looking for, even if you have terabytes of log data. The search queries could be saved.
5. Graylog offers an archiving functionality, so everything older than 30 days could be stored on slow storage and re-imported into Graylog when such a need appears (for example, when the dev team need to investigate a certain event from the past).
6. Python applications could be easily connected with Graylog as there is an out-of-box library for this.

Graylog vs ELK

Graylog server (the entire application and web interface), combined with MongoDB and Elasticsearch, is often compared to the ELK stack (Elasticsearch, Logstash, and Kibana). Though both solutions are pretty similar in terms of features set, there are a few differences to consider.

The most important distinction between the two lies in the fact that, from the very beginning, Graylog is positioned as a powerful logging solution, while ELK is a Big Data solution. Graylog can receive structured logs and standard syslog directly from an application through the network protocol. On the contrary, ELK is the solution that analyzes already collected plain text logs using Logstash and then parses and passes them to ElasticSearch.

In ELK, Kibana plays the role of a dashboard and displays the data received from Logstash. Graylog in this sense is more convenient as it offers a single-application solution (excluding ElasticSearch as a flexible data storage) with almost the same functionality. So the time needed to deploy a usable solution is smaller. Moreover, Graylog has a friendlier GUI right out of the box and superior permissions system compared to ELK. As Elasticsearch fans, we prefer Graylog over ELK as it perfectly meets our needs in terms of log managing.

Bottom Line

Combined with Grafana, Graylog is a powerful tool for logging and monitoring the technical state of web applications. For both non-production and production environments, Graylog is a nice silo for the logs. The tool has an intuitive GUI and offers alerting, reporting and custom analyzing features. Most importantly, it brings terabytes of data across multiple log sources and geographies. Based on these advantages, we preferred Graylog over another popular stack with a similar functionality, ELK. Over the years, Graylog proved effective and user-friendly, so we include it into the standards delivery and deployment kit for web products.