Aggregated Alerting for Logs with Logz.io and BigPanda

in previous posts, i wrote about the integration between logz.io and datadog and pagerduty . in this article, i’d like to introduce you to an additional new integration — this time with bigpanda.

just in case you’ve never heard of the platform, bigpanda aims to help users reduce the high signal-to-noise ratio in today’s it monitoring stacks. a typical monitoring configuration consists of multiple tools, with each one generating its own set of alerts. bigpanda promises to help teams to correlate these different alerts and identify the ones that really matter.

logz.io’s alerting feature allows you to create advanced alerts on specific log messages, with recent updates allowing users to create alerts using elasticsearch metric aggregations such as max , min , avg , and sum.

let’s see how these two platforms play together.

creating a new integration in bigpanda

first, we need to set up a new integration within bigpanda to retrieve a token and api key.

this is easily done via the integrations tab in bigpanda. when asked to select what type of integration, select rest api. on the integration page, you will then be required to generate an app key. in this example, i have named mine “logz.”

you then have the two pieces needed to create the integration in logz.io: (1) the app key and (2) a token:

creating a new logz.io endpoint

now that we have the required pieces to set up the integration, we’re going to set up a new alert endpoint in logz.io.

under the alerts tab, select the alert endpoints page and then click “add endpoint”:

select “bigpanda” from the “type” drop-down menu, and fill in the required details using the app key and token that we retrieved above.

after you save the new endpoint, it will be added to the list of endpoints defined in logz.io:

creating a new alert

congratulations! the integration is set up, and all that is left now is to configure a new log-based alert to test whether the integration is working as expected.

creating a new alert in logz.io is done from the “discover” tab in kibana. enter your query in the “search” field — in this case, i’m looking at an apache access log that reports a 504 error:

once you’ve narrowed down the type of log message on which you want to be alerted, click the “create alert” button. the query is copied to the alert creation wizard, where it can be modified as desired:

on the first page of the wizard, you will need to configure the trigger conditions — or, in other words, select which specific conditions will trigger an alert.

then, after naming and describing the alert on the following page, all that’s left to do is select the bigpanda endpoint on the third and final page of the wizard:

create the alert and open the incidents page in the bigpanda ui. once an alert is triggered in logz.io, it will show up as a new incident in bigpanda.

open the incident and select it to see more details on the log message:

note: to reduce the amount of “log noise,” logz.io will send a maximum of five samples of the same event.

once in bigpanda, you can use the rich sharing and event management features to manage and correlate with alerts from other integrations.

additional resources

• logz.io alerts faqs
• bigpanda documentation