You've Got IT Monitoring and IT Alerting All Wrong

it alerting and it monitoring are not what they used to be. in years past, software releases were scheduled a few times per year. often, one monitoring tool would review the infrastructure and would catch and spit out alerts. sorry, but those days are gone. nowadays, start-ups use containers and microservices, continuous integration, and continuous delivery. as such, monitoring can and needs to be at multiple points along the pipeline.

if you are not taking the time to calibrate your systems to reduce the amount of noise and ensure effective alerting, then you’ve got monitoring and alerting all wrong. don’t worry, though. it’s not a death sentence – thankfully. there are clear methods for turning it monitoring noise into actionable it alerting.

come on feel the noise

it’s not just a catchy line from quiet riot. "come on feel the noise" also encapsulates how many engineers in it ops experience monitoring. because of the need to monitor multiple points in the stack, multiple monitoring tools have arisen. because there are multiple monitoring tools, there is a lot of noise. per big panda’s cto :

the old “one tool to rule them all” approach no longer works. instead, many enterprises are selecting the best tool for each part of their stack with different choices for systems monitoring, application monitoring, error tracking, and web and user monitoring.

as companies add more tools, the number of alerts that they must field can grow by orders of magnitude. it’s simply impossible for any human, or teams of humans, to effectively manage that.

indeed, it is impossible for dev, ops, it, or secops to stay on top of 100 alerts during the day and night. instead, these groups need to find a way to make order of the madness. teams need to be nimble to remain competitive and support the multiple moving parts that comprise their groups.  as big panda’s cto goes on to add:

if organizations [do not adjust their monitoring strategies] they will not only cripple their ability to identify, triage, and remediate issues, but they run the risk of violating slas, suffering downtime, and losing the trust of customers.

furthermore, by failing to order the noise engineers and corporations will suffer a predictable set of problems:

• alert fatigue . too many alerts waking engineers up at night will not only cause tired engineers but also hurt your team’s effectiveness at maintaining effectiveness.
• decreased mttr . because there are too many alerts, it will take extra time for engineers to respond intelligently to the issue or begin proper escalation.
• missed alerts. like the boy who cried wolf, after too many false positives, engineers will begin to ignore alerts and, as a result, miss important issues.

need to order the it alerting noise

the very purpose of monitoring is to set thresholds that inform the team on how to act upon them. if the monitoring tools along with alerting tools are not providing actionable events, then there is a problem with how the system is set up.

by bringing a strong testing mindset to bear, monitoring and alerting can help solve many of the issues. solarwinds gets this just right when they indicate:

it is only with continuous monitoring that a network admin can maintain a high-performance it infrastructure for an organization. adopting the best practices can help the network admin streamline their network monitoring to identify and resolve issues much faster with very less mttr.

how can (and should) organizations make order of the noise? our best practices encourage devops, it, or secops to implement the following procedures:

• establish a baseline for the system. initially, set the it monitoring and it alerting parameters somewhat loosely so that you can determine the overall health and robustness of your system. while initially painful, this will allow you to see what types of alerts are garbage and which are meaningful. you won’t always know this type of information from the outset, so it is a necessary part of the process. as our friends at solarwinds go on to note, “once normal or baseline behavior of the various elements and services in the network are understood, the information can be used by the admin to set threshold values for alerts.”
• ensure that the alerts come with proactive messaging. messaging allows engineers to quickly solve problems. by having proactive messaging included, engineers can know if the problem needs escalation or if they can handle the issue.
• in order to keep up with the pace of change that will inevitably befall your system, it is important that every component of your it stack follow this process. otherwise, you will quickly be drowning in alerts.

not every attack of heartburn is a heart attack. similarly, not every alert is high priority requiring a 2 a.m. wake-up call. you need to know how to tell the difference.

control the noise

if you want to maintain your stack’s value and usefulness, you need to have alerting that is meaningful and useful. you need to create thresholds and analyze them. having a thousand alerts come through will cause the most tolerant of engineers to lose their cool.