Why Outages Are the Scariest Thing for DevOps - and Businesses

Written by Craig Powell on Halloween!

Forget the ghouls, witches, and monsters. The scariest thing for a DevOps person is the effect that a web outage will have on his or her company.

For example, take Facebook. The social media giant is projected to eclipse $10 billion in revenue for the first time ever this year, but with those soaring projections comes a greater cost for failures. With several outages already recorded this year, including the one in September that lasted over half an hour and left widespread effects on the entire internet community, you can bet that the company is seeing a significant drop in their revenue streams.

Using their latest outage from earlier this week as a case study (therefore using the latest revenue reports from Q3 2014, when Facebook pulled in $3.2 billion), The Atlantic calculated that the company loses an astounding $24,420 per minute when the service goes down. In the case of this outage, which lasted 35 minutes, those metrics would mean that it cost Facebook $854,700. Granted, the only way that an outage can cost that much is if your annual revenue rivals that of a medium-sized country, but the reality of the situation remains stark – the bigger the footprint that a company has, the more money it will lose when it fails.

It’s important to note that the fault of these outage probably doesn’t rely with Facebook itself. The sad truth of web performance is that between DNS, CDNs, third parties, etc., there are a large amount of factors that can have an adverse effect on a site’s availability which are entirely out of the hands of the internal DevOps teams. Of course, in the case of Facebook, they often serve as a third party themselves on millions of other sites – either through login, commenting, or sharing functions – and therefore, their failures send rippling effects throughout the internet.

As we’ve preached before, the best thing you can do is to guard against your own site being taken down by a third party’s outage like Facebook. That can be done by using asynchronous tags when you install them (they’ll help you through this process), and keeping them below the render start when constructing your page.

The possibility of failures, both internal and external, is as constant a presence for DevOps professionals as Jason Voorhees is for the denizens of Camp Crystal Lake. But remember, there are always steps that can be taken to protect yourself.

Happy Halloween!