The Future of Incident Notification in the Modern Enterprise

When the telephone first came out, making a call required the assistance of an operator. There was no direct dial. People picked up the phone and said, “Connect me to the Smith house.” In fact, people probably knew the operator by name. They’d say, “Madge, connect me to the Smith house,” and Madge would reply, “OK, hang on.”

After some time, dials were put on telephones and people connected to their party directly. However, for special services — international calls, person to person and reverse charged calls — you still needed an operator. Back in the 70’s when I was in college and running short on cash, I actually picked up a phone more than once, dialed zero and said, “Operator, I need to make a collect call to this number…“

Large companies that had their own internal phone systems still used operators for day-to-day work. But as technology improved, the number of corporate operators diminished. Today when you call a large company, you’ll most likely get a machine that directs your call to the right person. So, after 150 years of telephony, we’ve evolved back to Madge. Only this time Madge is a robot. But I digress.

Design Pattern: The Secretarial Pool

Back in the days of Mad Men, high level executives had personal secretaries that took dictation, typed letters and managed the appointment calendar, while lower level execs used the secretarial pool. The secretarial pool was, as the name implies, a group of individuals who sat in a big room on-call from 9 to 5. All were proficient in the skills of shorthand and typing. If you lived on the lower rungs of the white collar org chart and you needed a letter typed or copied, you called up the secretarial pool. A few minutes later a secretary was available to take your dictation on a shorthand pad and convert the scribble into typewritten pages for mailing or filing.

Things chugged along nicely keeping a lot of individuals gainfully employed with a clear career path from secretarial pool to executive secretary until the dedicated word processing machine, and the PC revolution brought us through WordPerfect and into Microsoft Word: a computer on every desktop and MS Word on every computer.

In one swift stroke of history, the secretarial pool went the way of the milkman delivering dairy products directly to your door everyday. Execs now have the technology to write and copy their own letters, as well as keep their own calendars. Executive assistants have come to replace the traditional secretary, but more and more the work has evolved and becomes more complex.

Do we see a pattern here? I don’t know about you, but I do, and it is: as technology progresses, the mundane tasks completed by humans has become or is becoming automated. The work that remains requires a more sophisticated type of human activity.

My firsthand experience

So, let’s talk about how this relates to incident monitoring. I’ll let you in on something. I once carried a pager. It was back in the 1980’s when I was working at a computer rental company. That was at a time when computers were so expensive that it made more sense to rent them to incur the operating expense rather than to buy the machines and have them sit around as depreciating assets.

I carried a pager one weekend a month. If any of the machines went down, the little black ornament hanging from my belt went off like a set of Christmas lights. I’d have to get on the phone and make things right. The guy who owned the company never carried the pager, but I did have to notify him when an important client called. 

Flash forward, 2016.

The other day I was talking to a guy at a pretty good-sized tech company. I asked him if he has to carry a pager, to which he replied no. Now, of course given the strides in automated alert monitoring and incident notification technology, both of us know that when you carry a pager, there is no longer a dedicated device hanging from your belt. Rather, we get to listen to our cell phones chirp away with the sounds of incoming emails and SMS messages.

So, I asked, “You never get notified.”

His response: “Well, I do get called if it has something to do with revenue, credit card processing goes down or we can’t bill.”

So I thought, He’s like my old boss at the computer rental company. He doesn’t want to be burdened with trivial incidents, but if something big comes along, he wants to know.

A few days later I talked to another guy. This guy is in charge of operations at a well known website that has a lot of traffic. I asked him if he carries a pager. 

His reply? “Yes.”

This guy is pretty high up in the org chart. He’s a knowledgeable executive engineer. He knows the difference between UDP and TCP. He understands why Javascript is not a strong typed language. And, he knows the company’s systems really well.

I said, “Really? You carry the pager. How come?”

And this was his reply — pay attention very closely: “There is so much automation. Infrastructure resiliency has been built into our cloud platform so that auto recovery of certain issues is fully automated. This enables engineers to invest back gained time into core engineering instead of monitoring and troubleshooting. In the new paradigm, alerts come to a human only when automated recoveries fail, or an anomaly in the pattern is found that may start to impact systems or services.”

So, what does this all mean? It means this: the pattern I described above holds true. So, let me repeat it:

As technology progresses, the mundane tasks that are done by humans has become or is becoming automated. The work that’s left requires a more sophisticated type of human activity.

The telephone operator that connected telephone calls is no longer needed.

Secretaries no longer take dictation and type. 

Cloud-based systems know how to repair themselves.

What’s a human to do? Deal with what’s going on between systems and outside the system.

In other words, “You mean I have to get the AWS invoice paid in order to get our websites back online?”

What does this all mean?

What it means is that as trends continue, more people higher up the org chart are going to be on pager duty for two reasons. First, problems that can’t self-remedy are going to be so complex that an advanced level of expertise will be required to fix them. Second, when something goes wrong, you are going to need someone with a lot of authority to fix the issue. In the old days, if a computer went down, I had to get in the truck, drive to the customer and swap out the broken part. In the not too distant future (I’m guessing, and maybe exaggerating a bit), the distressed computer will call an Amazon drone that delivers the part. Both machines will know how to do the installation. If an issue does occur, it will be on the order of drones colliding with one another. And if the pattern above holds true, the person holding the pager that goes off in response to the incident is going to need a lot of expertise and authority to make things right.

In a way, I find it funny. I mean, who would have thought that a symbol of corporate authority would one day be carrying the pager?