The Cost of Ignorance: Why You Need a Chain of Evidence, Not Just Logs

You may have seen my posting on East/West is the New North/South. The bottom line is that traditional API Gateway models simply don't provide the level of security we need in modern microservice architecture. The problem is that only 20% of the traffic (that is the inbound traffic up until the gateway) is secure, everything inside the data center is "trusted."

Sure, it may be secured by two-way certs, IP whitelisting, or other network models, but there is no identity. We don't know WHY "Service A" called to "Service B." We might not even know that "Service A" was what called to "Service B" just that something made a call.

But why do we care? We trust "Service A" don't we?

In Infrastructure, Legal and Embarrassment -- Why We Secure Our Systems, we looked at the three reasons we care. We looked at things we can do to lock down those services, and what we mean by "fine-grained access control." And then, in an offhand way, I said "log everything" which is a gross over-simplification.

You see simple logs aren't enough. If you have a breach, you're going to need to provide a complete record from what kicked off the original request, what other requests it kicked off, and what data was exposed.

Let's look at our three reasons we secure data, and what we need to know about those transactions:

Infrastructure — Root Cause Analysis Paralysis

Even if you have your services secure, if you don't know why "Service A" called to "Service B," you have a lot of trouble determining your RCA (root cause of the problem). I think most of us have been forced to answer, "well it stopped so I think we're okay but keep an eye on it." If you're lucky the problem doesn't come back, but relying on luck isn't really a "best practice."

If you can look at the logs, determine which transactions are suspicious, and then drill your way back up the chain to see that a very special user is able to start a cascading series of requests in just such a way, you can close that door. Without the upstream requests that caused Service A to fire off an odd request, you may never know who was responsible and never be able to adequately secure the environment.

Legal — Just How Bad Is the Problem?

You may feel you're in an industry not covered by the new California Privacy laws or by Europe's GDPR or even State of New York's Financial Services laws, but honestly, if you deal with money you probably are. And then there's HIPAA...

All of these require an immutable chain of evidence. Note that word "immutable." This is where syslogs aren't enough. Not only do you need to know User A called App B that called to Service A that needed to make a callout to Services B and C, but you can't just show a few lines of logfiles stitching those requests together. You have to be able to show that ever request was signed and that the record was not altered.

There aren't a lot of ways to do that unless you're registering each application with its own set of keys, and using those keys to send and receive data on each and every transaction. It's not about trusted and untrusted services anymore, it's about an untrusted logging system now because, you know, lawyers.

The other thing is that we get charged per exposure. The average cost per exposure varies anywhere from $110 to $336 per record, but if you can't determine exactly what was exposed, you could be on the hook for your entire database...

So even at $123 per record in the "Research" business, a Cambridge Analytica breach of 87 million records could have cost over $10 billion in fines.

Business — We Demand Answers

The Cambridge Analytica exposure wasn't a breach so it wasn't something that could be fined, but it definitely put Facebook in a tough place, and part of the reason it was tough was because they couldn't definitively answer what seemed like a simple question: "How many records were exposed?"

This stuff is complicated, it's hard enough talking to the DevOps team about the problem, but talking to your shareholders, a news outlet or lawmakers makes it even harder. Saying, "we don't really know what data was exposed" makes you appear to either be lying or incompetent. Either way, it's not a good day.

So to sum up, service to service security isn't just about securing the flow, it's about creating a full chain of evidence so your DevOps team can fix problems, you can limit your legal exposure, and you can simplify business explanations when something goes wrong.