It Costs That Much Because Observability Takes Hours

Today’s blog title is inspired by this song, "It Costs That Much." My daughter started singing it regularly after she opened her bakery. Read on for details on that story, and how it relates to observability.

I thought of it, and my daughter’s reasons for singing it, after a few responses to my recent blog post, "Observability Expenses: When ‘Pennies on the Dollar’ Add Up Quickly." It touched a nerve, which was nice to see. This is an important, nuanced, and complex conversation. I believe that getting folks involved in this conversation is better for everyone.

Some of the thoughtful responses included this one and this one.

But apparently for some folks, it touched the WRONG nerve. People misunderstood the point I was trying to make. Some people (and I’m not linking to those posts) took it as a reason to bash vendors for charging anything at all. To posit that all software (and especially all monitoring software) should be free.

I wasn’t saying that monitoring tools should be free. I wasn’t even saying that vendors are charging too much. I was saying that if there’s no plan for the observability data we collect, ANY cost seems like too much. The problem is in our lack of clear goals, not the cost of services.

If there’s no plan, you can’t manage (or even estimate) the costs.

Let me explain by going back to the title of the blog, and the song that inspired it. My daughter tends to sing the catchy little ditty after one of the (unfortunately many) calls with a prospective customer, who tells her – with all the confidence of someone who’s baking expertise extends only as far as opening a box of Duncan Hines cake mix – that her cakes are way too expensive, because, and I quote: “anyone could make them.”

To her credit, my daughter remains professional (and calm, which is more than I would have managed).

“Free” Isn’t (Always) the Right Answer

Monitoring and observability are ferociously difficult to get right, and the people who devote their time (and sometimes careers) to making it work – to say nothing about making it better – deserve all of our respect.

They also deserve to be paid for their efforts.

Like those folks who call my daughter’s bakery, you know who typically thinks they shouldn’t have to pay for observability? People who have only the most tenuous idea of what goes into it and how it works.

For folks at the forefront of observability, like Liz, this is nothing new.

There are those who will be quick to say, “But Leon, open source tools like Grafana are free! Why doesn’t everyone else do that?”

OK, but hear me out. While the phrase “Linux is only free if your time has no value” is far less true than when Jamie Zawinski wrote it back in 1998, the spirit of it still holds. A thing is only as free as the real total cost of it—inclusive of time, effort, and money.

Lest you think I’m working my way toward the old “Car Triangle” conundrum.

…that’s NOT actually my point. It actually IS possible to have all three. My daughter can throw together a cake, or a loaf of challah, or a batch of cookies faster, better, and cheaper than she can buy them.

But of course, that’s only if you discount all the time it took her to get to this point. I won’t waste your time sharing that complete (and oft-repeated) story of the itemized $10,000 repair bill which read:

Breakdown of Charges:
Turning the screw: $1
Knowing which screw to turn: $9,999

The point stands—the only cheat code for the car triangle dilemma is if you place yourself in the middle of it, bridging the gaps.

As the late 90s commercial says, “For everything else, there’s MasterCard.”

Not All Observability Data Is Equal

So back to my original point—my post recently was not trying to say observability should be free, or that vendors are charging too much. Re-iterating my main point: if there isn’t a meaningful plan for the observability data you collect, any cost is too high. The corollary is that if you DO have a meaningful plan, you will also know what it’s worth to you to collect, process, and store it.

But nothing in our lives as IT practitioners is ever as easy as coming up with a single plan for all our data and calling it a day. Different data types have different value propositions, and some of those value profiles are themselves variable based on outside factors like time of year (i.e. Black Friday).

Or what has just happened. Cribl understood this all the way back in 2022, when they framed this idea succinctly:

“As it happens, all of this high-fidelity data is completely worthless until it comes time for forensics, where it turns out to be invaluable. You have absolutely no need for such a high level of detail until the exact moment when you do need it – and you actually can’t do anything without it.”

To expand on that idea a little: I don’t think the nice folks at Cribl are saying that lo-fi data is fine for daily ops monitoring and observability, and you only need high-fi data until you’re in the middle of a retro. Instead, I think it’s akin to something my friend Alec Isaacson, Solutions Architect at Grafana, once said:

“Murphy’s law requires that you’re probably not collecting the data you need.”

Unless, of course, you are.

Cognitive (Data) Dissonance

We don’t need all the data, until we do, and in that liminal moment it goes from worthless to desperately important. We don’t want to pay anything to collect, process, or store the data. Until there’s a crisis, at which point we’re willing to spare no expense.

For the poor beleaguered monitoring engineer (and yes, I will die on the hill that this is a real job. Or should be.) this may feel like our professional Kobiyashi Maru scenario.

But it’s not. I believe there’s a way to thread this needle, and in the next blog I’m going to explore exactly that, along with how OTel fits into all of this.