When Downtime Means an Unlocked Front Door

Anyone who has carried a pager long enough develops a professional numbness. A queue backs up, a p99 drifts past budget, a deploy does something stupid at 40% rollout. You fix it, you write it up, you go back to sleep. The stakes are real but abstract: revenue per minute, an SLA credit, a churn number on somebody's spreadsheet.

That numbness doesn't survive contact with a product people rely on for safety.

Picture a regional connectivity degradation after midnight, and an impact estimate on a screen. Normal night, ordinary graph. Then consider what the number is actually counting. For some slice of households in that region, a camera at a front door has gone dark. A parent who checks whether their kid got home. Someone who installed cameras after a break-in and now sleeps better.

I want to be careful here, because "our uptime saves lives" is the sort of thing that makes engineers roll their eyes, rightly. Most minutes of most outages harm nobody. But the distribution has a tail, and when a product is somebody's sense of safety, the tail is where the meaning of the work lives. Taking that seriously changes what you measure, what you alert on, and what you're willing to drop under load.

Edge Reliability Is a Different Animal

Much of my earlier career was conventional infrastructure work with data-center consolidation, phased cloud migrations of several thousand workloads, where most of the energy goes into sequencing risk rather than into any particular technology. Good training in systems thinking. Almost no preparation for how consumer hardware behaves in the wild.

Three properties make this class of system unlike a web service.

• The edge is hostile, and you own none of it. Devices sit on consumer Wi-Fi, behind a cheap router, on an oversubscribed ISP, drawing power that browns out during exactly the storms when people most want their cameras working. You own none of the last mile and all of the customer's expectation of it.
• Demand is correlated, which breaks naive capacity planning. Web traffic averages out; event traffic from a physical fleet doesn't. A delivery wave or a thunderstorm crossing three states produces millions of events inside the same few minutes. Independent load is easy. Synchronized load is what pages you.
• Failure costs are wildly uneven. A dropped analytics event is a rounding error. A live view that spins for eight seconds while somebody stands on a porch is arguably worse than an outright failure, because the customer sat there and watched the product not work. If not all errors cost the same, they shouldn't page the same, yet nearly every monitoring setup I've encountered treats them identically.

Together, those produce the failure mode that should drive the whole design: you can be green on CPU, memory, and 5xx rate while customers are having a red night.

Define "Up" in the Customer's Language

The first fight in this kind of environment is about vocabulary. "The service is up" tends to mean "the servers are up," and those are not the same claim.

The alternative is SLIs built on journeys a person can perceive. For a home-security-shaped product, that's notification latency from edge event to push landing on the phone, live-view time-to-first-frame, and clip availability and retention.

The second one is where I'd push hardest, because there's a tempting shortcut. "Session established" is easy to measure and usually already instrumented. It's also a lie, since a session can establish and then deliver nothing watchable for another three seconds. Move to first rendered frame, and your numbers get noticeably worse — which is how you learn the old metric was flattering you.

An objective in that shape reads roughly:

Two choices there matter more than the threshold. Use rolling windows rather than calendar months, because calendar boundaries teach teams to hold their breath until the first and then ship anyway. And give the error budget real authority: burn it, the deploy freeze happens, no case-by-case negotiation with whoever has the loudest roadmap. That second one is where program-management discipline earns its keep more than any architectural decision — governance, entry and exit criteria, named decision rights. It sounds like bureaucracy right until the first freeze holds without anyone having to win a political argument.

One test worth applying to every proposed SLO: if breaching it wouldn't change what anybody does next week, it isn't an SLO. It's a dashboard.

Measuring Is Harder Than Target-Setting

Picking four seconds is easy. Knowing whether you hit it is hard, because the clock starts on a device you don't control and stops on a phone you don't either.

The approach that works is correlation across the full path. The device stamps an event ID at capture, and it rides every hop — ingest, media pipeline, notification service, push provider, client ACK. Each hop logs the shared ID with a timestamp, and you reassemble the journey afterward. In Splunk, that looks something like:

index=device_events (stage=capture OR stage=push_ack)
| stats earliest(_time) as t_capture, latest(_time) as t_ack,
        values(region) as region by event_id
| eval e2e_latency_s = t_ack - t_capture
| where isnotnull(t_ack)
| eval met_slo = if(e2e_latency_s <= 4, 1, 0)
| stats count as total, sum(met_slo) as ok,
        perc99(e2e_latency_s) as p99_s by region
| eval attainment = round(100 * ok / total, 3)
| sort - p99_s

Unglamorous, and it answers the only question that matters four minutes after waking up: are real people getting notifications on time, and if not, where. Sorting by regional p99 turns "something feels slow" into a location.

Now look at the isnotnull(t_ack) filter, because it's the most instructive line in the query. It quietly excludes every journey that never completed, which is the worst outcome for a customer. That exclusion is how an entire class of failure hides in plain sight.

The Failure Class That Never Pages You

The scenario I'd most want a team to design against is the one that generates no alert at all.

A subset of devices with one hardware revision, one firmware version, one specific reconnect path stops delivering notifications while continuing to report healthy. Capture succeeds. The event enters the pipeline. It simply never produces an ACK.

Every aggregate metric absorbs it. The affected population is small enough that regional p99 doesn't budge, attainment stays inside budget, and no error rate moves, because nothing errored. Detection ends up coming from a cluster of support tickets, which is the most expensive monitoring you can buy: it means your customers are doing it for you.

Two defenses follow directly. Alert on capture events with no matching ACK inside a window, as its own signal rather than folded into latency with slow and failed have different runbooks and different customer meanings. And break tier-1 attainment out by device model and firmware version, accepting the cardinality cost, because a well-behaved aggregate is very good at hiding a badly-behaved cohort.

Averages don't just lie about the tail. They lie about who's in it.

Running the Room

Detection without a fast, calm response is just expensive telemetry. Three things I'd insist on anywhere.

Separate the Incident Commander from the person fixing it. Those are genuinely different jobs, and you can watch the debugging degrade in real time when one person does both while fielding stakeholder pings. It's the cheapest reliability improvement available and costs zero engineering hours.

Define severity by customer impact, never by component. An internal dashboard degrading is a SEV-3 no matter how loudly its owner complains. Notifications delayed across a region is a SEV-1 immediately, because the product's core promise is broken. Writing that down ends a lot of arguments before they start.

Run game days, and expect the first one to be humbling. In my experience, the runbook is wrong in several places, at least one dashboard fails to load under load, and finding the person who can trigger a manual failover takes longer than anyone predicted. By the fifth game day it's boring, which is the entire point.

What I'd Tell My Earlier Self

Write SLOs in the customer's language and give the budget teeth. Instrument the journey with a correlation key, because component metrics tell you what broke while journey metrics tell you what the customer felt. Decide in advance what you shed under load, and that judgment is too important to make at 3 a.m. Break tier-1 metrics out by cohort, because aggregates hide the people you're failing. And separate the commander from the fixer.

The stack will keep moving: more inference at the edge, new codecs, whatever replaces today's push mechanics. The operating philosophy doesn't change. Measure what people actually experience, protect the moments that matter, fail gently, respond as it counts.

The best feedback this work gets is silence.