We built an LLM pipeline to help a large network operations team stay on top of trouble tickets. It ran quietly in production until the moment it was supposed to earn its keep. In early 2026, a major winter storm swept across a wide region and knocked out power to more than a million people; network equipment failed in bulk, tickets poured in, and the summarizer meant to help engineers triage the chaos went dark.

The root cause was not a bug in the usual sense. There was no null pointer and no bad deploy. We hit the Azure OpenAI tokens-per-minute (TPM) limit, our retries made it worse, and we had no fallback. This is the anatomy of that failure, and the architecture we built afterward to treat an LLM like the rate-limited, non-deterministic dependency it actually is.

The uncomfortable theme up front: our system was busiest during precisely the event it existed to handle. Demand and failure were correlated. If you put an LLM in front of any incident-driven workload, this will eventually be your story too.

What the System Did

Tickets in this environment originate from many channels, including network alarms, customer calls, emails, and proactive checks by operations staff. But by the time our pipeline sees them, they are already incidents and cases in ServiceNow. Our scope starts there.

ServiceNow streams ticket events out of the box through Stream Connect into Kafka. Our application, running in Azure and orchestrated with LangGraph, consumes those events, retrieves related context from Azure AI Search, and calls the Azure OpenAI API to produce three kinds of summary:

• Status notifications for the customers affected by an outage,
• Ticket summaries for the technicians actively working a ticket, and
• Executive summaries that roll up what is happening across a region.

The value is simple. Ticket logs are long, noisy, and full of machine-generated entries. A technician picking up a ticket, or a manager gauging the blast radius of an outage, does not want to read pages of log. They want five sentences. The LLM gave them five sentences, and on a normal day it sat comfortably within quota.

The Failure Timeline

Then the storm hit.

• Equipment failed in bulk. The storm drove power outages past a million customers across a wide region, and our network equipment failed along with the grid. The alarm systems did exactly what they were designed to do: they fired, in volume.
• Tickets surged. They grew to roughly six times our baseline.
• The token load surged far faster. This is what caught us. Our load is not measured in requests; it is measured in tokens. Storm tickets did not just arrive more often, each carried a longer log (more alarms, more correlated events). So, a ~6× jump in tickets became closer to a ~15× jump in tokens per minute.
• We hit the TPM ceiling. Azure OpenAI began returning 429 Too Many Requests with a Retry-After header.
• Retries deepened the throttle. Every layer that could retry, did. That included the SDK, our wrapper, and LangGraph nodes re-running on failure, all in near-unison, with no jitter. Each retry wave slammed the limit together and pushed our effective token rate higherwhile we were already over budget. And because every retry of a generation is another paid, token-billed call, the retries spent the very budget we had blown..
• There was no fallback. When retires were exhausted, there was nowhere to go. There was no cheaper model and no degraded path. Summarization simply stopped 
• The cascade. Customer status notifications stalled, technicians lost the ticket summaries they rely on, and executive summaries went stale. So, the team fell back to reading raw logs by hand. Summarization stayed degraded, on and off, for a multi-hour stretch, until we manually provisioned extra capacity and hand-routed traffic to other models to limp through the worst of it.

The shape of the overload, with illustrative numbers to make the dynamic concrete:

 (The figures are illustrative estimates that preserve the real proportions, not exact production measurements.)

The punchline is in the third row: a ~6× rise in tickets became a ~15× rise in tokens. That is the trap of a token-metered dependency, and the rest of this article is what it taught us.

How the original outage cascaded — and why naive retries made it worse.

Root Cause: An LLM is a Token-Metered Dependency, Not a Request-Metered One

Most writing on resilience, including circuit breakers, retries, and bulkheads, is framed around microservices, and most of it applies here. But an LLM API breaks a few assumptions those patterns quietly rely on, and each broken assumption showed up in our incident.

1. The limit is tokens, not requests. Classic rate-limit thinking counts calls; Azure OpenAI quota is measured in tokens per minute. Your load therefore depends on the size of your inputs — and for a summarizer that is the worst possible coupling: it burns the most quota exactly when documents are longest, which during an incident is exactly when logs are longest. A request-rate dashboard would have looked merely elevated while our token rate was off the chart.

2. Retries spend the budget you are already over. On a normal REST API, a retry is cheap. On a token-metered, pay-per-token backend, every retried generation is another full charge against the limit you just exceeded. Naive retries do not just fail to help. They actively deepen the throttle.

3. Synchronized retries are a self-inflicted DDoS. With no jitter, failed calls backed off by the same amount and returned together, re-tripping the limit on a clock. It is the classic retry storm, amplified by point #2 because each retry is token-expensive.

4. No fallback means peak demand is a single point of failure. One model, one deployment, one path is fine until that path is throttled, and it will be throttled at peak.

5. Demand correlates with failure. A summarizer for incident tickets is, by definition, busiest during incidents. The load spike and the operational emergency are the same event. Capacity planned for the average is capacity planned for the calm before the thing you actually built the system for.

The Fix: Classify, Route by Severity, and Govern the Token Budget

The redesign treats the LLM as a scarce, metered resource and spends it deliberately, turning the frantic, manual capacity-adding and model-rerouting we did by hand during the storm into a permanent, automatic capability.

Schedule in Redis, not Kafka. Our Kafka topics are shared by many interfaces and kept generic, so we could not repurpose them for prioritization. Instead, our consumer reads the generic stream and pushes work into Redis priority queues, where all the scheduling logic lives. Kafka stays the durable ingestion layer — a natural backpressure buffer, so a storm surge piles up safely in the log instead of hammering the model, and consumer lag becomes our early-warning storm metric.

Classify with a tiny model. A small, local ML classifier scores each ticket by priority, severity (P1–P5), and customer impact, using fields already on the ServiceNow ticket. It is deliberately not an LLM call: during a storm every Azure OpenAI token is contested, so spending premium tokens just to decide how to spend premium tokens is exactly backwards. When the classifier is unsure, it routes up, because under-serving a real P1 is far worse than over-spending on a P4.

Route by severity to isolated capacity. Each tier gets the cheapest treatment that still meets its need:

The key trick is that the cheaper tier is a different model, so it draws from a different Azure OpenAI quota pool — a flood of low-severity tickets cannot cannibalize the premium tier's TPM. This needs no provisioned throughput; two standard deployments on different models give you quota isolation for free.

Govern the token rate. A shared, Redis-backed token budget gates every LLM call: we estimate a job's tokens before dispatch and only proceed if the rolling per-minute budget allows, per deployment. Retries use bounded exponential backoff with jitter and honor Retry-After; the first worker to see a 429 sets a global cooldown the whole fleet respects, so the retry storm cannot form. Low-priority queues age and get promoted so they are never starved, and at-least-once delivery is made safe with idempotency keyed on ticket plus log version.

Put together, the request path becomes: ServiceNow → Stream Connect → Kafka → classifier → Redis priority queue → token governor → the right model (or extractive fallback). The queue absorbs the spike, the governor respects the ceiling, and severity routing decides who gets the scarce premium tokens when there are not enough to go around.

The redesigned pipeline: tickets are classified by severity, scheduled through Redis with a token governor, and routed to isolated model tiers.

What We Expect (By Design)

With this in place, the same storm should behave very differently.

• The 429 cascade cannot recur by construction. The governor caps dispatch at quota, so overflow becomes bounded queue lag — low-priority summaries delayed by minutes — rather than total failure.
• Premium capacity is protected. Routing roughly the top 15% of tickets to the premium tier and coalescing related incidents keeps it within quota even under the surge.
• Cost falls. Moving the bulk of volume to a cheaper model and the long tail to zero-token extraction projects on the order of a 50–65% blended token-cost reduction.

Takeaways

• Plan capacity in tokens, not requests: Your load is driven by input size, which spikes exactly when you can least afford it.
• Design for the spike, not the average:  Assume demand correlates with failure.
• Make retries jittered, bounded, and 'Retry-After'-aware: Remember each retry costs tokens.
• Tier your models by importance:  Put cheap or non-LLM paths under the long tail, and isolate premium capacity on its own quota pool.
• Always keep a degraded mode:  A rough summary delivered beats a perfect one that never arrives.