The Retry Budget Pattern: How to Stop Retry Storms in API-Led and Microservice Systems

The Production Story

Several years ago, my team made a decision that felt obviously correct: If a downstream call fails, retry it. More retries, more resilience. We set three retries on every integration touching our order-fulfillment platform, shipped it on a Thursday, and went home feeling good about our reliability posture.

Six weeks later, retries were the single largest source of traffic in the platform.

It surfaced during a routine inventory-sync slowdown. The inventory service got a little sluggish. Nothing dramatic. p99 crept from 200ms to maybe 1.2s. Our order API, sitting one layer up, started timing out and retrying. Three times each. The MuleSoft layer feeding the order API also had retries configured, so it retried the retries. By the time traffic reached the already-struggling inventory service, a single user click had turned into somewhere between nine and twenty-seven backend calls.

The inventory service didn't recover. It got buried. We took a partial outage caused entirely by our own retry logic trying to save us.

Figure 1: Retry amplification across API layers. One client request fans out to as many as 27 backend calls, while a retry budget keeps the downstream bounded.

Why It Happened

This part isn't obvious until you've watched it happen. Each retry decision was reasonable on its own, but together they were tearing the platform apart.

Every team had configured retries looking only at their own layer. Three retries seemed fine in isolation. The trouble is that retries multiply across layers, and nobody owned the end-to-end number. Two retries here, three there, and suddenly one click is nine calls. The math was sitting in plain sight, and none of us had done it.

What really bit us was how retries behave during a partial outage. When a downstream is healthy, retries are cheap, because failures are rare. When it's degraded, which is exactly when you're retrying the most, those retries pile more load onto a service that's already on its knees. So the system gets most aggressive at the worst possible moment. That's a feedback loop, and feedback loops like this end in outages.

I've written before about retry storms and bounded reliability, and about how AI-generated DataWeave can fail quietly in production. This is the same family of problem. No single component is broken here. What's missing is a limit that spans the whole call path. Retries without a budget are really just a slow, polite way to DDoS yourself.

The Bad Implementation

This is what almost everyone ships first. I've shipped it myself.

@Retryable(
    value = { RemoteServiceException.class },
    maxAttempts = 3,
    backoff = @Backoff(delay = 200, multiplier = 2)
)
public InventoryResponse checkInventory(String sku) {
    return inventoryClient.get(sku);
}

At first glance, this looks reasonable. Exponential backoff, a sane attempt count, a typed exception. Code review passes in thirty seconds.

The problem is there's no awareness of anything beyond this one method. It retries during a full downstream outage just as eagerly as during a one-off network blip. It has no idea that the caller above it is also retrying. Worse, it happily retries errors that will never succeed. A 400, a validation failure, a duplicate-order rejection. You're burning retries on requests that were dead on arrival.

The Good Implementation

A retry budget flips the control. Instead of asking "how many times should this one call retry," you ask "what fraction of my total traffic is allowed to be retries?"

The rule of thumb that's served me well: retries should never exceed 10% of your real request volume. If more than one in ten requests is a retry, something is genuinely broken, and hammering it harder won't fix it. It'll only dig the hole deeper.

Here's a token-bucket budget that enforces this. Successful calls slowly refill the budget; each retry spends from it. When the budget is empty, you stop retrying and fail fast.

public class RetryBudget {
    private final double retryRatio;      // e.g. 0.10 = 10%
    private final AtomicLong tokens = new AtomicLong();
    private final long maxTokens;

    public RetryBudget(double retryRatio, long maxTokens) {
        this.retryRatio = retryRatio;
        this.maxTokens = maxTokens;
    }

    // Every real request deposits a little budget back.
    public void onRequest() {
        tokens.updateAndGet(t -> Math.min(maxTokens, t + (long)(retryRatio * 100)));
    }

    // A retry is only allowed if the budget can pay for it.
    public boolean tryRetry() {
        return tokens.updateAndGet(t -> t >= 100 ? t - 100 : t) >= 0
            && tokens.get() >= 0 && spend();
    }

    private boolean spend() {
        return tokens.getAndUpdate(t -> Math.max(0, t - 100)) >= 100;
    }
}

The key behavior: under normal load, the budget stays full, and retries work as expected. Under a real outage, failures outpace successes, the budget drains, retries stop, and you protect the downstream instead of finishing it off.

Wiring it into a Spring Boot client looks like this. Notice the two gates before any retry happens. The error has to be retryable, and the budget has to allow it.

public InventoryResponse checkInventory(String sku) {
    budget.onRequest();
    try {
        return inventoryClient.get(sku);
    } catch (RemoteServiceException ex) {
        if (isRetryable(ex) && budget.tryRetry()) {
            return inventoryClient.get(sku); // single budgeted retry
        }
        throw ex; // fail fast, don't amplify
    }
}

private boolean isRetryable(RemoteServiceException ex) {
    int code = ex.statusCode();
    return code == 502 || code == 503 || code == 504 || code == 429;
}

Not every error deserves a retry. This distinction matters more than the budget math, because retrying a non-retryable error is pure waste.

The Architecture Pattern

In MuleSoft, the same idea applies, and it's where I see the most damage because retries get configured at multiple layers without anyone counting. Keep the platform-level retry shallow and let your error type drive the decision.

<until-successful maxRetries="1" millisBetweenRetries="500"
    doc:name="Budgeted Retry">
    <http:request method="GET" config-ref="Inventory_HTTP"
        path="/inventory/{sku}"/>
</until-successful>

Then classify errors in DataWeave so the flow only retries what's worth retrying, and so a budget breach degrades cleanly rather than throwing:

%dw 2.0
output application/json
var retryable = [502, 503, 504, 429]
---
{
  shouldRetry: retryable contains payload.statusCode,
  action: if (retryable contains payload.statusCode)
            "RETRY_IF_BUDGET"
          else
            "FAIL_FAST"
}

The surprising part, when we rolled this out, was how rarely the budget actually engaged. Under healthy conditions, you'd never know it's there. It only shows its value during the bad fifteen minutes that used to turn into a bad three hours.

Figure 2: The retry budget as a token bucket. Successful requests refill it, retries drain it, and once it falls below the 10% line, retries are disabled, and calls fail fast.

A Real Production Example

Picture a payment-processing flow calling an external gateway, with order-fulfillment and a Salesforce sync downstream. The gateway has a rough afternoon and starts returning intermittent 503s.

Without a budget: every failed charge retries three times, order-fulfillment retries the payment call, and the Salesforce sync retries too. The gateway, already wobbling, gets three-to-nine times its normal load and falls over completely. A partial degradation becomes a full payment outage during peak hours.

With a 10% budget: the first wave of retries is absorbed normally. As 503s climb, the budget drains within seconds. Retries stop, failed charges fail fast with a clear error, and customers see a retry-later message instead of a spinner. The gateway gets breathing room and comes back on its own. You take a small, honest failure now instead of a much bigger one you caused yourself.

Metrics That Matter

A budget you can't see is a budget you won't trust. These are the four numbers I put on a dashboard before I ship any retry change to production.

If your retry success rate is low, that's the tell. You're retrying things that were never going to succeed, and the budget is doing you a favor by cutting them off.

Monday-Morning Checklist

• Count your real end-to-end retry multiplier across every layer, not per service.
• Set a retry budget at roughly 10% of traffic and enforce it with a token bucket.
• Classify every downstream error as retryable or not, and never retry 4xx except 429.
• Cap retries to a single attempt at most layers; let the budget, not the attempt count, be your safety limit.
• Honor Retry-After on 429s instead of guessing backoff.
• Put retry ratio and budget-exhaustion metrics on a dashboard before you ship.
• Test it: degrade a downstream in staging and confirm retries actually stop.

Final Thoughts

Retries feel like reliability. Really, they're a loan against your downstream's capacity, and like any loan, they're cheap when you don't need them and expensive at the worst possible time. What makes the retry budget useful is that it ties retrying to the one number that should govern it: how much real traffic you're actually serving.

I've made this mistake myself, and I've watched sharp teams make it too, because every individual decision looked correct in review. The fix isn't fancier backoff or smarter jitter. It's a ceiling. Decide up front how much of your traffic you'll allow to be retries, then hold that line even when every instinct is screaming at you to push harder. Especially then.