Prevent Duplicate API Calls With Idempotency: Patterns That Work

A failure pattern I have seen repeatedly in enterprise integrations: a payment request times out at the caller, succeeds downstream, and is then retried as though it failed.

MuleSoft projects, this is especially easy to miss because a single inbound request may trigger payment, ERP, inventory, and messaging calls before the caller receives a final response. It succeeded 2 times.

The customer was charged twice. Inventory was reserved twice. The ERP received two invoice requests. Nothing crashed. Every service did exactly what it was designed to do.

The problem was simpler and more dangerous: the caller retried before anyone could prove whether the first request had already succeeded.

That is the part people miss when they say, "just add retries."

Retries are not reliability by themselves. A retry is a second attempt to perform a business operation. If the first attempt reached the server but the response was lost, a retry can create a duplicate order, a duplicate payment, a duplicate shipment, or a duplicate customer record.

Idempotency is what makes retries safe.

In enterprise integration, that distinction matters because the network cannot tell you the truth quickly enough. A timeout only tells you that the caller did not receive a response. It does not tell you whether the downstream system completed the work.

The Incident: Timeout That Became Two Invoices

Flow that looks normal:

Commerce API  →  MuleSoft order integration  →  Payment service  →  ERP

A customer placed an order. The Commerce API sent POST /orders to the integration layer. MuleSoft validated the payload, called the payment service, then created an invoice in the ERP.

The ERP was slow that morning. Not down. Just slow enough to produce a bad distributed-systems outcome.

The first request reached MuleSoft. Payment succeeded. The invoice request reached the ERP. Then the Commerce API timed out waiting for the response. Its retry policy did what it had been configured to do: retry once.

The second request was treated as a brand-new order. Payment ran again. The ERP created another invoice.

Sequence diagram showing a timeout followed by a retry that charges the card twice and creates two ERP invoices:

Figure 1: Before. The response was lost, not the work. The retry repeats every side effect, producing a duplicate charge and a duplicate invoice.

From the perspective of each individual component, this was reasonable. The caller saw a timeout and retried. MuleSoft received a valid request and processed it. Payment received two valid payment instructions. The ERP received two valid invoice requests.

The tech team may initially treat this as a timeout-tuning problem. Whereas it is an ownership problem: the API layer, integration layer, and system of record each assume another component will prevent the duplicate. Unless the business operation has one durable identity across those boundaries, none of them can reliably do it.

That is the real job of idempotency: give the system a durable way to recognize the same intent when it arrives again.

Idempotency Is Not "the API Returns the Same Response"

The formal definition is usually presented as: repeating the same operation produces the same result. That is technically useful, but it is not enough for a production integration.

For a business API, the practical definition is better.

For one business intent, perform the side effect at most once, then return the recorded outcome for every valid repeat.

That definition has three important parts.

First, it is tied to business intent, not merely an HTTP request. A user buying two identical laptops should create two orders. A retry of the same "buy one laptop" action should not.

Second, it protects side effects. Returning the same JSON response is meaningless if the system already charged the card twice.

Third, it requires a recorded outcome. If the first request succeeded, the retry should receive the original successful response. If the original request was rejected, the retry should receive the same rejection.

A unique database constraint helps, but it is only one layer. It does not automatically coordinate payment, messaging, invoice creation, and the response sent back to the caller.

The dangerous period is the gap between "the work may have happened" and "the caller knows the result." Distributed systems spend a lot of time in that gap. A connection can reset after the downstream commit. A load balancer can close an idle connection. A worker can complete the work and die before it writes the response. A message can be delivered again after a consumer restart.

You cannot eliminate every ambiguity. You can design so that ambiguity does not create duplicate business activity.

Two-column comparison contrasting a retry that repeats the side effect against a retry that replays the recorded outcome:

Figure 2: The retry is identical in both columns. Only the server's memory of the intent differs.

The Implementation Path: An Idempotency Key and a Durable Record

The implementation begins with an Idempotency-Key header. The client creates a unique key for one business attempt and sends the same key on every retry.

POST /orders HTTP/1.1
Idempotency-Key: 0a5a98a0-7b40-4a8f-a5e2-7cf94e75a825
Content-Type: application/json

The server stores that key with a request fingerprint, a processing state, the eventual response, and an expiry.

The key must be unique per operation. If a client reuses a key with a different payload, return a conflict. Never silently treat different requests as the same request.

The Critical Rule: Reserve the Key Before the Side Effect

The idempotency record must be created before payment, invoice creation, message publishing, or any other irreversible action. In a Spring service, the first durable action is an atomic insert.

public OrderResponse createOrder(String idempotencyKey, CreateOrderRequest request) {

    String requestHash = RequestHasher.sha256(RequestCanonicalizer.canonicalize(request));

    Reservation reservation = idempotencyService.reserve(OPERATION, idempotencyKey, requestHash);

    if (reservation.isReplay()) {
        IdempotencyRecord existing = reservation.record();

        if (!existing.getRequestHash().equals(requestHash)) {
            throw new ResponseStatusException(HttpStatus.UNPROCESSABLE_ENTITY,
                    "Idempotency key was reused with a different request payload");
        }

        switch (existing.getStatus()) {
            case COMPLETED:   return existing.replayResponse();
            case IN_PROGRESS: throw new OrderProcessingException(existing.retryAfterSeconds());
            case FAILED:      throw new ResponseStatusException(HttpStatus.CONFLICT,
                                  "This idempotency key already failed: " + existing.getFailureCode());
            default:          throw new IllegalStateException("Unknown state " + existing.getStatus());
        }
    }

    // We own the key. Every side effect below carries it as its business identity.
    try {
        PaymentResult payment = paymentClient.charge(
                request.customerId(), request.total(), request.currency(), idempotencyKey);

        ErpInvoice invoice = erpClient.createInvoice(
                request.orderReference(), payment.transactionId(), idempotencyKey);

        OrderResponse response = OrderResponse.created(
                request.orderReference(), payment.transactionId(), invoice.invoiceNumber());

        idempotencyService.complete(OPERATION, idempotencyKey, HttpStatus.CREATED.value(), response);
        return response;

    } catch (RuntimeException exception) {
        if (OutcomeClassifier.isTerminal(exception)) {
            idempotencyService.markFailed(OPERATION, idempotencyKey,
                    exception.getClass().getSimpleName());
        } else {
            idempotencyService.markAmbiguous(OPERATION, idempotencyKey,
                    exception.getClass().getSimpleName());
        }
        throw exception;
    }
}

Two details in that method are easy to get wrong.

The first is where uniqueness is enforced. Two requests can arrive at nearly the same moment, and both can pass an application-level "does this key exist?" check. The guarantee belongs in the database.

CREATE UNIQUE INDEX ux_idempotency_operation_key
    ON idempotency_record (operation, idempotency_key);

The reservation then treats a duplicate-key exception as an ordinary concurrency outcome rather than an error, and commits in its own transaction so that a later downstream failure cannot roll away the evidence that the attempt was made.

@Transactional(propagation = Propagation.REQUIRES_NEW)
Reservation reserve(String operation, String key, String requestHash) {
    try {
        return Reservation.acquired(repository.insertInProgress(operation, key, requestHash));
    } catch (DuplicateKeyException concurrentInsert) {
        // Another thread, pod, or retry won the race. Its record is authoritative.
        return Reservation.replay(repository.require(operation, key));
    }
}

The second detail is the catch block. Marking a record FAILED because a downstream call timed out is how teams reintroduce the original bug: the next retry sees a terminal state, decides the work never happened, and starts a second charge. An ambiguous outcome is not a failure. Keep it distinguishable, and let a reconciliation job resolve the true state against the payment provider and the ERP.

IN_PROGRESS Is a Real Production State

Many implementations get the happy path right and fail during concurrent retries.

The original request starts processing, but the client times out after two seconds and immediately retries. The original operation is still waiting on the ERP. What should the second request receive?

Not another payment attempt. It should receive a clear, boring answer.

HTTP/1.1 409 Conflict
Retry-After: 3

{
  "code": "ORDER_PROCESSING",
  "message": "This order request is already being processed."
}

The caller can wait and retry with the same idempotency key. Once the first request completes, the next retry returns the stored result. That behavior is not glamorous. It is predictable. In integration systems, predictable is often more valuable than fast.

MuleSoft: Enforce the Contract at the API Boundary

The integration layer is a strong place to enforce this contract, because it sees the inbound request before it fans out to multiple systems.

The important structural choice is that reservation is an insert, not a lookup. A SELECT followed by an INSERT is not safe under concurrent retries, so the flow attempts the insert first and interprets the unique-constraint error as "someone else owns this intent."

<flow name="create-order-api">
    <http:listener config-ref="httpListener" path="/orders" allowedMethods="POST">
        <http:response statusCode="#[vars.httpStatus default 201]"/>
        <http:error-response statusCode="#[vars.httpStatus default 500]"/>
    </http:listener>

    <validation:is-not-blank-string
        value="#[attributes.headers.'idempotency-key' default '']"
        message="Idempotency-Key header is required"/>

    <set-variable variableName="idempotencyKey"
                  value="#[attributes.headers.'idempotency-key']"/>
    <set-variable variableName="orderRequest" value="#[payload]"/>

    <ee:transform doc:name="Canonicalize request">
        <ee:message>
            <ee:set-payload resource="dw/normalize-order-request.dwl"/>
        </ee:message>
    </ee:transform>

    <set-variable variableName="requestHash"
                  value="#[dw::core::Crypto::SHA1(write(payload, 'application/json') as Binary)]"/>

    <try doc:name="Reserve idempotency key">
        <db:insert config-ref="OrderDb">
            <db:sql><![CDATA[
                INSERT INTO idempotency_record
                    (operation, idempotency_key, request_hash, status,
                     created_at, updated_at, expires_at)
                VALUES
                    ('CREATE_ORDER', :key, :hash, 'IN_PROGRESS',
                     CURRENT_TIMESTAMP, CURRENT_TIMESTAMP,
                     CURRENT_TIMESTAMP + INTERVAL '24' HOUR)
            ]]></db:sql>
            <db:input-parameters><![CDATA[#[{
                key: vars.idempotencyKey, hash: vars.requestHash
            }]]]></db:input-parameters>
        </db:insert>
        <set-variable variableName="keyOwned" value="#[true]"/>

        <error-handler>
            <on-error-continue type="DB:QUERY_EXECUTION">
                <set-variable variableName="keyOwned" value="#[false]"/>
            </on-error-continue>
        </error-handler>
    </try>

    <choice>
        <when expression="#[vars.keyOwned == true]">
            <flow-ref name="process-order-side-effects"/>
        </when>
        <otherwise>
            <flow-ref name="resolve-existing-idempotency-record"/>
        </otherwise>
    </choice>
</flow>

The resolution flow is where the states earn their keep. A hash mismatch returns 422. A COMPLETED record replays the stored body with its original status. An IN_PROGRESS record returns 409 with Retry-After.

<choice>
    <when expression="#[vars.record.request_hash != vars.requestHash]">
        <set-variable variableName="httpStatus" value="#[422]"/>
    </when>
    <when expression="#[vars.record.status == 'COMPLETED']">
        <set-payload value="#[read(vars.record.response_body, 'application/json')]"/>
        <set-variable variableName="httpStatus"
                      value="#[vars.record.response_status default 200]"/>
    </when>
    <when expression="#[vars.record.status == 'IN_PROGRESS']">
        <set-variable variableName="httpStatus" value="#[409]"/>
    </when>
</choice>

DataWeave: Canonicalize Before Hashing

The same logical order can arrive with fields in a different order, with different casing, or with optional values that are absent in one attempt and empty in the next. If you hash raw payload text, semantically identical requests produce different hashes, and your idempotency layer quietly stops working.

Normalize first, then hash the normalized output.

%dw 2.0
output application/json skipNullOn = "everywhere"

fun canonicalText(value) =
    trim(value default "") match {
        case s if s == "" -> null
        else -> s
    }

var lineItems = (payload.items default [])
    map (item) -> {
        sku:       upper(trim(item.sku)),
        quantity:  item.quantity as Number,
        unitPrice: item.unitPrice as Number
    }
    orderBy ((item) -> item.sku ++ "|" ++ (item.unitPrice as String))
---
{
    customerId:     trim(payload.customerId),
    orderReference: trim(payload.orderReference),
    currency:       upper(trim(payload.currency default "USD")),
    total:          payload.total as Number,
    items:          lineItems,
    shipTo: if (payload.shipTo == null) null
            else {
                line1:      canonicalText(payload.shipTo.line1),
                city:       canonicalText(payload.shipTo.city),
                region:     upper(trim(payload.shipTo.region default "")),
                postalCode: canonicalText(payload.shipTo.postalCode),
                country:    upper(trim(payload.shipTo.country default ""))
            }
}

Note what is excluded. Client timestamps, trace identifiers, and transport metadata do not belong in the fingerprint, because a legitimate retry carries new values for them. The goal is not cryptographic cleverness. The goal is to define what "the same order request" means in your domain, and to write that definition down in code.

With the key reserved and the fingerprint stable, the retry path changes shape entirely.

Sequence diagram showing a retry that finds a completed idempotency record and replays the original response without repeating payment or invoicing

Figure 3: After. The retry still happens. It finds the completed record and replays the original response instead of repeating the work.

Where Idempotency Belongs

Do not stop at the public API if the flow crosses more than one boundary. The useful pattern is to propagate the key as correlation metadata so that every system on the path shares one business identity.

Architecture diagram showing the business key propagated from the commerce client through MuleSoft, the idempotency store, the payment provider, the ERP, and the event bus:

Figure 4: The same business key becomes the provider idempotency header, the ERP external document reference, and the event deduplication key.

Each downstream system needs a compatible protection mechanism. Payment providers generally accept their own idempotency header. ERP systems usually support an external document reference with a unique constraint. Message consumers can record processed event IDs before applying the side effect. Databases should enforce unique business keys wherever the domain allows it.

A single API gateway record cannot make an entire distributed transaction atomic. It does, however, give every downstream call a stable business identity, which is what turns an ambiguous retry into an answerable question.

The Questions to Answer Before You Ship

An idempotency design is incomplete until the team can answer these:

• What is one business intent in this API?
• Who creates the idempotency key, and is it unique per attempt rather than per session?
• Which fields define the request fingerprint, and which are deliberately excluded?
• What happens when the same key arrives with a different payload?
• What does a caller receive while the original request is still in progress?
• Which downstream side effects receive the same business key?
• How do you recover records stuck in IN_PROGRESS after a crash?
• How long are completed keys retained?

The retention period should cover the realistic retry window, including asynchronous queues and client retry behavior. For many order APIs, 24 hours is a reasonable starting point; the right answer depends on the business process, not a generic framework default.

The stuck-record question deserves more attention than it usually gets. A pod that dies mid-operation leaves a reservation nobody owns. Without a sweeper that reconciles those records against the payment provider and the ERP, your safety mechanism becomes a source of permanent 409 responses for a key the client will keep retrying.

The Real Reliability Pattern

Retry budgets protect a system from retry storms. Circuit breakers protect a dependency from overload. Timeouts prevent callers from waiting forever.

Idempotency protects business operations when those mechanisms encounter uncertainty. That is why it belongs beside retries, not after them.

A timeout is not evidence of failure. It is evidence that the caller does not know the outcome.

My rule for enterprise integrations is simple: a retry must continue a known business operation, not create a new one. If the system cannot tell those two situations apart, retries are not a reliability feature yet.