Idempotency Under Concurrency: Why Your Application Check Isn't the Last Line of Defense

Every Kafka tutorial covers idempotency the same way:

"Store the event ID. Before processing, check if it exists. If it does, skip. Done."

That advice is correct. It is also incomplete in a way that will surface only under load — and only in production, at the moment you least want a surprise.

Here is what actually happens inside a multi-replica consumer group when two delivery attempts arrive at the same instant.

The Pattern You Know

A Kafka topic configured with acks=all and enable.idempotence=true on the producer side still delivers messages at least once. Network partitions, broker leader elections, retry storms — any of these can cause a consumer to receive the same message twice. The standard fix is to deduplicate on the consumer side using a stable event ID included in every message payload.

The implementation looks like this:

// Application-layer idempotency check
if (processedEventRepository.existsByEventId(eventId)) {
    log.warn("Duplicate event — skipping | eventId: {}", eventId);
    return false;
}

// Process and record
processedEventRepository.save(ProcessedEvent.builder()
        .eventId(eventId)
        // ...
        .build());

Under normal operation, this is correct. One consumer thread, one message, one check, one save. The second delivery hits the existsByEventId guard and is dropped cleanly.

The problem is that a Kafka consumer group under load does not serialize delivery at the event level.

The Race Condition You Don't See in the Tutorial

Kafka's partition assignment guarantees that only one consumer in a group receives messages from a given partition at a given time. What it does not guarantee is that the same event ID cannot arrive via two different partitions — for example, during a producer retry that sends to a different partition — or that two messages with the same semantics cannot appear if upstream logic emits duplicates before the broker sees them.

More concretely: if you run two replicas of your consumer service (and in Kubernetes, you will), two pods can receive the same event in the same wall-clock window.

The sequence looks like this:

T=0   Pod A receives event e1, calls existsByEventId("e1")  → false (not yet in DB)
T=0   Pod B receives event e1, calls existsByEventId("e1")  → false (not yet in DB)
T=1   Pod A passes the guard, begins processing
T=1   Pod B passes the guard, begins processing
T=2   Pod A saves ProcessedEvent("e1") → INSERT succeeds
T=2   Pod B saves ProcessedEvent("e1") → ???

Both pods read false from the application-layer check before either has written. Both proceed. Without a database-level guardrail, both complete, and you have sent the welcome email — or charged the card, or incremented the counter — twice.

The Fix: The Database as the Serialization Point

The application-layer check is an optimization — it avoids a write attempt for the obvious duplicate case. It is not, and cannot be, a correctness guarantee, because it is not atomic with the write.

The correctness guarantee has to live at the layer that is atomic: the database.

The schema includes a UNIQUE constraint on event_id:

CREATE TABLE processed_events (
    id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_id      VARCHAR(255) NOT NULL UNIQUE,  -- the guarantee is here
    event_type    VARCHAR(100) NOT NULL,
    processed_at  TIMESTAMP NOT NULL DEFAULT now(),
    topic         VARCHAR(255) NOT NULL,
    partition_number INT NOT NULL,
    offset_number    BIGINT NOT NULL
);

That UNIQUE constraint is enforced at the storage engine level — not at the application level, not at the ORM level. No two rows with the same event_id can be committed, regardless of how many concurrent writers attempt it. The database serializes the race.

What changes in the application is how you handle the loser of that race:

@Transactional
public boolean sendWelcomeNotification(UserCreatedEvent event, ...) {

    String eventId = event.getEventId().toString();

    // Fast path: skip if already recorded (optimization, not guarantee)
    if (processedEventRepository.existsByEventId(eventId)) {
        log.warn("Duplicate event detected — skipping | eventId: {}", eventId);
        notificationMetricsService.incrementDuplicate();
        return false;
    }

    try {
        // ... process the event ...

        processedEventRepository.save(ProcessedEvent.builder()
                .eventId(eventId)
                .eventType("USER_CREATED")
                .processedAt(LocalDateTime.now())
                .topic(topic)
                .partitionNumber(partition)
                .offsetNumber(offset)
                .build());

        return true;

    } catch (DataIntegrityViolationException ex) {
        // Two threads passed the application check simultaneously.
        // The DB constraint caught one of them. This is correct behavior.
        log.warn("Duplicate event caught at DB level — skipping | eventId: {} | userId: {}",
                eventId, event.getId());
        notificationMetricsService.incrementDuplicate();
        return false;
    }
}

The DataIntegrityViolationException is Spring's translation of a constraint violation from any supported database (PostgreSQL, MySQL, Oracle). Catching it and suppressing it is not an error condition — it is the correct outcome of a race that the DB resolved correctly.

Note that the exception is caught in the service layer, not in the Kafka listener. If it propagated up to the listener, Spring Kafka's retry mechanism would re-deliver the event, triggering more unnecessary attempts. Catching it at the service boundary terminates the race cleanly.

Two Layers, One Correct Answer

The full pattern has two distinct roles:

Removing either layer has a cost. Without the application check, every retry hits a write attempt that will fail, generating unnecessary DB load and constraint violations in your logs. Without the DB constraint, the application check is a promise with no enforcement — and distributed systems do not honor promises.

The right posture is to treat the application check as a performance optimization and the database constraint as the correctness invariant.

What This Looks Like Under Fault Injection

I verified this pattern against deliberate duplicate delivery in a Kafka consumer running two Spring Boot replicas. Injecting two simultaneous deliveries of the same eventId produced exactly one SENT notification log and one SKIPPED log (the DataIntegrityViolationException path). No double side effects, no exceptions visible to the retry mechanism, no compensating logic required.

The database caught the race that the application check could not.

Key Takeaways

1. Application-layer idempotency checks are not atomic with the writes they guard. Under concurrent delivery, both threads can pass the check before either writes.
2. The database's UNIQUE constraint is the actual serialization point. It is enforced at a level that the application cannot race.
3. DataIntegrityViolationException is a success path, not an error path. It means the DB caught a concurrent duplicate — suppress it, do not propagate it.
4. You need both layers. The application check eliminates unnecessary write attempts. The DB constraint enforces the invariant. Neither is sufficient alone.

If your Kafka consumer only has the application check, you have an idempotency aspiration, not an idempotency guarantee. The race condition described here may never surface on a single-replica dev environment. It will surface on a multi-replica production deployment, under the load conditions that make that second replica necessary in the first place.

The implementation described in this article is part of a production-grade event-driven microservices platform built with Spring Boot 4, Java 21, and Kafka. The full source is available at github.com/Rummy43/ai-microservices-platform.