Event Sourcing in Distributed Systems: Architecture, Consistency, and Fault-Tolerant Projections

If you've worked with a relational database for any length of time, you know the drill: state lives in a row, and every UPDATE overwrites whatever was there before. It's simple, it's fast to reason about, and it throws away history by design. Update a bank balance from $1,000 to $900, and the fact that a withdrawal happened and why is gone unless some separate audit log happened to catch it.

Event sourcing flips that model on its head. Instead of storing current state, you store the sequence of events that produced it, and you derive state on demand by replaying the log. History isn't a side effect you bolt on afterward; it is the write path.

This isn't a niche idea anymore. Netflix runs event-sourced pipelines that process over a billion viewer events a day for personalization and health monitoring. Uber pushes millions of events per minute ride requests, location updates, pricing signals through Kafka-backed streams with exactly-once semantics for fraud detection. LinkedIn built Kafka in the first place to serve as an append-only log for activity data. Slack pipes billions of daily messages through event-driven pipelines that carry full metadata so services don't have to make synchronous calls to each other.

What's harder to find is a clear-eyed look at what actually breaks in production and how teams fix it. That's what this article is about: the data model, the CQRS split, the consistency trade-offs, and the part that tends to get glossed over: how to keep read models correct when the same event might get delivered twice, three services need to stay in sync, and any one of them might be down when the message arrives.

Events as First-Class State

In a CRUD system, a withdrawal looks like this:

-- Traditional CRUD (destructive)
UPDATE accounts SET balance = 900 WHERE id = 123;

In an event-sourced system, nothing gets overwritten. You append a new fact:

-- Event sourcing (additive)
INSERT INTO events (stream_id, event_type, payload, version)
VALUES ('account-123', 'MoneyWithdrawn', '{"amount": 100}', 42);

Current state is reconstructed by folding over every event in a stream, in version order. It's a pure function — same events in, same state out, and it maps directly onto a left-fold/reduce if you've ever written one:

def reconstruct_balance(account_id):
    events = db.query(
        'SELECT * FROM events WHERE stream_id = %s ORDER BY version',
        [f'account-{account_id}']
    )
    state = {'balance': 0}
    for event in events:
        if event.type == 'AccountOpened':
            state['balance'] = 0
        elif event.type == 'MoneyDeposited':
            state['balance'] += event.payload['amount']
        elif event.type == 'MoneyWithdrawn':
            state['balance'] -= event.payload['amount']
    return state['balance']

The Event Store Schema

The event store itself is a boring relational table with one architectural rule bolted on: nothing is ever updated or deleted.

CREATE TABLE events (
    id BIGSERIAL PRIMARY KEY,
    stream_id UUID NOT NULL,
    event_type VARCHAR NOT NULL,
    payload JSONB NOT NULL,
    version INT NOT NULL,
    occurred_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE (stream_id, version)
);

That UNIQUE (stream_id, version) constraint is doing more work than it looks like. It's your optimistic concurrency control: if two processes both try to append version 5 to the same stream, the second one gets a unique-constraint violation instead of silently clobbering the first. It's compare-and-swap at the stream level, for free.

And to be clear — this table can live in Postgres, MySQL, Oracle, whatever you're already running. Event sourcing is a write strategy, not a database product. Don't let anyone tell you that you need a specialized event store to do this.

Snapshots

Replaying thousands of events every time you want to read an entity's state gets expensive fast. The standard fix is snapshotting: periodically materialize current state and store it next to the log. On read, load the latest snapshot and replay only what's happened since.

SELECT * FROM snapshots
WHERE stream_id = 'account-123'
ORDER BY version DESC LIMIT 1;

-- Then replay only events after snapshot.version
SELECT * FROM events
WHERE stream_id = 'account-123'
AND version > :snapshot_version
ORDER BY version;

The CQRS Boundary

In practice, event sourcing almost always shows up paired with CQRS (Command Query Responsibility Segregation) — splitting the write path (commands that produce events) from the read path (queries against a denormalized read model). This isn't just architectural tidiness. The event log is optimized for append throughput and sequential replay; it's a bad fit for ad-hoc queries, joins, sorting, or full-text search.

Consistency Models and the Lag Problem

Projection Lag Is Real, Design For It

Because the projection runs asynchronously, there's a window between an event landing in the store and the read model reflecting it. A user withdraws money and immediately checks their balance and might see the stale number. This is the central consistency problem in any CQRS/event-sourcing setup, and it needs an explicit answer, not a hope that it won't matter.

In healthy systems on Kafka or a comparable bus, projection lag typically sits around 50–300ms. Under load spikes, partitions, or consumer restarts, it can stretch to seconds or minutes. Your design needs to be correct under the bad case, not just the median.

Pick Your Battles: Selective Consistency

The real design question isn't "How do I eliminate eventual consistency?" It's "Where do I actually need strong consistency, and where can I let the lag ride?" Those two buckets look pretty different:

Read-Your-Own-Writes

For the strong-consistency bucket, the most common mechanism is read-your-own-writes (RYOW). The command API hands back the version number it just appended; the client sends that version back on its next read; and the query service refuses to answer until the read model has caught up.

# 1. Command returns the appended event version
POST /accounts/123/withdraw {amount: 100}
→ HTTP 200 {status: 'ok', event_version: 42}

# 2. Client sends version as constraint on next read
GET /accounts/123/balance
Headers: X-Min-Version: 42

# 3. Query service checks projection position
SELECT last_processed_version FROM projection_checkpoints
WHERE name = 'balance_projection';
-- If < 42, wait up to 500ms then retry or return 202 Accepted

When You Just Bypass the Read Model

For a small set of high-stakes flows such as payment confirmation screens, dispute resolution, regulatory audits, plenty of systems skip the read model entirely and rebuild state straight from the event log. It's always consistent, at the cost of latency. Snapshots (see above) keep that latency bounded even for entities with long histories.

Three Ways to Keep Projections in Sync

The projection needs to process each event exactly once and update the read model atomically with its own checkpoint. In practice, that comes down to three strategies, each suited to a different infrastructure setup.

1. Same-Database Transaction

The simplest option: put the read model and the checkpoint table in the same relational database, and wrap the update and the checkpoint advance in one transaction.

BEGIN;
UPDATE account_balances
SET balance = 900
WHERE account_id = 123;

UPDATE projection_checkpoints
SET last_event_id = 42
WHERE name = 'balance_projection';
COMMIT;

Either both writes land or neither does, which kills the two classic failure modes: processing an event without advancing the checkpoint (you'll reprocess it on restart) and advancing the checkpoint without writing the data (permanent, silent data loss). Great when your whole read model fits in one database.

2. Kafka + Idempotent Consumers

Once Kafka is your event bus, consumer offsets live inside Kafka and can't be committed in the same transaction as your read-model write. Kafka only promises at-least-once delivery, so on a consumer restart, some events may show up twice. The fix is to make the consumer idempotent, in which case it's safe to run twice on the same event.

INSERT INTO account_balances (account_id, balance, last_event_id)
VALUES (%s, %s, %s)
ON CONFLICT (account_id) DO UPDATE
SET balance = EXCLUDED.balance,
    last_event_id = EXCLUDED.last_event_id
WHERE account_balances.last_event_id < EXCLUDED.last_event_id;

That WHERE clause is the whole trick. If event version 42 gets replayed after it's already been applied, 42 < 42 is false and the update is a no-op.

The INSERT branch only ever fires once, when the entity's stream first shows up. Every event after that takes the ON CONFLICT path, and the version guard is what makes it idempotent.

3. The Transactional Outbox Pattern

Things get harder when a projection needs to fan out to several different stores at once, say a relational read model, an Elasticsearch index, a Redis cache, and a webhook receiver. No single database transaction can span all of those. This is where the transactional outbox pattern earns its keep.

The projection writes its read-model update and an outbox row in one local transaction. A separate relay process picks up pending outbox rows and delivers them to each downstream target, retrying until each one confirms. Because the outbox write is atomic with the read-model write, the relay always knows exactly what still needs to go out.

-- Atomic: both rows commit or neither does
BEGIN;
UPDATE account_balances SET balance = 900 WHERE account_id = 123;

INSERT INTO outbox (id, payload, es_sent_at, redis_sent_at, webhook_sent_at)
VALUES (gen_random_uuid(),
        '{"account_id": 123, "balance": 900, "event_version": 42}',
        NULL, NULL, NULL);
COMMIT;

Making the Outbox Relay Fault-Tolerant

The relay delivers to each target independently and tracks confirmation per target. The scenario worth designing for: the relay writes successfully to target A, then crashes before marking the row delivered, so on restart, target A gets the same write again. Every target needs to handle that gracefully.

Elasticsearch: Let the Document ID Do the Work

Elasticsearch indexing is idempotent as long as you supply a stable document ID yourself. Index the same ID twice and you just overwrite the first doc with identical content, having no duplicates.

# Non-idempotent: ES generates a random ID
es.index(index='accounts', body={'balance': 900})
# → Two calls produce two documents

# Idempotent: stable ID derived from the outbox row
es.index(
    index='accounts',
    id=outbox_row.id,  # same on every retry
    body={'balance': 900, 'event_version': 42}
)

Redis: Absolute Values, Not Deltas

SET in Redis is naturally idempotent — setting a key to the same value twice is the same as setting it once. The catch: your relay has to write absolute values, never deltas. INCRBY and LPUSH are not idempotent and have no business being in a relay path.

# Non-idempotent: applying a delta twice corrupts state
redis.incrby('account:123:balance', -100)

# Idempotent: writing the derived absolute value
redis.set('account:123:balance', 900)

Webhooks: Idempotency Keys

External webhook receivers are the trickiest case, because you don't control their implementation. The standard answer is the one Stripe documents publicly: an Idempotency-Key header. The relay sends the outbox row's UUID as that key on every attempt; the receiver keeps a table of keys it's already seen and just replays the original response for a repeat.

Getting this right on the receiver side means handling three edge cases:

1. Concurrent duplicate requests: Two requests with the same key arrive before either has been saved. Fix it with an atomic INSERT under a unique constraint; the second insert fails, and that thread waits on the first.
2. Key reuse with a different body: Same key, different payload. Reject with a 422 after comparing a hash of the stored body to the incoming one.
3. Key expiry: Keys can't live forever. Stripe uses a 24-hour TTL, which means your relay's retry window (and dead-lettering) has to finish well inside that boundary.

The key record and the business-logic effect have to commit in the same transaction. If the business logic runs but the key never gets saved, a retry re-executes the logic and that's not a missed optimization, that's data corruption.

def handle_request(request):
    key = request.headers['Idempotency-Key']
    req_hash = sha256(request.body)

    with db.transaction():
        row = db.execute('''
            INSERT INTO idempotency_keys
                (key, request_hash, created_at, expires_at)
            VALUES (%s, %s, now(), now() + interval '24 hours')
            ON CONFLICT (key) DO UPDATE SET key = EXCLUDED.key
            RETURNING key, request_hash, response_status, response_body
        ''', [key, req_hash]).fetchone()

        if row.request_hash != req_hash:
            return 422, 'Key reused with different request body'

        if row.response_status is not None:
            return row.response_status, row.response_body

        result = run_business_logic(request)

        db.execute('''
            UPDATE idempotency_keys
            SET response_status = %s, response_body = %s
            WHERE key = %s
        ''', [result.status, result.body, key])

        return result.status, result.body

Dead-Letter Escalation

Transient failures — timeouts, a target being briefly down — get exponential backoff with per-target retry counters on the outbox row. Permanent failures — a schema mismatch, a decommissioned endpoint — shouldn't be allowed to block the relay forever. After a configurable number of attempts (10 is a common default), move the row to a dead-letter table for someone to look at.

Who's Actually Running This in Production

Takeaways

A few things worth internalizing if you're building (or maintaining) one of these systems:

• Eventual consistency is a contract, not a bug. RYOW turns the consistency requirement into an explicit versioned header instead of an assumption. Idempotency keys turn "don't double-process this" into a first-class API contract. The outbox pattern turns delivery into a durable, queryable table instead of a fire-and-forget call. Once it's explicit, you can test it and monitor it — that's the whole point.
• Idempotency is the load-bearing primitive. It's standing in for distributed transactions, which are either unavailable, too slow, or too fragile for high-throughput event-driven systems. At-least-once delivery plus idempotent receivers gets you effectively-once behavior without any distributed coordination.
• The pattern underneath is always the same: a unique key derived from something stable (an event version, an outbox row's ID), stored atomically next to the effect it protects, checked before that effect is applied, and expired on a TTL that bounds how long you need to keep retrying.
• None of this locks you into specific technology. Event store in Postgres, EventStoreDB, or Kafka's compacted log — doesn't matter. Read model in MySQL, Redis, or Elasticsearch — doesn't matter. The outbox just needs to live in the same ACID store as the read model it's paired with. That's a low bar, and it's why teams can adopt this incrementally next to whatever they're already running.

The hard part of event sourcing was never the event log where appending immutable rows is the easy 20%. The real engineering work is keeping derived state correct across multiple heterogeneous targets when any one of them can fail at any moment. Idempotency, applied consistently, is what makes that tractable without reaching for distributed transactions.