Consumer Lag Is the Wrong Finish Line

A Kafka recovery often looks successful before it actually is.

The replay job exits. Consumer lag drops. Error rate returns to normal. The dashboard is green enough for everyone to want the incident closed.

Then somebody checks the downstream projection, account balance, inventory state, enrichment table, or audit report and finds the real problem: data moved, but state did not converge.

That is the failure this article is about.

Consumer lag is a progress signal, not a recovery verdict. It proves position movement. It does not prove terminal outcomes, idempotent side effects, owned failures, or source-to-target convergence.

If an incident review ends with “lag went back to zero,” the team has proved movement, not recovery.

Declare the Recovery Cohort First

The first observability decision is not which metric to graph. It is what work the recovery run is supposed to prove.

A useful cohort names:

• the topic, partitions, and offset range
• the event types included in the run
• the business-key scope
• the expected record count
• the recovery reason
• the owner
• and the stop conditions

A compact contract object makes that scope reviewable:

recovery_run:
  id: rec-2026-09-05-01
  reason: projection_drift_after_consumer_restart
  owner: billing-platform-oncall
  run_type: incident_repair
  source:
    topic: billing-events
    event_types:
      - InvoicePosted
      - CreditApplied
    offsets:
      - partition: 8
        from: 492100
        to: 492881
      - partition: 9
        from: 388010
        to: 388522
  business_scope:
    key_type: account_id
    selector: incident_account_hashes_2026_09_05
  expectations:
    expected_record_count: 1294
  reconciliation:
    source_of_truth: billing_ledger
    checks:
      - account_day_balance_matches
      - no_missing_decision_events
      - no_unowned_final_failures
  safety:
    dry_run_required: true
    max_retry_attempts: 3
    stop_on_reconciliation_mismatch: true

The object does not have to be beautiful. It has to be unambiguous. During an incident, ambiguity is expensive: “replay the billing topic” is not a scope, and “check the dashboard” is not evidence.

The cohort also forces a design decision: do you need partition/offset proof, business-key proof, or both? Offsets are excellent for Kafka lineage. Business keys are what downstream systems usually reconcile. A serious recovery plan connects them instead of pretending one replaces the other.

Count Terminal Outcomes, Not Processed Records

Processed count is a dangerous comfort metric. A record can be processed and still be wrong. It can be skipped correctly. It can be suppressed as a duplicate. It can be retried. It can land in a dead letter queue. It can be quarantined because replaying it would violate a business rule.

The recovery run should account for outcomes explicitly:

The key equation is simple:

expected = applied + updated + skipped + duplicate_suppressed + failed_final + quarantined

Retryable failures should not count as terminal until the retry policy is exhausted or the run is stopped. If the math does not close, the recovery has not been proven.

Keep Metrics Bounded

Recovery runs create tempting labels: run IDs, account IDs, order IDs, offset ranges, incident IDs. Put those blindly into metrics and the observability system becomes the next incident.

Use metrics for bounded dimensions and put high-cardinality detail in decision logs, traces, audit tables, or exemplars.

That split matters. Metrics tell you that recovery is unhealthy at a class level. The decision record tells you which run, offsets, business keys, and owners are involved.

Emit Decisions You Can Inspect

A useful recovery log does not just say that a handler ran. It says why a record took a path.

{
  "recoveryRunId": "rec-2026-09-05-01",
  "eventId": "evt-1001",
  "businessKeyHash": "bkh_7f3a",
  "topic": "billing-events",
  "partition": 8,
  "offset": 492881,
  "eventType": "InvoicePosted",
  "outcome": "DUPLICATE_SUPPRESSED",
  "reason": "idempotency_key_exists",
  "schemaVersion": "v4",
  "consumerGroup": "usage-aggregator",
  "runType": "incident_repair",
  "decidedAt": "2026-09-05T18:42:31Z"
}

Notice what is missing: raw payloads, customer identifiers, tokens, and private business data. Recovery logs should explain decisions without becoming a second data-leak surface.

The Java side can stay small. The following Java 17/Spring/Micrometer sketch is illustrative, not a complete production service. It assumes repository, meterRegistry, objectMapper, and log are class fields:

record RecoveryDecision(
    String recoveryRunId,
    String eventId,
    String businessKeyHash,
    String topic,
    int partition,
    long offset,
    String eventType,
    RecoveryOutcome outcome,
    String reason,
    String consumerGroup,
    String runType,
    Instant decidedAt
) {}

void emit(RecoveryDecision decision) {
    repository.insert(decision);

    Tags tags = Tags.of(
        "event_type", decision.eventType(),
        "run_type", decision.runType(),
        "outcome", decision.outcome().name()
    );

    meterRegistry
        .counter("kafka_recovery_records_decision_total", tags)
        .increment();

    if (decision.outcome() != RecoveryOutcome.FAILED_RETRYABLE) {
        meterRegistry
            .counter("kafka_recovery_records_terminal_total", tags)
            .increment();
    }

    log.info("kafka_recovery_decision={}", toJson(decision));
}

String toJson(RecoveryDecision decision) {
    try {
        return objectMapper.writeValueAsString(decision);
    } catch (JsonProcessingException error) {
        return "{\"serialization_error\":\"recovery_decision\"}";
    }
}

That emitter is not the recovery system. It is the instrumentation boundary. It makes the record decision visible in the durable decision log, the metric stream, and the incident timeline.

Reconciliation Is the Verdict Layer

The hardest part of recovery observability is deciding what “correct” means.

For some domains, a count is enough. For most serious domains, it is not. A count can match while the wrong keys, amounts, versions, or timestamps remain wrong.

The source of truth must be named. The tolerance window must be named. Late arrivals, clock skew, eventually consistent sinks, and manual exceptions must be handled explicitly. Otherwise reconciliation becomes another dashboard panel that looks precise while hiding the policy decision.

For example, if the recovery cohort is supposed to repair account/day billing state, compare the source ledger to the target projection for the same cohort. This is PostgreSQL-style SQL; use an equivalent pattern for your database:

WITH expected AS (
  SELECT
    account_hash,
    business_date,
    SUM(amount_cents) AS expected_amount_cents
  FROM billing_ledger
  WHERE recovery_run_id = :recovery_run_id
  GROUP BY account_hash, business_date
),
actual AS (
  SELECT
    account_hash,
    business_date,
    amount_cents AS actual_amount_cents
  FROM billing_daily_projection
  WHERE recovery_run_id = :recovery_run_id
)
SELECT
  COALESCE(e.account_hash, a.account_hash) AS account_hash,
  COALESCE(e.business_date, a.business_date) AS business_date,
  COALESCE(e.expected_amount_cents, 0) AS expected_amount_cents,
  COALESCE(a.actual_amount_cents, 0) AS actual_amount_cents
FROM expected e
FULL OUTER JOIN actual a
  ON e.account_hash = a.account_hash
 AND e.business_date = a.business_date
WHERE COALESCE(e.expected_amount_cents, 0)
   <> COALESCE(a.actual_amount_cents, 0);

Also check whether every selected event produced a decision:

SELECT c.event_id
FROM kafka_recovery_cohort_event c
LEFT JOIN kafka_recovery_decision_log d
  ON c.recovery_run_id = d.recovery_run_id
 AND c.event_id = d.event_id
WHERE c.recovery_run_id = :recovery_run_id
  AND d.event_id IS NULL;

Those queries are intentionally plain. Recovery confidence should not depend on a heroic debugging session. It should depend on executable checks that can run before anyone is asked to close the loop.

Alert on Evidence Failure, Not Just Kafka Symptoms

Recovery alerts should catch bad recovery, not only broken Kafka.

Page when terminal outcomes do not close after the deadline, reconciliation fails after the check has completed, or an active recovery run stops making progress. Do not page simply because an in-progress recovery still has work remaining.

groups:
  - name: kafka-recovery
    rules:
      - alert: KafkaRecoveryUnaccountedRecords
        expr: |
          kafka_recovery_records_unaccounted > 0
          and on(topic, event_type, run_type)
          kafka_recovery_deadline_reached == 1
        for: 15m
        labels:
          severity: page
        annotations:
          summary: "Kafka recovery has overdue unaccounted records"

      - alert: KafkaRecoveryRunTooOld
        expr: |
          time() - kafka_recovery_last_progress_timestamp_seconds > 1800
          and on(topic, event_type, run_type)
          kafka_recovery_run_active == 1
        for: 5m
        labels:
          severity: ticket
        annotations:
          summary: "Kafka recovery run has stopped making progress"

The alert is not the verdict. It tells the operator where to look. The verdict should still come from the recovery evidence record.

Test the Evidence Path

Recovery safety should not only be described. It should be testable.

At minimum, test the evidence path with a small matrix:

Run these tests in two places. First, keep a small contract-test suite around the recovery library or consumer adapter so decision emission cannot regress. Second, run a replay drill in a non-production environment with a deliberately small cohort. The drill should prove that the metrics, decision log, reconciliation query, alerts, and final verdict all agree before the same pattern is trusted during an incident.

Recovery Definition of Done

Before closing a Kafka recovery run, require evidence for each item:

This full pattern is not free. Structured decision logs add write volume. Reconciliation consumes compute. Audit tables need retention and access control. Quarantine creates human workflow.

There are also cases where the full pattern is too much. A stateless consumer that only updates an internal cache may not need decision tables, reconciliation jobs, and formal verdicts. Start with the recovery definition of done, then choose the minimum evidence that proves the business risk you actually have.

Kafka gives teams a powerful recovery primitive because the log can be replayed. That does not make recovery safe by itself.

Lag tells you the consumer moved. Recovery evidence tells you whether the system became correct again.