Building an Idempotent Job Queue in Node. js That Never Runs the Same Task Twice

Today, in modern backends, you probably have those distributed job queues for everything, including sending emails, processing payments, generating reports, and syncing data to third parties. As soon as you add retries to handle transient failures, however, you inherit a hard problem: how do you ensure that when the network, worker, or broker can fail at any point, your job runs exactly once?

The short answer is: "exactly once delivery" is a great concept, but in practice it's mostly fiction given the nature of distributed systems. What you really can make is at-least-once delivery + idempotent processing, yielding exactly once effects. This article demonstrates how to accomplish this in Node.js with a tangible, functioning implementation.

The Problem: Retries Cause Duplicates

Take a worker that charges the customer and then marks the job completed

async function processJob(job) {
  await chargeCustomer(job.customerId, job.amount);
  await markJobComplete(job.id);
}

This seems fine until you consider that the work crashes after chargeCustomer succeeds but before markJobComplete executes. Because the queue does not receive an acknowledgement, it redelivers the job. The customer gets charged twice.

This is not a rare edge case. Do any significant amount of throughput and workers fall over, containers reschedule, network calls default after the server has already worked its way through them. If you have side effects in your job, then you can always assume any job may be delivered more than once.

The Solution: Idempotency Keys

The main concept is to give every job created a unique, deterministic idempotency key and log the output of processing that key. The worker only checks if a key has been processed before doing any work. If so, it simply returns the result that was saved and does not redo the work.

Here is the schema for how we can keep track of processed jobs.

CREATE TABLE processed_jobs (
  idempotency_key VARCHAR(255) PRIMARY KEY,
  status          VARCHAR(20) NOT NULL,  -- 'in_progress' | 'completed'
  result          JSONB,
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  completed_at    TIMESTAMPTZ
);

The job must encode its sensitive payload and key, not randomly generated at enqueue time. Good keys will be things like charge:order_12345, which will hopefully be stable across retries of the same logical operation.

A Working Implementation

The trick is to acquire the key atomically before doing anything useful. To claim the job, we execute a single atomic operation in PostgreSQL, which is an INSERT... ON CONFLICT DO NOTHING:

const { Pool } = require('pg');
const pool = new Pool();

async function processIdempotent(idempotencyKey, work) {
  const client = await pool.connect();
  try {
    // Step 1: Try to claim the key atomically.
    const claim = await client.query(
      `INSERT INTO processed_jobs (idempotency_key, status)
       VALUES ($1, 'in_progress')
       ON CONFLICT (idempotency_key) DO NOTHING
       RETURNING idempotency_key`,
      [idempotencyKey]
    );

    // Step 2: If we did NOT claim it, someone else already did.
    if (claim.rowCount === 0) {
      const existing = await client.query(
        `SELECT status, result FROM processed_jobs
         WHERE idempotency_key = $1`,
        [idempotencyKey]
      );
      const row = existing.rows[0];
      if (row.status === 'completed') {
        return row.result; // Return the cached result — no double work.
      }
      // Still in progress elsewhere — let the queue retry later.
      throw new Error('JOB_IN_PROGRESS');
    }

    // Step 3: We own the key. Do the actual work.
    const result = await work();

    // Step 4: Record the result.
    await client.query(
      `UPDATE processed_jobs
       SET status = 'completed', result = $2, completed_at = now()
       WHERE idempotency_key = $1`,
      [idempotencyKey, result]
    );

    return result;
  } finally {
    client.release();
  }
}

Now the worker becomes:

async function processJob(job) {
  return processIdempotent(`charge:${job.orderId}`, async () => {
    const charge = await chargeCustomer(job.customerId, job.amount);
    return { chargeId: charge.id };
  });
}

The key is already completed, and whenever this job is delivered the second (or more) time it will return the chargeId that was previously stored without charging again.

Handling the Stuck "in_progress" Case

The last failure mode that remains is where a worker picks a key, sets it to in_progress, and then dies without completing. Now this key is stuck, and any retry gives JOB_IN_PROGRESS forever.

The solution is an expiration-lease for the lease. 1. Add locked_until column, make expired lock reclaimable:

const claim = await client.query(
  `INSERT INTO processed_jobs (idempotency_key, status, locked_until)
   VALUES ($1, 'in_progress', now() + interval '5 minutes')
   ON CONFLICT (idempotency_key) DO UPDATE
     SET locked_until = now() + interval '5 minutes',
         status = 'in_progress'
     WHERE processed_jobs.status = 'in_progress'
       AND processed_jobs.locked_until < now()
   RETURNING idempotency_key`,
  [idempotencyKey]
);

It only requires a lock if the circuit is in progress and its lease has timed out, which means that some worker abandoned it earlier. The completed jobs will never be reclaimed, because the WHERE excludes them.

Why not simply use a distributed lock

One of the most common instincts here is to grab ourselves a Redis lock (if not using redis-lock, do SETNX with a TTL). Despite a lock being a solution for mutual exclusion, they do not solve idempotency by themselves. The job is already done, but because it uses a lock to prevent two workers from running at once. If you only use a lock, the job will be reprocessed when the lock expires and a redelivery is attempted. What you need is a permanent record of completion and that is what the processed jobs table provides. Locks and idempotency keys address two separate problems, yet durable systems typically require both.

Takeaways

• Assume at-least-once delivery; make each job handler idempotent.
• Use the intent from job to derive idempotency keys; ensure they are stable across retries.
• Store results and claim keys atomically with INSERT... ON CONFLICT so duplicates return the cached result.
• Lease with an expiration because crashed workers should not block a key forever.

Idempotency is certainly not useful, but it helps the queue to be the difference between something you can trust and a facility that will silently double charge your customers because of load. Treat it as a first-class citizen, because adding it via retrofitting after failing is way worse.