How to Design a Distributed Job Scheduler

Almost every backend eventually needs to run code on a schedule. Send the invoice at midnight. Retry the failed payment in five minutes. Generate the weekly report every Monday at 7 AM. Clean up expired sessions every hour.

On one server, this is easy. You write a cron line and move on. The trouble starts when one server becomes ten. Now the same cron line lives on every box, so the invoice job fires ten times instead of once. Move the cron to a single “scheduler” box, and that box becomes a single point of failure. Every time you deploy new code, that process restarts, and if it crashes or the host dies, there is no second node to cover for it. Any job due during that downtime window silently never fires.

A distributed job scheduler solves this. It runs jobs reliably across a fleet of machines, fires each job once even when nodes crash, and keeps working when parts of the system fail. This post walks through how to design one, the trade-offs at each step, and the mistakes that bite teams in production.

What the Scheduler Has to Do

Before drawing boxes, it helps to pin down the requirements. They split into two groups.

Functional requirements:

• Run a job once at a specific time (a one-time job).
• Run a job on a repeating schedule, usually defined with cron (a recurring job).
• Support job dependencies, where job B runs only after job A succeeds.
• Retry a job automatically when it fails.
• Respect priority, so urgent jobs run before bulk jobs.
• Cancel or pause a job that is scheduled or already running.

Non-functional requirements:

• Durability. Once the system accepts a job, it must not lose it, even if a node dies one second later.
• At-least-once execution. Every due job runs at least one time.
• Scale. The design should handle millions of jobs per day across many workers.
• Fault tolerance. A crashed worker must not block other jobs, and its work should be picked up by someone else.

One requirement is worth calling out early. People often ask for “exactly-once” execution. In a distributed system, you cannot truly get it. What you can build is at-least-once delivery plus idempotent jobs, which together behave like exactly-once from the outside. More on that later.

The Core Architecture

The single most important idea in this design is to separate deciding when a job runs from actually running it. These are two different problems with different scaling needs, so they become two different components.

A clean design has four parts:

1. A scheduler that watches the clock and decides which jobs are due.
2. A queue that holds ready-to-run jobs and hands them out.
3. A pool of stateless workers that pull jobs and execute them.
4. A datastore that holds job definitions and execution history, and acts as the source of truth.

Why decouple the queue from the workers at all? Because load is bursty. At midnight, a thousand daily jobs may become due at the same second. If the scheduler called workers directly, that spike would hit them all at once. The queue absorbs the spike and lets workers drain it at a steady rate. It also lets you scale workers up and down without touching the scheduler. This is the same reason queues show up across system design, which I covered in detail in Role of Queues in System Design.

Modeling Jobs in the Database

The datastore is the source of truth, so the schema matters. A common approach uses two tables. One holds the recurring definition, the other holds individual runs.

CREATE TABLE jobs (
    id            BIGINT PRIMARY KEY,
    name          TEXT NOT NULL,
    cron          TEXT,            -- null for one-time jobs
    payload       JSONB,
    next_run_at   TIMESTAMPTZ,     -- when this job is next due
    enabled       BOOLEAN DEFAULT TRUE
);

CREATE TABLE job_runs (
    id          BIGINT PRIMARY KEY,   -- unique id per run
    job_id      BIGINT REFERENCES jobs(id),
    status      TEXT NOT NULL,        -- PENDING, RUNNING, SUCCEEDED, FAILED, DEAD
    attempt     INT NOT NULL DEFAULT 1,
    scheduled_at TIMESTAMPTZ,
    started_at   TIMESTAMPTZ,
    lease_until  TIMESTAMPTZ
);

CREATE INDEX idx_jobs_due ON jobs (next_run_at) WHERE enabled = TRUE;

The partial index on next_run_at is the workhorse. The scheduler asks “which jobs are due now” many times per second, and this index keeps that query fast even with millions of rows.

Each run moves through a small set of states. Drawing the state machine makes the retry and failure logic obvious.

Defining Schedules With Cron

Recurring jobs need a way to express “every day at 2:30 AM” or “every 15 minutes.” Cron is still the standard. A classic cron expression has five fields:

 minute hour day-of-month month day-of-week
   30     2       *          *        *        -> 2:30 AM every day

The Java world often uses Quartz cron, which adds a seconds field at the front and a year field at the end, giving six or seven fields. The two formats look similar but are not interchangeable, and mixing them up is a frequent source of jobs that never fire.

The scheduler stores the cron string and computes a concrete next_run_at timestamp from it. After a run is enqueued, it computes the next one. This raises a real question: what happens if the scheduler was down for an hour and three runs were missed? This is the misfire problem. You generally pick one of two policies:

• Catch up. Run every missed occurrence in order. Correct for billing, expensive for everything else.
• Skip. Run only the next future occurrence and forget the missed ones. Right for jobs like cache refreshes where stale runs add no value.

Make this an explicit setting per job. Teams that leave it implicit get surprised after the first outage.

Picking Which Jobs to Run

The scheduler needs to find due jobs and hand them off. There are three common ways to find them.

• Polling. Every second, query the database for jobs where next_run_at <= now(). Simple and reliable. The partial index keeps it cheap. The cost is a small delay, up to your poll interval.
• Timer wheel. Keep upcoming jobs in an in-memory structure sorted by time. Very precise and great for short delays, but you have to rebuild it from the database after a restart.
• Push. An external timing service fires an event when a job is due. Real-time, but now you depend on another moving part.

For most systems, polling with a one-second interval is the right default. It is boring, and boring is good for a component you are trusting with billing runs.

The harder problem is concurrency. If you run several scheduler instances for availability, they will all poll the same table at the same time. Without care, two of them pick the same job, and it runs twice. The clean fix in PostgreSQL is row locking with SKIP LOCKED:

SELECT id FROM jobs
WHERE enabled = TRUE AND next_run_at <= now()
ORDER BY next_run_at
LIMIT 100
FOR UPDATE SKIP LOCKED;

FOR UPDATE locks the rows this instance selects. SKIP LOCKED tells other instances to ignore locked rows and grab the next free ones instead. Many schedulers can now poll in parallel, each claiming a different batch, with no coordination service and no duplicate pickups. Airflow uses exactly this approach instead of a heavier consensus protocol, which is a good reminder that the simplest mechanism that meets the requirement usually wins.

Why Exactly-Once Is a Myth

Here is the scenario that breaks naive designs. A worker pulls a job, runs it successfully, and then crashes before it can tell the system “done.” The system still thinks the job is running. The lease expires, another worker picks it up, and the job runs a second time. You charged the card twice.

You cannot delete this scenario. Networks drop messages and processes die at the worst moment. So you stop chasing exactly-once delivery and instead make the work safe to repeat. That means two things working together:

1. At-least-once delivery. The system guarantees a due job runs at least one time, accepting that it may occasionally run more than once.
2. Idempotent jobs. Running the same job twice has the same effect as running it once.

The standard trick is an idempotency key built from stable identifiers, for example {job_id, run_id, attempt}, or a key tied to the business action like invoice_2026_06_charge. The worker records that key before committing side effects. If the same key shows up again, the worker sees the work is already done and acknowledges without repeating it.

This is why each run gets its own unique id. A time-ordered id such as a Snowflake id or a ULID works well, because it is unique across the whole fleet without coordination and it sorts by creation time, which keeps the job_runs table naturally ordered. I explained the structure of these ids in How Snowflake IDs Work, and the deduplication pattern itself in Idempotent Receiver Pattern.

There is one more subtle gap. The worker has to update the database and publish to the queue, and those are two systems. If it writes to the database and then dies before publishing, the job is lost. The transactional outbox pattern closes this gap by writing the job and an outbox row in one local transaction, then publishing from the outbox separately. I covered that in The Transactional Outbox Pattern.

Coordinating at Scale

A single scheduler instance has a throughput ceiling. Past a certain number of jobs per second, one process polling one database cannot keep up. There are two ways to grow.

The first is leader election. You run several scheduler instances, but only one is active at a time. The others stand by and take over if the leader dies. A coordination service like etcd or ZooKeeper holds the leadership lock. This is simple to reason about, but the single active leader is still a throughput bottleneck.

The second is sharding. You split the job space across many active schedulers. A simple scheme hashes the job id into one of N partitions, and each scheduler owns a set of partitions. Every job has exactly one owner, so there are no duplicate pickups, and throughput grows by adding schedulers. Consistent hashing makes it cheaper to add or remove schedulers without reshuffling everything.

Sharding has one sharp edge. During a handover, while leases for a partition are changing hands, two schedulers can briefly believe they own the same partition. This is split brain. You do not try to make it impossible, because that is expensive. Instead, you let the worker-side idempotency check be the final safety net. If both schedulers enqueue the same run, the idempotency key means it still executes once.

Google’s cron service takes a stricter route for its most sensitive launches. It writes the launch record to a quorum using Paxos before the job actually starts, so a failover cannot lose or double-fire it. For most teams, leases plus idempotency are enough, and full consensus is overkill.

Detecting Failures and Recovering

Workers crash. The scheduler has to notice and reassign their work, without stealing jobs from workers that are simply slow.

The mechanism is a lease with a heartbeat. When a worker claims a run, it sets lease_until to a short time in the future, say 30 seconds. While the job runs, the worker periodically extends the lease. If the worker dies, it stops extending, the lease expires, and a recovery sweep moves the run back to PENDING so another worker can take it.

-- recovery sweep: reclaim runs whose lease has expired
UPDATE job_runs
SET status = 'PENDING'
WHERE status = 'RUNNING' AND lease_until < now();

Two details make this robust. First, the lease timeout must be comfortably longer than a normal heartbeat interval, or a brief pause will cause a healthy job to be wrongly reclaimed. Second, you need protection against a zombie worker, one that froze on a long garbage collection pause, lost its lease, and then woke up and tried to finish writing results. A fencing token solves this. The reclaimed run gets a higher token, and the datastore rejects any write carrying an older token. I went deeper on time-bound ownership and fencing in The Lease Pattern in Distributed Systems.

Retries Done Right

A failed job should usually be retried, but retrying badly makes outages worse. If a downstream service is struggling and every failed job retries immediately, you pile on more load at the exact moment it can least handle it.

The fix is exponential backoff with jitter. Each retry waits longer than the last, and a random jitter spreads the retries out so they do not all fire at the same instant.

attempt 1 fails -> wait ~1s
attempt 2 fails -> wait ~2s
attempt 3 fails -> wait ~4s
attempt 4 fails -> wait ~8s
(each wait randomized by +/- a few hundred ms)

After a fixed number of attempts, stop. A job that keeps failing should not retry forever. Move it to a dead letter queue, a separate place for runs that exhausted their retries, and alert a human. The dead letter queue keeps a poisoned job from clogging the pipeline while preserving it for investigation.

Operating the Thing

A scheduler is infrastructure other teams depend on, so it has to be observable and controllable.

For observability, track the metrics that tell you the system is healthy:

• Queue depth. A queue that keeps growing means workers cannot keep up.
• Scheduling lag, the gap between when a job was due and when it actually started.
• Run outcomes per minute, split by succeeded, failed, and dead.
• Lease reclaims, which spike when workers are crashing.

For control, give operators real knobs. They should be able to pause a queue, drain a worker before a deploy so it finishes current jobs and takes no new ones, and replay a dead-lettered job after fixing the cause. Building these in from the start saves a lot of pain during the first incident.

How Real Systems Approach This

None of this is theoretical. The same building blocks show up across well-known tools, each making a different trade-off.

• Quartz. A mature Java scheduler. Multiple instances coordinate through a shared database using row locks, the same idea as the SKIP LOCKED approach above.
• Airflow. Orchestrates dependency graphs of tasks. Its scheduler uses database locks rather than a consensus protocol, favoring operational simplicity.
• Temporal. Models workflows as code and replays an append-only event history to recover state after a crash, which sidesteps a whole class of mid-task failure bugs.
• Celery. A popular task queue in Python, with a beat component that handles periodic scheduling.
• Kubernetes CronJobs. Run containerized jobs on a cron schedule inside a cluster, with configurable policies for missed runs and concurrency. See the Kubernetes CronJob docs.
• Google distributed cron. Writes launch state to a Paxos quorum before launching, so a leader failover never loses or doubles a run.

The pattern across all of them is consistent. Decouple scheduling from execution, lean on the database or a quorum for coordination, accept at-least-once and make jobs idempotent, and design for failure as the normal case.

Takeaways

If you remember five things from this, make it these.

1. Separate the decision of when a job runs from the work of running it. They scale differently.
2. Do not chase exactly-once. Build at-least-once delivery and make every job idempotent.
3. Use the database as a coordination primitive. SELECT ... FOR UPDATE SKIP LOCKED lets many schedulers poll safely.
4. Use leases with heartbeats and fencing tokens to detect dead workers and reclaim their runs without double execution.
5. Retry with exponential backoff and jitter, cap the attempts, and send the rest to a dead letter queue.

A good scheduler is not clever. It is careful. It assumes nodes will die, messages will duplicate, and clocks will drift, and it keeps running anyway.