Designing a Polling Pipeline for Rate-Limited APIs Without Losing Data

Every system that ingests data from someone else's API eventually writes a poller. It starts as fifteen lines in a cron job: hit the endpoint, compare against the last timestamp, insert what's new. It works immediately, which is the problem — it keeps appearing to work long after it has started losing records.

I maintain an ingestion layer that polls public discussion APIs continuously, and the gap between "the poller is running" and "the poller is correct" took considerably longer to close than I expected. What follows are the four failure modes that caused real data loss, and the patterns that fixed them.

None of this is exotic. It is the set of things I wish had been in one place before I wrote the first version.

Failure Mode 1: The Fixed Interval

The naive poller runs on a fixed schedule — every 60 seconds, every source, forever.

This is wrong in both directions simultaneously. High-volume sources produce more records between polls than a single page returns, so you lose everything past the page boundary. Low-volume sources return nothing on the vast majority of calls, burning quota that the high-volume sources needed.

With a shared quota across sources, the fixed interval guarantees you are under-sampling exactly where the data is.

The fix is to let observed yield drive the interval. If the last poll came back full, poll sooner. If it came back empty, back off.

interface PollState {
  sourceId: string;
  intervalMs: number;
  consecutiveEmpty: number;
}

const MIN_INTERVAL = 30_000;
const MAX_INTERVAL = 1_800_000; // 30 min

function nextInterval(state: PollState, recordsFound: number, pageSize: number): number {
  // Page came back full — we are probably truncating. Poll aggressively.
  if (recordsFound >= pageSize) {
    return MIN_INTERVAL;
  }

  if (recordsFound > 0) {
    // Productive but not saturated. Ease toward the current rate.
    return clamp(state.intervalMs * 0.8, MIN_INTERVAL, MAX_INTERVAL);
  }

  // Empty. Back off geometrically, but bounded.
  const factor = 1.5 ** Math.min(state.consecutiveEmpty, 6);
  return clamp(MIN_INTERVAL * factor, MIN_INTERVAL, MAX_INTERVAL);
}

function clamp(n: number, lo: number, hi: number): number {
  return Math.min(hi, Math.max(lo, n));
}

The condition worth highlighting is recordsFound >= pageSize. A full page is not a success signal — it is evidence that records may exist beyond what you fetched. Treat it as a truncation warning and either paginate immediately or shorten the interval. Systems that treat a full page as "great, got everything" lose data proportional to source velocity, which means they lose the most from the sources that matter most.

After deploying adaptive intervals, our total request volume dropped by roughly 40% while records captured went up, because quota moved from silent sources to active ones.

Failure Mode 2: The Watermark Boundary

The standard incremental pattern stores the timestamp of the newest record seen and requests everything after it:

SELECT * FROM source WHERE created_at > :watermark ORDER BY created_at

This loses records, for three independent reasons.

• Clock skew. The upstream timestamp is assigned by their server, not yours. Distributed writers mean created_at is not guaranteed to be monotonic in the order records become visible to the API.
• Visibility lag. A record can be written at T but not appear in query results until T+2s due to replication or index lag. Poll at T+1s, advance the watermark past T, and that record is never returned.
• Ties. Multiple records sharing the same timestamp, split across a page boundary, silently drop the remainder when you use a strict >.

The fix is to rewind the watermark by a safety margin and deduplicate downstream:

const OVERLAP_MS = 120_000; // tune to observed upstream lag

function queryWindow(watermark: number): { since: number; until: number } {
  return {
    since: watermark - OVERLAP_MS,
    until: Date.now(),
  };
}

This deliberately re-fetches records you already have. That is the point — you trade a small amount of redundant bandwidth for the guarantee that a record has to be missed by two consecutive polls to be lost permanently.

Size OVERLAP_MS From measurement, not intuition. Log the delta between upstream created_at and your own ingest time, take the 99th percentile, and double it. Ours sat around 40 seconds at p99, so 120 seconds gives real headroom without meaningful cost.

Failure Mode 3: Deduplication That Isn't

Overlap windows only work if dedup is correct, and ID-based dedup handles less than people assume.

Upstream records get edited. A post whose body changes after ingestion keeps its ID, so an INSERT ... ON CONFLICT DO NOTHING on the ID silently discards the update. Meanwhile, some sources reuse or re-issue identifiers across deletion, so the ID alone is not always a stable key either.

Hash the fields that constitute a meaningful change, and store both:

import { createHash } from "node:crypto";

function contentHash(record: SourceRecord): string {
  // Only fields whose change means "this record is materially different."
  // Volatile counters (score, comment count) are excluded on purpose.
  const canonical = JSON.stringify({
    id: record.id,
    title: record.title ?? "",
    body: record.body ?? "",
    author: record.author ?? "",
  });
  return createHash("sha256").update(canonical).digest("hex");
}

INSERT INTO records (source_id, external_id, content_hash, payload, first_seen_at, last_seen_at)
VALUES ($1, $2, $3, $4, now(), now())
ON CONFLICT (source_id, external_id) DO UPDATE
SET content_hash  = EXCLUDED.content_hash,
    payload       = EXCLUDED.payload,
    last_seen_at  = now(),
    revision      = records.revision + 1
WHERE records.content_hash IS DISTINCT FROM EXCLUDED.content_hash;

Two details carry most of the weight. The WHERE clause means the overlap window costs nothing when nothing changed — no write amplification, no spurious updated_at churn, and downstream consumers watching for changes don't get woken up by re-fetches.

And excluding volatile counters from the hash is what makes this viable at all. Include a score that ticks every few seconds and every single poll registers as a change, which defeats the purpose and turns your overlap window into a write storm.

Failure Mode 4: Retry Storms and Undetected Gaps

When a rate-limited API returns 429, the instinct is to retry. Retry immediately, and you extend the penalty. Retry on a fixed delay across N workers, and they resynchronize into a thundering herd that hits the limit together, forever.

Most rate-limited APIs tell you what you need in response headers. Read them instead of guessing:

interface RateLimitState {
  remaining: number;
  resetAt: number;
}

function parseRateLimit(res: Response): RateLimitState | null {
  const remaining = res.headers.get("x-ratelimit-remaining");
  const reset = res.headers.get("x-ratelimit-reset");
  if (remaining === null || reset === null) return null;

  return {
    remaining: Number(remaining),
    // Some APIs send seconds-until-reset, others an absolute epoch. Normalize.
    resetAt: Number(reset) > 1e9 ? Number(reset) * 1000 : Date.now() + Number(reset) * 1000,
  };
}

async function backoff(attempt: number, res?: Response): Promise<void> {
  const retryAfter = res?.headers.get("retry-after");
  if (retryAfter) {
    const ms = Number(retryAfter) * 1000;
    if (Number.isFinite(ms)) return sleep(ms);
  }

  // Full jitter — see AWS Builders' Library on backoff.
  const ceiling = Math.min(60_000, 1000 * 2 ** attempt);
  return sleep(Math.random() * ceiling);
}

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

Full jitter matters more than the backoff curve. Exponential backoff with a deterministic delay keeps clients synchronized; randomizing across the whole interval is what actually spreads load. Retry-After, When present, it is authoritative — honor it over your own calculation.

The subtler problem is that these failures are invisible. A source that throws 429 for six hours produces no errors in your application logs if the poller catches and backs off correctly. It just quietly returns nothing, and your data has a hole in it that nobody notices until someone asks why a report looks wrong.

Detect gaps explicitly rather than inferring them from error rates:

async function detectGaps(sourceId: string): Promise<Gap[]> {
  const { rows } = await db.query(`
    SELECT
      window_end   AS gap_start,
      LEAD(window_start) OVER (ORDER BY window_start) AS gap_end
    FROM poll_log
    WHERE source_id = $1 AND status = 'ok'
      AND window_start > now() - interval '48 hours'
    ORDER BY window_start
  `, [sourceId]);

  return rows
    .filter(r => r.gap_end && r.gap_end - r.gap_start > GAP_THRESHOLD_MS)
    .map(r => ({ sourceId, from: r.gap_start, to: r.gap_end }));
}

This requires logging every poll attempt with its requested window and outcome — a table most people skip because it feels like overhead until the first time it tells you about four hours of missing data you would otherwise never have found. Feed detected gaps into a low-priority backfill queue that runs when quota is available.

What Actually Matters

If you take three things from this:

• Log the window, not just the result. Without a record of what range each poll requested, gaps are undetectable. This is the single highest-value thing to add to an existing poller.
• Overlap deliberately, dedup by content. A safety margin plus content-hash dedup converts "we might have missed something" into "a record must be missed twice to be lost."
• Treat a full page as a warning. It is the most common silent truncation point in production pollers, and it is one comparison to check for.

None of these are difficult. They are just easy to omit, and their absence produces a system that looks healthy while quietly dropping records.

I work on the ingestion layer at SignalMelo, which polls public discussion sources continuously — the patterns above came out of production incidents there. Disclosure: it's a commercial product I help build.