Treating Web Scrapers Like Production Services

Most teams treat a broken web scraper as a chore. Someone notices bad data, opens the page in a browser, patches a selector, and redeploys. That works when you have one scraper and someone who looks at its output every day. It stops working when you have a dozen, and the breakage surfaces a week later as a strange number in a downstream report. In this article, I walk through how I built scrapers that are cataloged, health-checked, traced, and repaired like any other production service, with an LLM proposing fixes and plain Python deciding whether those fixes ship.

The project came out of the WeMakeDevs hackathon, where it took first place. The product itself is small: it pulls about twelve days of TV guide data across three streaming providers, ranks the top ten upcoming movies and top ten upcoming shows, and lets you record any of them on Philo with one click. I am going to skip the product and focus on the plumbing: the drift rules, the state model, the verifier, and the code behind each. Those pieces carry over to any team that runs scrapers, or to any team that lets an LLM modify a live system.

The architecture follows one split. Diagnosis is probabilistic; acceptance is not. The LLM figures out what broke and proposes a repair. A deterministic diff against a frozen sample decides whether that repair is correct, and nothing gets promoted without passing it.

Why Do Scrapers Fail, and Why Old Fixes Are Not Enough

The usual way to build a scraper is with structural selectors. Someone opens the page, finds that the airtime lives in div.card > span:nth-child(3), and hard-codes it. This is fragile by nature because it describes where something is, not what it is. Rename a CSS class and it fails. Reorder elements and it fails. Wrap the time in a new tag and it still fails.

There is a newer, AI-based approach: you write what the data means, and a model finds it in the webpage DOM. For my project, I tried Bright Data's Scraper Studio, which creates a collector from a natural-language prompt. When you build the channel-schedule scraper, the prompt example reads like the following:

PROMPT = (
    "This page lists a TV channel's upcoming schedule as program cards. Extract every "
    "program card. For each: the program title as title, the episode title if shown as "
    "episode_title, the airing date exactly as displayed (like Sat, Aug 22) as date_raw, "
    "the start time exactly as displayed (like 2:00 PM) as start_raw, the time range text "
    "if shown (like Sat, Aug 22 2:00 PM - 2:30 PM) as time_range_raw, the category text "
    "as category. Include every card.")

Notice what is not in there: no class names, no element positions, no DOM paths. The prompt describes everything. That matters for two reasons. First, a semantic description survives a redesign far better than a selector does, and second, this is the part that took me a while to appreciate: the prompt is the healing artifact. When the scraper drifts, the repair tool reasons over that same description plus a plain-language symptom report. A prompt that said "the third span inside each grid cell" would give the healer nothing to work with.

Also notice date_raw and start_raw. I always capture the raw displayed string alongside any parsed value. If start_raw is "8:00 PM" but start_utc parsed to null, that is not a data quirk. That is a drift signal, and one of the detection rules below keys on exactly that.

AI extraction alone does not solve the problem, though. It reduces the breakage rate, but it cannot eliminate it, and it introduces a new risk. When you ask a model to repair a scraper, how do you know the repair is correct? A model will confidently produce output for a broken extraction. Two rows that look plausible are still two rows when there should be thirty.

Scrapers as Cataloged Services With Health States

The first design decision was to stop treating scrapers as scripts and start treating them as services with an on-call rotation. Each scraper is a cataloged entity in an internal developer portal in Port IO, another vendor in the hackathon, with a health property that moves between healthy, drifting, and broken. Every scrape run is its own entity with a status of success, drift, or error. Every repair attempt is a heal_event with its trigger being the drift description.

Once scrapers are entities with state, the rest of the platform falls out. Workflows react to entity changes. A scrape_run landing in success triggers publish. A scrape_run landing in error or drift triggers heal. Scheduled workflows kick off the daily scrape at 08:00 UTC and the catalog refresh at 07:30. A scorecard grades each scraper from Basic through Bronze, Silver, and Gold based on whether it verified today, whether verification is passing, and whether it is currently healthy. An operator dashboard shows all the scrapers by health, the recent health events, and the current ranked output.

None of this is new. It is what you would do for any microservice. The point is that scrapers rarely get this treatment, and they need it more than most services because their upstream is a third party that ships changes without telling you.

Detecting Drift Deterministically

Before you can repair anything, you have to notice what is broken, and you have to notice it without a human staring at the output. I wrote seven deterministic drift rules that run on every scrape in plain Python, with no model involved. The thresholds live in one place:

REQUIRED_FIELDS = ("channel", "title", "start_utc")
NULL_RATE_LIMIT = 0.10
EMPTY_PAGE_LIMIT = 0.20     # >20% of channel pages returning nothing is breakage
CHANNEL_COLLAPSE = 0.80     # <80% of the requested channels yielding rows is breakage
LINEUP_DELTA_LIMIT = 0.10   # >10% channel-set churn needs two confirming runs

The row band sets how many rows a single channel page should yield. A page returning 3 cards for a channel that airs around 40 programs a day is broken, not news; so is one returning thousands.

PAGE_ROW_BANDS = (15, 800)

Alongside the row band, the rules check the null rate of required fields, the case where a raw time is present but the parsed time is null, channel collapse (fewer than 80 percent of requested channels yielding rows), the empty-page rate, a change in the schema hash of returned keys, and lineup disappearance.

That last one deserves a note because it exposes the hardest problem in drift detection. A broken scraper and a legitimately changed source look identical in a single sample. If 15 percent of channels vanish from a provider's lineup, that could be a scraper failure, or it could be a real carriage change, which happens regularly with streaming lineups. I handle it two ways. Lineup churn requires two consecutive runs agreeing before it is accepted as real. A single run showing the change is flagged as unconfirmed drift. If the scraper still reproduces the expected output from a frozen HTML page, the scraper is fine, and the source actually changed.

delta = _lineup_delta(prev["rows"], rows)
if delta > LINEUP_DELTA_LIMIT:
    signature = f"lineup:{_channel_sig(rows)}"
    seen = state.confirm_change(state_key, signature)
    if seen < 2:
        drift.add("lineup_change",
                  f"channel set moved {delta:.0%} (unconfirmed, run {seen}/2)")

One problem I encountered: a single collector serves all three providers because every channel page on the guide site has the same structure. But drift state has to be keyed per collector:provider pair because comparing one provider's channel set to another provider's snapshot reads as mass channel loss. 

When drift is detected, two things happen. The pipeline immediately sets the scraper's health to drifting in the catalog (the fast path, within seconds). It also increments a scraper.drift.detected metric that an alert rule in the observability stack (SigNoz, the third hackathon sponsor) watches, firing a webhook roughly five minutes later (the safety net, and the only path that catches a pipeline that died entirely and never ran the inline detector). Both paths hit the same idempotent /heal endpoint. Whichever fires first does the work; the other finds the scraper already healthy and exits.

The Repair Loop and the Gate

Here is the heal flow, with the part I care most about in the middle:

/heal → scraper heal (semantic symptom description) → preview_result
                                                           |
                                          Verifier: diff vs frozen fixture
                                                           |
                                               ┌───────────┴───────────┐
                                              pass                    fail
                                               |                       |
                                            approve                  reject
                                               |                       |
                                            healthy                  broken
                                                                       |
                                                                  human queue

The heal itself is delegated. Bright Data's scraper /heal command takes a collector ID and a plain-language description of what broke, reasons over the original semantic prompt, and returns a proposal in an awaiting_approval state with a preview_result of what the repaired scraper would extract. In my system, a Triage agent in the developer portal authors that symptom description by reading the run and heal history. That is judgment, and I was skeptical of handing it to a model, so I decided to introduce a verifier. The verifier is deliberately not an agent:

"""The Verifier -- deliberately NOT an agent.

Judgment goes to the LLM agent; proof stays deterministic. A verification gate you
cannot trust deterministically is not a gate, so this is plain code: normalise both
sides, diff, return a verdict. It is the only thing allowed to authorise a promotion.

"""

COMPARE_FIELDS = ("title", "date_raw", "start_raw")

def verify(provider: str, rows: list[dict]) -> Verdict:
    want, got = _normalise(expected(provider)), _normalise(rows)
    if want == got:
        return Verdict(True, [])
    diff = []
    missing = [w for w in want if w not in got]
    extra = [g for g in got if g not in want]
    if len(want) != len(got):
        diff.append(f"row count {len(got)}, expected {len(want)}")
    for item in missing[:3]:
        diff.append(f"missing {item}")
    for item in extra[:3]:
        diff.append(f"unexpected {item}")
    return Verdict(False, diff)

It normalizes both sides (strip, lowercase, round floats), sorts, and compares. If the diff passes, the healer approves the proposal and sets health back to healthy. If it fails, the healer explicitly rejects the proposal, sets health to broken, writes a heal_event with outcome rolled_back, and leaves the scraper in a human's queue.

The fixture that backs this is three files: channel.html, a real captured channel page trimmed to 30 program cards; channel.expected.json, the hand-checked correct output for that exact page; and channel.mutated.html, the same 30 programs with every structural hook deliberately broken. Classes are renamed, card order is shuffled, and airtimes sit inside extra wrapper elements. That file lets the tests and the demo trigger drift on demand. A selector-based scraper fails on it; a semantic one should not.

I enforce two rules about fixtures, and I wrote them into the repo's agent instructions so the coding agent working on the codebase would respect them too. Fixtures are frozen inputs (hardcoded schema) never regenerated to make a test pass, and a rejected bad repair is a success, not a failure. That second one is counterintuitive until you think about what the alternative is. A repair that gets promoted without proof is exactly how a broken scraper silently poisons everything downstream.

Walking Through a Rejected Repair

The case this design exists for is a repair that looks fine and is wrong. Here is the state sequence from a real end-to-end run where that happened:

1. A fault-injected run produced a scrape_run entity with status error.
2. The event workflow fired, and the Triage agent wrote a drift description from the run and heal history.
3. The /heal endpoint sent that description to the vendor and got back a proposal in awaiting_approval, with a preview_result.
4. The verifier diffed the preview against channel.expected.json and returned row count 2, expected 30.
5. The healer rejected the proposal, set health to broken, wrote a heal_event with outcome rolled_back, and queued it for a human.
6. The next clean scheduled run passed every drift rule and moved health back to healthy.

The proposal in step 3 was well-formed and confidently produced. Without step 4, it would have been promoted as the fix, and the ranking would have been built from 2 programs instead of 30 while the system recorded the repair as a success.

Observability

To make every heal cycle inspectable after the fact, the whole pipeline runs under OpenTelemetry auto-instrumentation with traces, metrics, and logs shipped to SigNoz (self-hosted locally and exposed over ngrok). The trace for a scrape run has one scrape.fetch span per channel page (tagged with provider, channel ID, row count, and fetch mode) followed by scrape.extract, scrape.validate, and when relevant scrape.heal.reprompt and scrape.verify.fixture. Because the FastAPI service itself is instrumented, even the developer portal's automation calling /heal shows up as a span.

The drift description, the heal prompt, and the verifier's diff are all logged inside the relevant span, so trace_id is attached automatically. 

The developer portal's own action and workflow runs, and the scraping vendor's remaining budget, are not things those systems emit anywhere, so a small background watcher samples both and re-emits them as metrics. Runs are de-duplicated in SQLite so a restart never double-counts.

Applying the Same Split to Shipping Code

Once the pattern worked for scraper repair, I applied it to the software change loop, too. A feature request submitted through the portal becomes a requirement entity, which triggers a local coding agent (Claude Code, running on my machine with file tools only) to edit a scratch clone. Then deterministic code takes over: branch, commit, push, open a PR with a fixed first-line contract that the deploy workflow parses later. GitHub Actions runs pytest, a TypeScript type check, and the Next.js build. A human reviews and merges. Only then does the deploy workflow ship it and flip the requirement to deployed.

The agent is forbidden from touching fixtures, the verifier module, and the drift thresholds, and the run aborts if the resulting diff touches any of them. The model gets to write the feature.

Implementation Checklist

If you build something similar, these are the details that mattered most in practice:

Store start_raw next to start_utc, and follow the same pattern for every parsed field. A raw value that is present alongside a null parsed value was the earliest and most dependable drift signal in this build.

If one scraper serves multiple providers, key drift state by collector:provider, not by collector. Otherwise, one provider's channel set gets compared to another's snapshot and reads as mass channel loss.

Route the inline detector and the alert webhook to the same /heal endpoint, and make it a no-op when the scraper is already healthy. With two triggers, duplicate calls are a certainty, not an edge case.

Require two consecutive runs to agree before accepting a lineup change as real. A single sample cannot tell a broken scraper from a changed source.

Write the "fixtures are frozen" rule down where both humans and coding agents will read it. The temptation to regenerate expected output to make a red test green is strong, and an agent will do it without hesitation if you have not told it not to.

Draw a hard line between diagnosis and acceptance. Diagnosis can be fuzzy. Acceptance should be code short enough to read on one screen and trust completely.

Limitations

To be fair about what this is and is not, the factory runs on the operator's machine, reached from the cloud through a tunnel, so the portal operates it but does not host it. The catalog scraper has no fixture because the catalog legitimately changes daily. There is no retry or backoff on vendor calls. And the Triage agent's reasoning over long run histories is shallow. It authors a good symptom description, but it is not yet doing root-cause analysis. 

But the core loop — detect deterministically, repair with judgment, promote only on proof — has now survived both the happy path and a real rejected repair, and I think it is the right shape for anyone letting an agent touch production data.