Freshness as a First-Class Schema Concern: Modeling Data Staleness Across 83 Data Sources

I am not a developer, and I built a public reef atlas with AI agents. It pulls from 83 data sources that each update on their own schedule, some daily, some weekly, some once a decade. The hardest problem in the whole build was staleness, harder than the ocean science and harder than the frontend: how do you build a schema that tells the truth about how old each piece of data actually is, when the sources age so differently?

The agents did more than write the code. They walked me through each schema decision as we went, explaining every tradeoff until I understood it well enough to make the next call myself, which is the only reason I can write it up now. It is a problem any app that blends live feeds with slow-moving records runs into, so here is how it showed up in the codebase and how the schema ended up solving it.

The Lie That Is Easy To Tell by Accident

Early on, every card in the atlas just showed a number. Coral cover: 32 percent. Fishing pressure: high. A user looking at that card has no way to know if the coral cover number is from a survey last month or a survey from 2010. Both render identically and feel equally current, which is exactly the problem: a UI that shows a number without its provenance is quietly asserting that all of its data is equally fresh. For us, that assertion was false in a way that mattered. A dive site can look improving on a stale 2010 baseline and be declining today.

So the real fix was a schema decision. Freshness needed to be a first-class field on every data record, present on everything, rather than a caveat a human remembers to add in the copy later.

Three Data Shapes

Once I actually mapped our 83 sources with an agent, they sorted into 3 distinct freshness shapes, and each one needed its own contract.

Live. Data with an automated ingest running on a schedule, where "updated" has a real, checkable timestamp. NOAA Coral Reef Watch thermal stress data refreshes daily at 06:30 UTC through a GitHub Actions cron job, no API key required, which made it the cleanest source to model against. Global Fishing Watch fishing pressure and IUCN Red List species status update weekly. For this shape, the schema stores an ISO timestamp and the UI is allowed to say the word "live," because it is actually true.

Snapshot. Data from a real survey with a real date attached, but no automated pipeline behind it, because the source organization itself does not publish on a schedule. A lot of coral cover falls here. NCRMP, the NOAA National Coral Reef Monitoring Program, does not expose an API, so its numbers update when a report gets published, not on any cadence we control. For many of our locations, that means only 2 coral cover data points exist, a baseline around 2010 and a current reading from 2024. That is a before and after. The schema has to carry a surveyDate, and the UI has to show how many years old that survey actually is, because a 2-year-old survey and a 14-year-old survey should not look the same on the page.

Presence. Data that confirms a species was observed somewhere, sourced from GBIF and OBIS, but carries no freshness claim and no population trend at all. It just says: this animal has been recorded here. A presence record has no trend that can go stale, so it carries no date at all and gets its own visual treatment, kept clearly apart from the numbers that do age.

What This Looks Like as an Actual Component

The pattern that made this maintainable was building one shared component, DataFreshnessLabel, with a discriminated union type instead of 3 different optional props bolted onto one interface.

type LiveProps = CommonProps & {
  variant: "live";
  source?: string;
  updatedAt?: string;
};

type SnapshotProps = CommonProps & {
  variant: "snapshot";
  surveyMethod: string;
  surveyDate?: string;
};

type PresenceProps = CommonProps & {
  variant: "presence";
  source?: string;
};

export type DataFreshnessLabelProps = LiveProps | SnapshotProps | PresenceProps;

The discriminated union does the enforcement work that a code review would otherwise have to do by hand. What it makes mandatory is the freshness shape itself: every value has to declare whether it is live, snapshot, or presence, and a snapshot will not compile without a surveyMethod. That is the whole reason to model it as a union, so the compiler checks the provenance contract at the call site instead of trusting a reviewer to remember it.

The survey date itself is deliberately optional, because some sources give a method and a rough vintage but no exact day, and I would rather model that gap than invent a precise date. What the union still guarantees is that a dateless snapshot renders as a snapshot. The date passes through a fmtDate helper that returns a literal dash when it is missing, so the label reads Snapshot · AGRRA · surveyed —, an explicit admission of unknown vintage. There is no shape in the union that renders as a bare, confident number, so the failure the article opened with cannot happen by accident.

Each variant also gets its own color and its own copy, on purpose. Live is emerald with a pulse dot. Snapshot is amber, and if the survey is more than 2 years old, the component computes that itself and appends "(X years ago)" directly onto the label, so the staleness is not something a reader has to go dig for.

function yearsAgo(iso?: string): number | null {
  if (!iso) return null;
  const d = new Date(iso.length === 10 ? iso + "T00:00:00Z" : iso);
  if (Number.isNaN(d.getTime())) return null;
  const years = (Date.now() - d.getTime()) / (365.25 * 24 * 3600 * 1000);
  return Math.floor(years);
}

Freshness Has To Reach the Classification Logic Too, Not Just the Label

The label solves the display problem. It does not solve the harder problem, which is that our core feature, classifying every reef as Improving, Stable, or Declining, is a derived value built on top of these mixed freshness inputs. The classification function pulls the worst thermal stress alert on record and the best coral cover reading on record, then applies thresholds:

// alertRank 3 is NOAA's first bleaching alert level (alert-1); "change" is the
// internal state key that renders to the public label "Declining".
if ((bestCover !== null && bestCover < 25) || alertRank >= 3) {
  return "change";
}

That single function is quietly reading from both a live daily feed (thermal stress) and a snapshot that might be 4 years stale (coral cover), and producing one confident looking label. If I had not separated freshness at the schema level first, this function would have no way to distinguish "coral cover crashed last month" from "coral cover was measured once in 2010 and we are still using that number." Because the freshness contract is settled upstream in the schema, this function stays a plain threshold check. Every consumer of the data reads the same explicit field instead of re-deriving staleness on its own, so the rule for how old a number is lives in exactly 1 place.

The Honest Number, in the End

After auditing all 83 sources against this 3-shape model, the honest count came out smaller than I expected, and it forced a distinction I had been blurring. How often a source ingests is a different axis from which freshness shape it carries. 8 of the 83 ingest on a real automated schedule: NOAA thermal stress daily, Global Fishing Watch fishing pressure, IUCN status, the biodiversity feeds from iNaturalist, GBIF, and OBIS, and the AGRRA and MERMAID survey ingests. Ingesting on a schedule is not the same as carrying the Live freshness shape, though. The GBIF, OBIS, and iNaturalist feeds refresh often, yet every record they produce is still Presence, because a fresh pull of occurrence data does not make any single sighting newly true, so it carries no staleness claim at all. Coral cover is a snapshot for most locations, because the science itself does not move faster than a report cycle, though AGRRA now feeds live multi-year coral cover for the Caribbean through its public data explorer. Species sightings were, for a while, a snapshot that was quietly synthetic, meaning the backfill process had generated one plausible sighting per site to avoid empty states, which is its own lesson about how staleness bugs can hide inside data that looks populated. That has since moved to a real weekly iNaturalist and GBIF ingest.

None of that would have surfaced if freshness had stayed a caveat in the copy instead of a field the schema enforces. If you are building anything that blends live feeds with slow survey data, model the freshness shape first, and make it a required part of the type rather than an optional afterthought, so every label reads from it. The display work gets much simpler once the schema is the thing that knows how old each number is.

Scuba Season is a free, nonprofit reef atlas at scubaseason.fun.