Decoupling a Data Pipeline From the Tables It Was Never Supposed to Depend On

I built a pipeline framework on top of a convenience feature, and for about a year it looked like the best decision in the design. The framework generates analytics pipelines from declarative configuration: a user describes the events, stages, and metrics they care about, and the framework produces the Spark jobs, output tables, and dashboards. To make those pipelines incremental, I required every upstream source to be a table with change data feed enabled. The table format's changelog told us exactly which rows had been inserted, updated, or deleted since the last run, so every generated pipeline processed only deltas, for free, with no bookkeeping code of its own.

The fragility hidden in that decision took longer to surface than it should have because it never showed up as a framework bug. It showed up as other people's schema changes.

The Coupling Nobody Signed Up For

When your only supported source type is a physical table with a change feed, every pipeline you generate is bound directly to that table's physical schema. Not to the concept the table represents, and not to some resilient subset of columns. It is bound to the table itself, exactly as its owning team defined it this week.

Upstream teams in a large organization evolve their tables constantly, and they are right to. A column gets renamed to match a naming convention. A type gets widened. A table gets rebuilt as part of a storage migration and picks up a new layout. From the owning team's perspective, these are internal changes to an asset they own. From my framework's perspective, each one was a breaking change delivered without notice to every downstream pipeline reading the table. Because the framework made pipelines cheap to create, there were a lot of them. One upstream rename could break a dozen production pipelines in a single morning, none of which had changed at all.

The standard answer to this problem is an isolation layer: put a view between the physical table and its consumers, and absorb schema changes in the view definition. That is exactly the layer my design made impossible. Change data feeds are a property of physical tables. A view does not emit a changelog. By requiring change-data-feed sources, I had not just failed to build an isolation layer. I had structurally forbidden anyone from adding one. The convenience feature and the fragility were the same feature.

It is worth being precise about why the usual mitigations do not work here. Asking upstream teams to freeze their schemas does not scale past two or three teams, and should not, because it converts your fragility into their paralysis. Materializing private copies of every upstream table restores isolation but doubles storage and adds a freshness lag you now have to monitor. Contract tests detect the breakage earlier, which is worth something, but detection is not absorption. The pipelines still break; you just get a nicer error message. The only structural fix is to let pipelines depend on an interface rather than an implementation, which meant the framework had to stop requiring physical tables at all.

Re-Architecting for Views and Arbitrary SQL

Supporting views and arbitrary SQL queries as first-class upstream sources sounds like adding a source type. It is actually a rewrite of the framework's incremental-processing model, because the thing you lose when you give up the change feed is the framework's entire notion of "what's new."

The replacement is less elegant and more honest: date-field-based processing. Every view or SQL source declares a date or timestamp field that partitions its data by processing period. In the declarative config, the difference between the old world and the new one is small on the surface:

sources:

  # Old style: physical table, incremental via change data feed

  orders_cdf:

    type: table

    name: prod.orders_v2



  # New style: view or SQL, incremental via a declared date field

  orders_view:

    type: view

    name: analytics.orders_stable

    date_field: order_date



  sessions_sql:

    type: sql

    query: >

      SELECT s.session_id, s.user_id, s.started_at, d.channel

      FROM prod.sessions s

      JOIN prod.session_details d ON s.session_id = d.session_id

    date_field: started_at

Underneath, everything changes. Instead of asking the storage layer which rows changed, the framework tracks which date ranges each pipeline has processed and generates predicates that read exactly the unprocessed ranges:

SELECT *
FROM analytics.orders_stable
WHERE order_date >= '{{ watermark }}'
  AND order_date <  '{{ run_end }}'

This is watermark-based incremental processing, one of the oldest patterns in data engineering, and there is a reason it has survived: it depends only on the logical content of the data, never on any physical property of how the data is stored. A view can answer a date-range predicate. A three-table join can answer it. A changelog can only be asked for by the physical table's direct reader.

Moving to date-range semantics forced two other pieces of the re-architecture.

The first was backfill handling. Change-feed processing has no natural story for reprocessing history; the feed only moves forward, and most implementations retain limited history anyway. Date-range processing makes backfill a first-class operation. Reprocessing means re-running a set of date ranges, and correctness means each range must be idempotent, overwriting that period's output atomically rather than appending to it. Building that idempotency into every generated pipeline, instead of leaving it to each pipeline author, was most of the work. It also required adding a true batch mode, because a three-year backfill and a daily increment are the same operation at different scales only if the execution engine treats them that way.

The second piece was a source-contract layer, and this is the part I would build first if I were starting over. Each pipeline declares the columns it needs from each source, with expected types and nullability, and the framework validates every source against its contract before any run:

def validate_contract(source_schema, contract):

    failures = []

    for col in contract.columns:
        actual = source_schema.get(col.name)
        
        if actual is None:
            failures.append(f"missing column: {col.name}")

        elif not compatible(actual.dtype, col.dtype):
            failures.append(
                f"type drift on {col.name}: "
                f"expected {col.dtype}, found {actual.dtype}"
            )

        elif col.required and actual.nullable:
            failures.append(f"nullability drift on {col.name}")

    if failures:
        raise SourceContractError(source=contract.source_name,
                                  failures=failures)

When an upstream schema changes now, one of two things happens, and both are enormous improvements over silent breakage. Either the view definition absorbs the change, and the contract still validates, in which case downstream pipelines never know anything happened; or validation fails before the run starts, producing a named, actionable discrepancy (this column, this type, this source) instead of a runtime failure deep inside a generated Spark job at 3 a.m.

What Actually Changed

The measurable effect was the disappearance of a whole failure category. Schema drift incidents went from a recurring operational tax, each one a cross-team scramble to identify what changed and patch every affected pipeline, to a non-event absorbed at the view layer, usually by the upstream team itself as part of making their change.

The less measurable effect was on how teams model their data. Once views were first-class sources, upstream teams could treat their physical tables as private implementation details and publish stable view interfaces on top, which is how table ownership should work anyway. The framework had previously punished that discipline. Now it rewarded it.

The lesson I keep returning to is about where convenience features belong in a design. Change data feeds are a genuinely good optimization. My mistake was promoting an optimization into an interface. The framework's contract with its sources became "you are a physical table with a changelog" when it should have been "you can answer a date-range query and honor this column contract." Optimizations belong behind interfaces, where you can keep them for the sources that support them and fall back cleanly for the sources that do not. When an optimization becomes the interface, every one of its physical preconditions becomes a coupling point, and you inherit fragility from parts of the system you never intended to depend on. The tables were never supposed to be the dependency. The data was.