From Legacy ETL to Modern ELT: Engineering a Governed dbt and Databricks Architecture

A few years back, I sat through a production incident review where the root cause turned out to be a single stored procedure nobody on the team had touched in four years. The original author had left the company. Nobody knew why a particular join filtered out cancelled orders, only that removing it broke three downstream reports. That afternoon summed up, for me, everything wrong with how a lot of organizations still build their data pipelines.

Legacy ETL was never really about "extract, transform, load" as much as it was about hiding logic somewhere convenient at the time and hoping the convenience lasted. It usually didn't.

Why ETL Started Creaking

Traditional ETL tools like Informatica or SSIS made sense when compute was expensive and storage was the constraint you designed around. You transformed data before it landed in the warehouse because you couldn't afford to store raw, messy versions of everything. That constraint shaped an entire generation of architecture decisions, and honestly, it wasn't a bad approach for its time.

The trouble is that constraint mostly disappeared. Cloud storage is cheap now. Compute is elastic. Yet a lot of teams are still running the same transform-before-load pattern, just on newer infrastructure, which is a bit like buying a Tesla and still driving it like it takes ten minutes to warm up.

Here's roughly what that legacy shape looks like in practice:

Notice where the pain actually concentrates. It's not really the source systems or the warehouse; it's that middle stretch where an ETL server holds all the transformation logic, usually in a proprietary format, usually without meaningful version control, and usually without anyone testing whether a schema change upstream will quietly break a job three hops downstream.

What ELT Changes, and What It Doesn't

Moving to ELT isn't a magic fix, and I'd push back a little on anyone who frames it that way. What it does is separate two things that legacy ETL bundled together: moving data and transforming data. You load raw data first, cheaply, and you transform it later using the compute and tooling that's actually good at transformation, which for most teams now means dbt on top of a lakehouse like Databricks.

This matters more than it sounds like on paper. When transformation logic lives in dbt models rather than a GUI-based ETL tool, it becomes something you can review in a pull request, test automatically, and trace through version history. A junior engineer six months from now can open the repo and actually see why cancelled orders get filtered, because it's sitting there in a documented model, not buried in a stored procedure that requires archaeology to understand.

A Governed Architecture, Not Just a Faster One

The part people skip, though, is governance. Plenty of "modernization" projects swap Informatica for dbt and call it done, then wonder why data quality issues and access sprawl show up six months later, just in a different shape. Speed without governance is still going to bite you; it just takes longer.

Here's the architecture I'd actually recommend, built around the medallion pattern with governance woven through it rather than added on afterward:

A few things worth calling out here, because the diagram alone won't tell you everything:

Bronze is genuinely raw. No transformation, no filtering, just an append-only landing zone. This is your safety net. If a transformation bug corrupts your Silver layer, you can always rebuild from Bronze because nothing was thrown away.

Silver is where dbt earns its keep. Cleaning, deduplication, conforming dimensions, basic business rules. This layer should be boring and predictable, which is exactly what you want from something everything else depends on.

Gold is opinionated on purpose. These are business-facing marts, built for specific consumption patterns, whether that's a finance dashboard or a feature store feeding a fraud model.

Unity Catalog isn't decoration. Row and column level access control, PII tagging, and lineage tracking across the whole Bronze to Gold journey happen here. When an auditor asks who touched customer PII and when, you actually have an answer instead of a guess.

dbt tests gate the pipeline, not the postmortem. Schema tests, not-null checks, referential integrity, and increasingly, formal data contracts between teams run before code merges rather than after something breaks in production.

I'll admit the CI and testing layer is the part teams most often underinvest in, mostly because it's less visible than a shiny new lakehouse. But it's the difference between catching a schema drift issue in a pull request versus catching it because a VP asks why the revenue dashboard looks wrong on a Monday morning.

Where This Still Falls Short

I don't want to oversell this. Governance frameworks like Unity Catalog add real operational overhead, and smaller teams sometimes find the setup cost disproportionate to their scale, at least initially. dbt also assumes a certain SQL-first maturity in your team; if most of your transformations genuinely need Python or Spark-native processing, you're looking at a hybrid setup rather than a pure dbt one. And migration itself is rarely a clean cutover. Most organizations run legacy ETL and modern ELT in parallel for a stretch, which is its own kind of operational headache.

Still, for teams dealing with genuine scale, audit requirements, or just the accumulated technical debt of a decade of ad hoc SQL scripts, this pattern tends to hold up. Not because it's trendy, but because it separates concerns that never should have been tangled together in the first place, and it gives you a paper trail for decisions that used to live only in someone's memory.

That's really the underlying shift here. Legacy ETL asked you to trust the person who wrote the job. Governed ELT asks the system to show its work.