DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Database Bottlenecks Nobody Talks About: Optimizing SQL Queries Beyond Indexing
  • Seeding Postgres When Your Schema Has Foreign-Key Cycles
  • The Aggregate Reference Problem
  • The Serverless Ceiling: Designing Write-Heavy Backends With Aurora Limitless

Trending

  • The Self-Hosting Math Changed While We Were Still Arguing About Sovereignty
  • Build Production-Ready AI Phone Agents: Infrastructure, Models, and Telephony
  • Prompt Caching for LLMs, Part 1: Exact-Match Caching in Production
  • Going All In On the JVM 25 Virtual Threads

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Data Engineering
  3. Databases
  4. Solving the Stateful Exception: Database Branching and the Unification of CI/CD

Solving the Stateful Exception: Database Branching and the Unification of CI/CD

Learn how Databricks Lakebase uses copy-on-write branching to enable production-data testing and unified database CI/CD pipelines.

By 
Harish Shenoy
·
Nov. 19, 26 · Analysis
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
33 Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

Abstract

Continuous integration and delivery reshaped how stateless application code is built, tested, and released, yet relational databases remained a conspicuous holdout. Because a production database could not be cheaply reproduced per change, teams tested schema migrations against empty or stale schemas and routed real changes through out-of-band review queues, preserving a structural separation between the developer's integration loop and the database administrator's production-change process. 

This article argues that copy-on-write database branching, when it is cheap enough to be ephemeral and modeled as a first-class object, removes the constraint that produced that separation. It uses Databricks Lakebase, a serverless Postgres-compatible database built for the lakehouse, as the object of study: first explaining what Lakebase is and why its architecture differs from conventional managed Postgres, then showing why that architecture is what makes a unified pipeline possible. A runnable reference implementation grounds the discussion, and the article analyzes the consequences for test validity, cost, and security, as well as the role of the reviewer, before turning to the limitations that branching does not resolve.

Introduction: The Stateful Exception

The practices that define modern software delivery were developed, for the most part, against stateless artifacts. A container image, a compiled binary, or a bundle of interpreted source can be produced deterministically from version control, instantiated as many times as a pipeline requires, exercised in isolation, and discarded. The economics of that reproducibility are what make per-branch continuous integration possible: a pull request can spin up its own disposable copy of the entire application and tear it down minutes later at negligible cost.

Relational databases never fit this model. A database is defined by its state as much as its schema, and that state has historically been expensive to reproduce. The consequence is familiar to anyone who has shipped schema changes: teams share a single development database, test migrations against schemas that are empty or that have drifted from production, and submit the changes that matter to a separate review process owned by a database administrator. The developer's fast, automated integration loop and the administrator's deliberate, manual production-change process evolved as two distinct systems, joined by a ticket rather than a pipeline.

This article advances a single claim. That separation traces to one missing capability: the ability to produce a full, data-inclusive copy of a production database cheaply enough to do it per change. Nothing deeper about databases requires it. When that capability exists, the two systems can collapse into one. Developers and administrators can work in the same pipeline, on the same migration artifact, against the same data shape, with the administrator's judgment shifted from reviewing abstract SQL to approving an artifact that has already been verified.

The capability is not hypothetical. The bulk of this article is therefore concrete. Sections 3 and 4 explain what Lakebase is, why its architecture departs from conventional managed Postgres, and why that departure is precisely what supplies the missing capability. Section 5 grounds the argument in a working reference implementation, and the sections that follow examine what the unification changes and what it leaves unresolved.

Background

Three strands of practice frame the problem.

The first is the maturation of schema migration as a versioned, ordered change. Tools such as Flyway and Liquibase established that schema evolution should be expressed as a sequence of migration scripts held in version control, applied in a defined order, and recorded in an audit table so that a database's position in the sequence is always known. This treats schema change as code and is a precondition for any automated database delivery. It does not, by itself, address where those migrations are tested.

The second is the shared-development-database antipattern and its descendants. Because provisioning a realistic database for each developer or branch was impractical, teams converged on a small number of shared environments. These environments drift from production in both schema and data volume, and they serialize work: two engineers cannot safely run conflicting migrations against the same shared schema at once. Testing against an empty schema, the common fallback in automated test suites, is worse still, because it silently excludes the entire class of defects that depend on pre-existing rows.

The third is the long effort to make database copies cheaper. Database engines and managed services have offered progressively faster cloning: snapshot and restore on traditional engines, database snapshots on SQL Server, and fast cloning on Amazon Aurora, which is the closest incumbent analog to what this article examines. These mechanisms reduced the time and cost of a copy but did not change its status. A clone remained a derived, second-class artifact that a human provisioned deliberately, not a first-class object that a pipeline created and destroyed as freely as a Git branch. The licensing and compute cost of keeping such copies alive, particularly on per-core commercial engines, kept per-branch databases out of reach for most teams.

The system studied here sits at the convergence of these three strands: it assumes migrations-as-code, it rejects the shared and empty-schema compromises, and it depends on a cloning primitive cheap enough to be ephemeral. That primitive is a property of the database's architecture, which the next section describes.

What Lakebase Is, and Why Its Architecture Is Different

Lakebase is Databricks' operational database: a serverless, Postgres-compatible transactional (OLTP) engine integrated into the lakehouse. It was introduced in 2025 and is built on the Neon serverless Postgres architecture, which Databricks acquired that same year. To a developer, it appears to be ordinary Postgres. You connect with psql or any Postgres driver, you get Postgres 16 semantics, and common extensions such as pgvector and PostGIS are available. What differs is everything behind the connection string, and those differences are what matter for CI/CD.

Storage is separated from compute, and durable state lives in object storage. A conventional Postgres server couples a database's data files to the lifetime of a machine: the data lives on a disk attached to a running instance. Lakebase, following the Neon design, decouples the two. Durable state — the data pages and the write-ahead log (WAL) history — is kept in cloud object storage rather than on an instance's local disk. A caching layer (the pageserver in Neon terminology) reconstructs and serves pages to compute on demand, and a durable WAL quorum (the safekeepers) provides write durability. Short-lived Postgres compute nodes attach to this durable layer when a query arrives. The database's data no longer depends on any particular machine staying alive.

Compute is serverless and scales to zero. Because compute is detachable, it can be suspended entirely when a database is idle and resumed on demand, and it can be right-sized automatically within configured bounds measured in compute units. An idle database on Lakebase can therefore cost nothing for compute, which is categorically different from a provisioned Postgres instance that bills for its VM even when no one is connected. The trade-off is a cold-start effect: a resumed compute node must re-warm its buffers, so the first queries after a long idle period can be slower until the working set is cached.

A branch is a first-class object, created by copy-on-write. This is the property that the rest of the article turns on. Because a durable state is a set of pages and WAL in object storage, Lakebase can create a new branch by having it reference the parent's existing pages rather than copying them. Only pages that the branch subsequently changes become new objects. A branch is therefore created in seconds, is data-inclusive at the parent's full scale from the instant it exists, and consumes incremental storage only as its data diverges from the parent. Branches are named objects in the service's own hierarchy, each with its own lifecycle operations and its own compute endpoint, rather than snapshots that a human copies and tracks out of band. A Lakebase project is a tree of such branches, with a root branch holding the authoritative production state.

Identity and access run through the Databricks control plane. Lakebase does not require a static database password. A client obtains a short-lived OAuth token tied to the authenticated Databricks principal (a user or service principal) and presents it as the Postgres password. Tokens expire in roughly an hour. There is no long-lived secret to store, rotate, or leak.

It is governed and connected as part of the lakehouse. Lakebase operates under Unity Catalog governance and can synchronize data with Delta tables, so operational and analytical data share a single governance model and can flow between the transactional and analytical sides of the platform. This integration is not central to the CI/CD argument, but it is why the database exists within the lakehouse rather than alongside it.

How this differs from the incumbents is now specific. A self-managed Postgres server, or a classic managed offering such as Amazon RDS, ties data to an always-on instance, bills for that instance continuously, uses managed passwords for authentication, and offers no notion of a cheap, disposable branch. Amazon Aurora is closer: it separates storage from compute and offers fast cloning, the closest existing analog to Lakebase branching. But an Aurora clone is still a derived object that bills for its compute while it runs, and no workflow treats it as an ephemeral, scale-to-zero branch keyed to a pull request. The distinctive combination in Lakebase is that a copy is simultaneously instant, data-inclusive, free when idle, and modeled as a first-class branch. That combination is the missing capability from Section 1.

Why Does That Architecture Fit the CI/CD Model

The fit follows directly from the four architectural properties above, each of which answers an objection that kept databases out of per-change CI.

The objection "we cannot afford a copy of production per branch" is answered by copy-on-write: the copy is free until it diverges, so a branch per pull request is affordable. The objection "test environments cost too much to leave lying around" is answered by scale-to-zero: an idle branch's compute suspends, so the standing cost of a test environment approaches zero. The objection "a clone is a heavyweight thing a human has to manage" is addressed by making branches first-class API objects: a pipeline can create and delete them programmatically, with no human in the loop. And the objection "a credential that works in CI is a credential we have to store and protect" is answered by short-lived OAuth tokens: there is no static secret to put in the pipeline at all.

A database branch, in other words, acquires exactly the properties that stateless CI always assumed of its artifacts: cheap to create, cheap to keep, disposable, and scriptable. Once a branch has those properties, the per-change isolation that defines modern CI becomes available for the database, and the architecture designed for serverless OLTP and lakehouse integration turns out, almost incidentally, to be the one that lets database change management rejoin the pipeline.

A short illustration makes the branch primitive concrete. Creating a branch is a single declarative call that names its source:

Shell
 
databricks postgres create-branch "$PROJECT" "$BRANCH_ID" \
  --json '{"spec": {"source_branch": "'"$PROJECT"'/branches/production"}}'


The branch comes up in seconds, carrying production data by reference on its own endpoint that scales to zero when idle. Everything in the pipeline below is built on operations as ordinary as this one.

A Reference Architecture

A runnable reference implementation implements the unified model as a single pipeline with two paths, selected by the build type. Its layout is as follows.

Plain Text
 
Lakebase project: projects/orders-api
├── production        (protected)         ← administrator owns promotion
│   └── primary endpoint
└── ci-pr-42          (ephemeral)         ← one per pull request
    └── primary endpoint (scale-to-zero)     copy-on-write from production


The orchestrator is a Jenkins multibranch pipeline, but its stages are deliberately thin wrappers over a set of shell scripts, so that the identical commands run on an engineer's laptop and on the CI agent. This symmetry is itself a design claim: the pipeline is a scheduler for commands a developer can run and reason about directly, with no privileged behavior hidden inside it.

On a pull request, the pipeline runs the developer path:

Groovy
 
stage('Create DB branch') { when { changeRequest() } steps { sh './scripts/create_branch.sh' } }
stage('Migrate')          { when { changeRequest() } steps { sh './scripts/migrate.sh' } }
stage('Test')             { when { changeRequest() } steps { sh './scripts/test.sh' } }


A branch named for the pull request is created from production, the migrations under review are applied to it, and the test suite runs against it. A post block tears the branch down whether the build passed or failed, reclaiming its compute and copy-on-write storage.

On the main branch, the same pipeline runs the promotion path:

Groovy
 
stage('DBA approval') {
  when { branch 'main' }
  steps { input message: "Promote migrations to production?", submitter: 'dba-team' }
}
stage('Promote to production') {
  when { branch 'main' }
  steps { sh './scripts/promote.sh' }
}


The migration applied to production is the same artifact that was exercised on the ephemeral branch; promotion is literally the migration step run with the target set to the protected production branch. Two systems that were historically distinct, the developer's integration loop and the administrator's production change, are now two conditional paths through one definition, sharing one artifact and one set of scripts.

The architecture does not require abandoning established migration tooling. The reference implementation supports both plain ordered SQL and a Liquibase changelog behind a single switch, so a team that wants versioned ordering, an applied-change audit trail, and one-command rollback can adopt it without changing the pipeline's shape. Branching is orthogonal to the choice of migration tool: it changes where migrations are tested, not how they are expressed.

Analysis: Four Consequences

The architecture is simple. Its interest lies in what the branch primitive changes once it is in place. Four consequences are worth drawing out.

1. Test Validity Against Production-Shaped Data

The most consequential change is epistemic: it alters what a passing test means. The reference implementation centers on a deliberately ordinary migration that adds a non-nullable column with a default to a table that already holds rows:

SQL
 
ALTER TABLE orders
    ADD COLUMN IF NOT EXISTS fulfillment_status VARCHAR(20) NOT NULL DEFAULT 'pending';


Against an empty schema, this migration always succeeds, because there are no rows to violate the constraint, and so does any test that merely checks the column now exists. Against production-shaped data, the migration must also backfill every pre-existing row, and the failure mode of forgetting to do so becomes observable. The test suite asserts exactly that property:

Python
 
def test_existing_rows_were_backfilled(conn):
    cur.execute("SELECT count(*) FROM orders")
    total = cur.fetchone()[0]
    cur.execute("SELECT count(*) FROM orders WHERE fulfillment_status IS NULL")
    nulls = cur.fetchone()[0]
    assert total > 0, "branch was not data-inclusive"
    assert nulls == 0, f"{nulls} pre-existing orders left with NULL"


The first assertion is as important as the second: it fails if the branch arrived empty, treating "the test environment did not resemble production" as itself a test failure rather than a silent precondition. This inverts the usual relationship between test environments and confidence. In the shared or empty-schema model, a green suite was compatible with a migration that would break on contact with real data, and practitioners learned not to fully trust it. When every pull request is tested against a data-inclusive clone of production, a green suite exercises the conditions that actually break migrations, and the signal regains its meaning.

2. The Ephemeral Cost Model

The branch model also changes the economics of test environments from a standing cost to a marginal one. A traditional per-branch database, where it existed at all, was a provisioned resource that consumed license and compute, whether or not it was in use, which is why teams rationed such environments to a shared few. A copy-on-write branch on scale-to-zero compute inverts the default: an idle branch costs nothing effectively, storage is charged only for divergence from the source, and the branch exists only for the life of the pull request. The cost of isolation, therefore, scales with the activity of change rather than with the number of environments kept alive, which is the same cost structure that made per-branch stateless CI economically ordinary. The pipeline reinforces this by tearing branches down unconditionally in its post block, so the ephemeral default is enforced rather than left to developer discipline.

3. Secretless Authentication

Unifying the laptop and the pipeline raises an obvious objection: a credential that works in both must be stored somewhere. The reference implementation sidesteps this by never minting a static database password. Each connection generates a short-lived OAuth token at the moment it is needed:

Shell
 
token=$(databricks postgres generate-database-credential "$endpoint_path" | jq -r '.token')


The token is valid for roughly an hour and is tied to the authenticated principal, which, on a CI agent, is a service principal. There is no long-lived database password to store as a pipeline credential, to leak into build logs, or to rotate on a schedule. The same primitive that makes branches disposable, a managed control plane that mediates access to compute, also makes it natural to replace static secrets with ephemeral, identity-bound credentials. Security improves as a byproduct of the architecture rather than as a bolt-on control.

4. The Relocation of the Reviewer

The fourth consequence is organizational. In the separated model, a database administrator reviews proposed SQL in the abstract, reasoning about what a migration will do to production without having watched it do so. In the unified model, the administrator's approval gate sits on the main branch, and fires only after the exact migration artifact has already been applied to a production-shaped clone and verified by the test suite. The reviewer is no longer the first line of defense against a migration that fails to apply; automated verification is. Human judgment is repositioned to address the questions that automation cannot settle: timing, blast radius, coordination with dependent systems, and whether the change should go to production now. This is a more defensible division of labor than the one it replaces, in which the administrator was asked to catch mechanical defects by inspection, a task at which human review is known to be unreliable.

Limitations and Threats to Validity

Branching removes one constraint; it does not, by itself, make database delivery safe, and an honest account must state what remains.

Forward-only promotion. The reference implementation promotes migrations forward and relies on migration tooling for ordering and, in the Liquibase path, for rollback. Branching makes testing safe but does not make a destructive or non-idempotent production migration reversible. A migration that drops a column or rewrites data still demands the same care it always did; the branch verifies that it applies, not that it is wise.

Data divergence over a branch's life. A branch is a point-in-time snapshot. A long-lived branch tests against data that is increasingly stale relative to production, and a migration validated on a week-old branch may meet rows in production that did not exist when the branch was created. The ephemeral, per-pull-request discipline mitigates this, but the guarantee is only as strong as the branch is fresh.

Conclusion

The absence of continuous integration for databases was a rational response to a real constraint, not a failure of discipline. There was no cheap way to reproduce a production database for every change, so the discipline stateless code enjoyed could not reach stateful data, and the developer's loop and the administrator's process remained separate systems joined by a queue.

Lakebase's architecture supplies the missing capability almost as a side effect of being serverless Postgres for the lakehouse. By keeping a durable state in object storage, detaching compute so it can scale to zero, and exposing copy-on-write branches as first-class objects, it makes a data-inclusive copy of production cheap, disposable, and scriptable. Those are exactly the properties that stateless CI always assumed of its artifacts. With them, a single pipeline can test every change against production-shaped data, carry one migration artifact from the pull request to production, and reposition human review from catching mechanical defects to exercising the judgment that automation cannot. The branch does not make database change effortless or risk-free, and the limitations above are real. But it removes the specific constraint that made databases the stateful exception to modern delivery, and in doing so, it lets database change management rejoin the practices the rest of software delivery has taken for granted for a decade.

Database Relational database

Published at DZone with permission of Harish Shenoy. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Database Bottlenecks Nobody Talks About: Optimizing SQL Queries Beyond Indexing
  • Seeding Postgres When Your Schema Has Foreign-Key Cycles
  • The Aggregate Reference Problem
  • The Serverless Ceiling: Designing Write-Heavy Backends With Aurora Limitless

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook