DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Spark Performance Deep Dive on Databricks: Shuffle Tuning, Skew Handling, and Z-Ordering With Delta Lake + Unity Catalog
  • Beyond Partitioning and Z-Order: A Deep Dive into Liquid Clustering for Unity Catalog Managed Tables
  • Why We Chose Iceberg Over Delta After Evaluating Both at Scale
  • Databricks Lakeflow Spark Declarative Pipelines Migration From Non‑Unity Catalog to Unity Catalog

Trending

  • Consumer Lag Is the Wrong Finish Line
  • From Issue to Reviewable Patch: Build a Sandboxed Coding Agent With Deep Agents and Automated Tests
  • The LLM Evaluation Score Improved. Production Quality Didn't.
  • Building a Scope-Constrained AI Agent for Automated CI Failure Repair
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Delta Lake UniForm to Snowflake: Cross-Platform Table Access Without a Second Copy

Delta Lake UniForm to Snowflake: Cross-Platform Table Access Without a Second Copy

Delta Lake UniForm lets Snowflake query a Databricks-owned table as Iceberg, with no export job and no second copy — the four-piece setup, including private connectivity.

By 
Jeevan Krishna Paruchuri user avatar
Jeevan Krishna Paruchuri
·
Oct. 22, 26 · Tutorial
Likes (0)
Comment
Save
Tweet
Share
59 Views

Join the DZone community and get the full member experience.

Join For Free

Making a Databricks-owned Delta table queryable from Snowflake without copying it is one of those things where every individual piece — Delta Lake UniForm, Unity Catalog, private connectivity — is documented somewhere on its own, but nobody writes up how they actually fit together end to end. If you've been asked to give Snowflake read access to a dataset that Databricks owns, without standing up a nightly export job, this is for you.

Why the Two Platforms Don't Share Data Natively

Delta Lake and Apache Iceberg are both open table formats, but they track a table's files, schema, and change history differently. Delta Lake is Databricks' native format. Snowflake has strong native support for Iceberg. A table written as Delta Lake isn't something Snowflake can open directly, and a table written as Iceberg by some other engine isn't something Databricks reads as Delta.

Historically, the fix for this mismatch was a second copy: export the Databricks table, land it somewhere Snowflake could read, and run a job to keep the copy in sync. That's double the storage, a sync job that needs monitoring, and a copy that's stale the moment the source table changes. None of that gets billed as "vendor lock-in" on any invoice, but that's exactly what it is.

What UniForm Changes

Delta Lake UniForm, generally available since Delta Lake 3.2 (Databricks Runtime 14.3+), writes Iceberg metadata alongside the Delta transaction log on every commit. Same Parquet files on disk, one additional metadata layer. Enabling it is a table property:

SQL
 
ALTER TABLE my_table SET TBLPROPERTIES (
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

Once that's set, Databricks keeps writing Delta exactly as before. Snowflake reads the same table as Iceberg, through Unity Catalog's Iceberg REST Catalog endpoint. No export step, no second storage bill.

The Setup, End to End

Getting from "UniForm is on" to "a Snowflake analyst can query the table" is four pieces.

Piece 1: Enable UniForm on the source table. Covered above. One property, per table, Runtime 14.3 or later.

Piece 2: Scope a service principal at the Unity Catalog level, not the storage account. Register a service principal and grant it Unity Catalog privileges — EXTERNAL USE SCHEMA specifically, which isn't part of ALL PRIVILEGES and is easy to grant around by accident. There's no separate storage role to assign. Unity Catalog holds the storage credential itself and vends short-lived, scoped credentials to Snowflake at query time, so the service principal only ever authenticates against the catalog API, never against the storage account directly.

Piece 3: Put the connection on private networking. This is the piece that involves the most moving parts, and it's worth walking through because the Snowflake side does the provisioning, not Databricks. Snowflake account needs to be on Business Critical edition or higher for this, and both accounts need to be in the same Azure region.

Provision an outbound private endpoint from Snowflake toward the Databricks workspace:

SQL
 
SELECT SYSTEM$PROVISION_PRIVATELINK_ENDPOINT(
  '<databricks_resource_id>',
  'databricks_ui_api'
);


That endpoint sits in a Pending state until someone with access to the Azure resource approves it in the Azure Portal — a step that's easy to lose a day to if you don't know to go looking for it. Then register the catalog integration:

SQL
 
CREATE OR REPLACE CATALOG INTEGRATION uc_catalog_int
  CATALOG_SOURCE = ICEBERG_REST
  CATALOG_API_TYPE = PRIVATE
  CATALOG_URI = 'https://<databricks-workspace-url>'
  ...
  ENABLED = TRUE;


The detail that trips people up: use the standard public hostname for the workspace URL, not the PrivateLink endpoint's own hostname. Snowflake routes traffic through the private endpoint automatically once it exists — pointing the catalog integration at the PrivateLink hostname directly just breaks it. A second private endpoint, and an external volume with USE_PRIVATELINK_ENDPOINT = TRUE, does the same thing for the storage account so file reads stay off the public internet too.

Piece 4: Link the catalog — don't hand-create Iceberg tables. The last step is a catalog-linked database, not individual CREATE ICEBERG TABLE statements:

SQL
 
CREATE DATABASE uc_linked_db
  LINKED_CATALOG = (
    CATALOG = 'uc_catalog_int'
  )
  EXTERNAL_VOLUME = 'uc_external_vol';


Once this exists, whatever is in Unity Catalog — including every UniForm-generated Iceberg table — shows up in Snowflake automatically, and stays in sync on its own. New tables and schemas created later in Databricks appear without touching Snowflake again. It's a refresh on an interval under the hood, not a live push, so don't oversell the freshness to whoever's asking for "real-time."

Writes Are Newer, and Direction Still Matters

Everything above is Snowflake reading a Databricks-written table. Writing the other way is newer, and support isn't symmetric yet. Snowflake now supports writing to Databricks-managed Iceberg tables in Unity Catalog — GA on Azure as of earlier this year. Snowflake Horizon's federation is bidirectional by design, using vended short-lived credentials rather than long-lived IAM roles in either direction. 

Unity Catalog's federation is still mostly inbound: strong at letting other engines read and write into it, weaker when Databricks itself needs to reach out to a catalog it doesn't own. If a design depends on writes going both directions, check current support for the specific cloud and catalog combination before committing to it — this part is moving fast.

Where It Still Falls Short

Row- and column-level masking is GA for Spark and expanding to other Iceberg REST-compatible engines, but Snowflake's Scan Plan API — cross-engine, attribute-based access control for federated Iceberg data — was still in private preview as of mid-2026. Don't assume a masking policy enforces the same way on both sides until it's been tested.

Iceberg's own spec keeps moving underneath this too. Iceberg v3 features like deletion vectors and row lineage are in public preview on Databricks now, which is good news for performance, but it means "Iceberg compatible" isn't a fixed target — check that both platforms agree on the table spec version before standardizing on it.

And disaster recovery for federated catalog setups is still gated or preview-stage on the Unity Catalog side as of this writing. Don't lean on this as part of a DR plan without confirming that directly.

One Thing I'd Tell Anyone Doing This

Check the edition and region requirements before you design anything, not after. Business Critical or higher on the Snowflake side, same Azure region on both sides — if either of those isn't true for your accounts, the private connectivity piece doesn't work at all, and there's no workaround short of moving one of the accounts. Confirm that first, then start building.

DELTA (taxonomy) unity Big data

Opinions expressed by DZone contributors are their own.

Related

  • Spark Performance Deep Dive on Databricks: Shuffle Tuning, Skew Handling, and Z-Ordering With Delta Lake + Unity Catalog
  • Beyond Partitioning and Z-Order: A Deep Dive into Liquid Clustering for Unity Catalog Managed Tables
  • Why We Chose Iceberg Over Delta After Evaluating Both at Scale
  • Databricks Lakeflow Spark Declarative Pipelines Migration From Non‑Unity Catalog to Unity Catalog

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook