Data Engineering

A transaction reads one row and then remains open. Later, a deployment adds a column using ALTER TABLE ... ALGORITHM=INSTANT. The migration waits. A new request issues an ordinary SELECT against the same table, and that request waits too. The original query has already finished. The schema change requires no table rebuild. The new query accesses a different row. Yet all three sessions participate in the same blocking sequence. The explanation sits above InnoDB’s row-lock layer. The open transaction retains a metadata lock, the schema change requests an incompatible lock, and that pending request can make later queries queue behind it. Instant DDL reduces the work required to change a table. It does not eliminate the coordination required to begin or complete that change. Reproduce the Queue With Three Connections Use a disposable database and three independent MySQL connections. A tiny table keeps data volume out of the explanation: MySQL CREATE DATABASE mdl_demo; CREATE TABLE mdl_demo.orders ( id BIGINT PRIMARY KEY, status VARCHAR(20) NOT NULL ) ENGINE=InnoDB; INSERT INTO mdl_demo.orders VALUES (1, 'NEW'), (2, 'NEW'), (3, 'DONE'); In Session 1, start a transaction and read one row: MySQL START TRANSACTION; SELECT * FROM mdl_demo.orders WHERE id = 1; -- Leave the transaction open. The query returns immediately. Do not commit yet. In Session 2, request the schema change: MySQL ALTER TABLE mdl_demo.orders ADD COLUMN source VARCHAR(32) NULL, ALGORITHM=INSTANT; After the ALTER begins waiting for a metadata lock, issue another read in Session 3: MySQL SELECT * FROM mdl_demo.orders WHERE id = 2; This later read can also wait, despite accessing a different row. Committing Session 1 allows the migration to proceed, followed by the queued read. MySQL’s verified Bug #106480 documents this three-connection behavior for MySQL 8.0. Test the sequence against your deployed version rather than assuming every DDL operation has identical locking details. Separate Execution Cost From Lock Admission For supported operations, ALGORITHM=INSTANT changes metadata without rebuilding the table. Whether a particular alteration qualifies depends on the operation, table characteristics, and server version. MySQL publishes these restrictions in its online DDL operation matrix. That optimization removes potentially expensive data movement. It does not promise a maximum elapsed time. The migration’s duration includes time spent acquiring required locks. A metadata change that executes quickly after admission can still wait behind an existing transaction for minutes. This distinction should influence deployment planning. Estimating migration risk from table size alone misses the failure demonstrated here. A three-row table can experience the same admission problem as a much larger one. The operational question is therefore twofold: how expensive is the change once admitted, and how long may it wait while affecting other requests? The Blocker Can Be Idle For tables used within a transaction, MySQL retains transactional metadata locks until the transaction ends. Finishing a SELECT does not necessarily release them. See MySQL’s metadata-lock lifetime documentation. Consequently, searching only for slow queries can miss the session responsible. Application code commonly stretches transactions beyond their SQL work. An endpoint reads data, calls a remote service, performs additional processing, and commits afterward. The database query takes milliseconds, but the transaction remains open throughout the external work. A connection can therefore appear idle while still preventing a schema change. Measure transaction age separately from query duration. Review transaction boundaries around network calls, user interaction, and background work. Moving unrelated operations outside a transaction can reduce exposure without changing the migration itself. Preflight checks help identify existing blockers, but they cannot guarantee that another transaction will not start immediately afterward. Why Later Readers Wait Without the pending ALTER, the two reads in the example could coexist. The DDL request changes the scheduling situation. MySQL normally prioritizes metadata write-lock requests over read-lock requests, with documented exceptions. That policy prevents a continuous stream of readers from indefinitely starving a writer. The priority rules appear in the metadata-lock acquisition documentation. On a busy table, this creates a widening queue. One open transaction delays the migration, and the waiting migration delays newly arriving work. Those requests can occupy application connections until the pool fills. Subsequent requests may then wait for a connection before reaching MySQL at all. This is why a migration incident can look like general database saturation even when CPU and storage activity remain modest. The database is coordinating access, not necessarily executing expensive queries. Inspect Ownership and Pending Requests Start with a diagnostic query that connects waiting sessions to potential blockers: MySQL SELECT object_schema, object_name, waiting_pid, waiting_lock_type, blocking_pid, blocking_lock_type FROM sys.schema_table_lock_waits WHERE object_schema = 'mdl_demo' AND object_name = 'orders'; The sys view documentation describes these session and lock relationships. Use the result alongside transaction age and application ownership. Identify which connection belongs to the migration runner and which belongs to the application. For lower-level investigation, inspect performance_schema.metadata_locks and correlate its owner thread identifiers with performance_schema.threads. Distinguish granted locks from pending requests rather than treating every displayed lock as an active blocker. Do not automatically terminate the oldest connection. It may hold legitimate work whose rollback is expensive. When the migration is deferrable, removing its waiting request may be the less disruptive intervention. Set the Correct Timeout The familiar innodb_lock_wait_timeout controls InnoDB row-lock waits. It does not bound the metadata-lock wait responsible for this incident. Configure the migration connection instead: MySQL SET SESSION lock_wait_timeout = 3; ALTER TABLE mdl_demo.orders ADD COLUMN source VARCHAR(32) NULL, ALGORITHM=INSTANT; In MySQL 8.4, lock_wait_timeout governs metadata-lock acquisition attempts and defaults to 31,536,000 seconds. The timeout applies separately to each acquisition attempt, so it is not a universal three-second statement deadline. See the server variable reference. Three seconds is illustrative. Choose a budget based on acceptable application disruption. Apply the setting on the actual connection executing the migration. Setting it on a separate administrative connection has no effect on the migration session. A dedicated connection also avoids leaving unexpected timeout settings in an application pool. Verify That Timeout Releases the Queue Extend the reproduction using a fresh table definition. Keep Session 1’s transaction open. Start the migration with a short metadata-lock timeout, then issue Session 3’s read while the migration is waiting. When the ALTER times out and its pending request is removed, the later read should proceed in this isolated scenario even though Session 1 remains open. That observation identifies the queue boundary. The two reads were compatible; the waiting schema change prevented the later one from advancing. A failed migration must remain visible to deployment automation. Do not mark it complete merely because application traffic recovered. Retry with backoff and a bounded attempt count rather than repeatedly recreating the same queue. After a client disconnect or ambiguous execution result, inspect the actual schema before retrying. Make Admission Part of the Deployment Contract A production migration policy should specify the permitted algorithm, metadata-lock wait budget, retry behavior, and conditions for postponement. Monitor pending DDL alongside transaction age, connection-pool usage, and request latency. Alerting only on migration duration can detect the problem after application capacity has already been consumed. Test both successful admission and blocked admission before deployment. The latter proves that the migration yields when production conditions are unfavorable. Instant DDL remains valuable because it avoids unnecessary table reconstruction. Its safety depends on the surrounding lifecycle: transactions must finish, lock requests must be admitted, and waiting migrations must have a bounded opportunity to interfere with traffic.

A security review started after connection strings were visible through a Kubernetes management tool. The first request sounded simple: remove Secrets from the cluster and move the values to Azure Key Vault. That request focused on which system stored the values. The access review found a wider problem. Kubernetes stored a deployment service account's legacy token in a kubernetes.io/service-account-token Secret. The token had no expiry in its normal use, and RoleBindings allowed the same identity to operate in three application namespaces. The Role used resources: ["*"] across several API groups and allowed get, list, watch, create, update, patch and delete. The Role included the core API group, so the token could read every Secret in each of those namespaces. The review found no evidence of token misuse. It found an avoidable access path with a long lifetime and a larger scope than the pipeline needed. Moving application passwords to Key Vault would improve their source of truth and rotation. It would not revoke this token, narrow its RBAC permissions or remove the cross-namespace bindings. Those controls needed their own fix. Why the Kubernetes Secrets Existed The application images carried JSON configuration files for local development. Database, cache and message-broker endpoints pointed to localhost. Those defaults were useful on a developer machine and unusable inside the cluster. The shared Helm chart supplied the deployment values at runtime. It rendered an Opaque Secret from a secrets map, then loaded the Secret into the application container through envFrom. The reduced chart looked like this: YAML {{- if .Values.secrets } apiVersion: v1 kind: Secret metadata: name: {{ include "app.fullname" . }-secrets type: Opaque stringData: {{- range $key, $value := .Values.secrets } {{ $key }: {{ $value | quote } {{- end } {{- end } The Deployment then loaded every entry as an environment variable: YAML envFrom: - secretRef: name: {{ include "app.fullname" . }-secrets The application configuration placed environment variables after the file-based settings, so the runtime values won. That ordering made the Kubernetes Secret part of the configuration path. Removing it without replacing that path would leave the containers pointing at local services. Figure 1. The existing deployment used a Kubernetes Secret as the runtime override for local file defaults. The next question was, "Which components can retrieve the value, and how long does each form of access last?" Key Vault Does Not Always Remove the Kubernetes Secret The Azure Key Vault provider for the Secrets Store CSI Driver supports more than one consumption model. A pod can read mounted files directly from a CSI volume. This path can avoid creating a native Kubernetes Secret. The application still needs a configuration method that reads those files and responds safely when their contents rotate. The driver can also copy mounted values into a native Kubernetes Secret through the secretObjects field of a SecretProviderClass. Microsoft documents this as secret synchronisation. It is useful when an application already consumes secretKeyRef or envFrom, which was the case here. It also means that moving the source value to Key Vault does not remove the Secret object from Kubernetes. YAML apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: app-vault spec: provider: azure secretObjects: - secretName: app-runtime-secrets type: Opaque data: - objectName: database-password key: Database__Password The choice affects rotation too. Microsoft notes that an existing process does not receive updates when a container consumes a Kubernetes Secret through an environment variable. The team must restart the pod before the process receives the new value. Key Vault remains a sound source for secrets. The important point is narrower: the storage migration and the Kubernetes access review solve different problems. The Service Account Crossed Namespace Boundaries The next part of the review followed the identities that could read Secrets. Kubernetes allows a RoleBinding in one namespace to name a ServiceAccount from another namespace. That feature is useful for controlled cross-namespace work. It also means that checking only the ServiceAccount's home namespace can understate its reach. The review found a layout equivalent to this generalized example: YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pipeline-deployer namespace: environment-b rules: - apiGroups: ["", "apps", "batch", "autoscaling", "networking.k8s.io", "policy"] resources: ["*"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pipeline-deployer-from-a namespace: environment-b subjects: - kind: ServiceAccount name: pipeline-deployer namespace: environment-a roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: pipeline-deployer Two bindings of this form extended the identity from its home namespace into two more namespaces. The same bearer token could therefore authenticate once and use the permissions that each Role granted. Figure 2. Two cross-namespace bindings extended one long-lived credential into three namespaces. The identity was not a cluster administrator. The Role listed its API groups, and it did not include rbac.authorization.k8s.io. The token could not grant itself new RoleBindings through this Role. That limit mattered, but it did not make the existing scope acceptable. The core API group still covered Secrets, and the wildcard resource rule covered much more than the deployment needed. The following authorization check avoids printing any secret value: Shell identity="system:serviceaccount:environment-a:pipeline-deployer" for namespace in environment-a environment-b environment-c do printf '%s: ' "$namespace" kubectl auth can-i list secrets \ --namespace "$namespace" \ --as "$identity" done For the reviewed identity, the effective result was yes in all three namespaces. The Token Type Made the Scope Harder to Contain The service account had one manually created Secret of type kubernetes.io/service-account-token. Kubernetes describes this as a legacy mechanism that provides a long-lived credential. The current documentation recommends the TokenRequest API for short-lived tokens instead. The difference changes the failure window. A deployment job can request a token that expires shortly after the job finishes. If someone copies a static token into a CI/CD service connection, that credential can remain useful until somebody deletes or rotates it. An old export, agent workspace or copied configuration can preserve the same access during that period. The cluster version already supported projected service account tokens. Another environment-specific service account had no legacy token Secret, which confirmed that the static pattern was not present in every namespace. The First Fix Reduces the Token's Reach The quickest containment step is to remove the two cross-namespace RoleBindings. That immediately reduces the token's potential reach from three namespaces to one without changing the applications. Each environment should then have its own deployment identity, Role and RoleBinding. A pipeline for one namespace should not borrow the service account from another. This keeps the credential, audit trail and revocation boundary aligned with the environment it deploys. The desired shape is simple: Plain Text pipeline A uses a short-lived token for its Role in namespace A pipeline B uses a short-lived token for its Role in namespace B pipeline C uses a short-lived token for its Role in namespace C These deployments require no cross-namespace subject. Figure 3. The target design aligns one service account and its short-lived token with one namespace. A Short-Lived Token Replaces the Static Credential Kubernetes can issue a bounded token with the TokenRequest API. kubectl create token is a convenient way to exercise that API: Shell kubectl create token pipeline-deployer \ --namespace environment-a \ --duration 15m The command covers only one part of a CI/CD design. The runner still needs a trusted way to request the token. The pipeline should receive the credential at run time, the requester should limit its audience, and the token should expire soon after the deployment window. The service connection must stop storing the old bearer token before that change has value. After the new path succeeds, delete the legacy token Secret and confirm that the API rejects the old credential. Explicit RBAC Replaces Wildcard Access resources: ["*"] made future chart changes part of the role automatically. Kubernetes warns that wildcard rules can grant access to resource types that future API extensions add. A deployment role should name the resource types and verbs the release actually uses. The rendered manifests provide the initial resource list: Shell helm template app ./chart \ --namespace environment-a \ --values values.environment-a.yaml \ > rendered.yaml Inventory every apiVersion and kind in that output. Then compare the result with deployment calls in API audit logs or a dry run. The resulting Role may need access to Deployments, Services, ConfigMaps, Secrets, Jobs, Ingresses and autoscaling resources. It should not receive every resource merely because the chart might use one later. If a separate controller owns runtime Secrets, the deployment identity can lose Secret write access as well. If Helm still renders those Secrets, the role needs the exact verbs required by that release process. The policy must match the delivery design instead of hiding a required permission until the next deployment fails. The Review Found Other Paths to the Same Values The cluster token was the widest durable access path that the review found. Several pipeline variables also stored credentials as ordinary variables. Azure DevOps treats secret variables as protected resources. It encrypts their values at rest and masks exact values in logs. A task can still send a value elsewhere, and masking does not always hide partial strings. Marking the variables as secret still removes routine plaintext visibility from the library interface and logs. The remediation plan combined four related controls: Mark credential variables as secret and restrict which pipelines can use each variable group.Remove routine human permission to read Kubernetes Secrets through cluster tools.Rotate the database, cache and broker credentials that broader access paths exposed.Move non-sensitive settings out of the Helm secrets map and into a ConfigMap. Moving non-sensitive settings to a ConfigMap makes the review surface honest. Feature flags, log levels and endpoint names do not need to sit beside credentials merely because the original release populated only one values map. Validation Uses the Identity's Point of View The validation must test permissions from the identity's point of view. For each deployment identity, build an explicit matrix: Plain Text own namespace other namespaces read required objects yes no apply release objects yes no read application Secrets only if required no modify RBAC no no Then verify the old paths: the deleted legacy token fails authenticationeach identity returns no for Secret access outside its namespacethe pipeline can still deploy and roll backvariable-group values appear masked in the interface and logscluster users without an operational need cannot reveal Secret valuesrotated application credentials work and the previous values fail This sequence keeps the deployment usable while shrinking the access path in measurable steps. Secret Location and Secret Access Need Separate Reviews The visible Kubernetes Secret started the investigation, but deleting that object first would have left the long-lived deployment credential untouched. Using CSI secret synchronization could even have recreated the same type of object from a better source. The useful review order was identity, scope, lifetime, source and rotation. It exposed one token that crossed three namespace boundaries, wildcard permissions that exceeded the release surface, and pipeline values that lacked basic secrecy controls. Key Vault can improve storage and rotation. Short-lived credentials and narrow RBAC control who can reach the Kubernetes API. Environment-specific identities contain the effect of a leaked token. These controls work together, and none of them can substitute for the others. References Kubernetes documentation: SecretsKubernetes documentation: Service AccountsKubernetes documentation: Managing Service AccountsKubernetes documentation: RBAC good practicesMicrosoft Learn: Azure Key Vault provider configuration options for AKSMicrosoft Learn: Access Key Vault through the CSI Driver on AKSMicrosoft Learn: Secrets in Azure PipelinesMicrosoft Learn: Manage Azure Pipelines variable groups

The enterprise paradigm for artificial intelligence has fractured. While massive cloud-hosted large language models continue to dominate consumer applications, highly regulated sectors like defense, aerospace, and high-tier finance are actively blocking external API calls to public AI services due to strict data sovereignty mandates. The solution is migrating toward small language models (SLMs) localized at the tactical edge. Deploying an SLM into an air-gapped, "sovereign cloud in a box" environment requires absolute cryptographic certainty over the supply chain. A compromised base image running deep inside a disconnected network invalidates the entire security perimeter. The recent release of Docker Hardened Images (DHI) and the expansion of Google Distributed Cloud (GDC) air-gapped appliances give backend engineers the exact toolchain required to host models like Google's Gemma with zero-trust enforcement. This technical guide details an end-to-end framework for curating a zero-CVE container, packing a fine-tuned Gemma SLM, and pushing it to an isolated GDC cluster. The Target Infrastructure: GDC Air-Gapped Google Distributed Cloud air-gapped is an integrated hardware and software platform designed to operate physically disconnected from the internet. It allows enterprises to deploy container-based workloads and managed services like Vertex AI in completely isolated environments. The architecture supports rigorous compliance frameworks, including NIST, ISO 27001, and SOC II. Notably, GDC offers packaged generative AI solutions enabled by Gemma, providing local, conversational search capabilities on-premises. However, when deploying custom fine-tuned checkpoints or building the Java microservices that handle backend inference orchestration- a core focus of my daily work as an IEEE Senior Member and backend developer — we cannot rely on standard public container registries. The container must be hardened from the base layer up. Base Image Curation With Docker DHI To build our localized SLM container, we start with Docker Hardened Images. DHI provides minimal, secure, production-ready container images maintained by Docker. Rather than blindly pulling a generic Alpine or Debian base image and running a vulnerability scanner after the fact, DHI guarantees a hardened foundation. Using the docker dhi CLI, available in Docker Desktop 4.65 and later, we can inspect and mirror these hardened artifacts directly to our enterprise registries. Shell # Verify local DHI CLI installation docker dhi --help # Mirror a hardened Python or Java base image to your local org docker dhi mirror start --org enterprise-registry \ dhi/python,enterprise-registry/dhi-python By switching our Dockerfile FROM instruction to enterprise-registry/dhi-python, we structurally eliminate the attack surface before installing our inference engine dependencies. Generating the SLM Artifact and SBOM As I detailed in my Springer book chapter Introduction to Trustworthy AI, accountability in machine learning relies heavily on technical robustness and traceability. We cannot deploy a black-box container into an air-gapped network. We must generate a cryptographic Software Bill of Materials (SBOM) that catalogs the exact SLM weight hashes, the inference engine binaries, and the OS packages. The DHI CLI handles this transparently. Once you package your Gemma model weights into the hardened container, you can retrieve the exact attestations and SPDX SBOMs attached to the base images. Shell # Retrieve the SLSA provenance and SPDX SBOM for audit docker dhi attestation get dhi/python:3.11 sha256:<digest> -o provenance.json This JSON payload is ingested into the local CI/CD pipeline. The pipeline validates that no unauthorized dependencies were injected during the model packaging phase. If the cryptographic signature fails, the build halts immediately. Deploying to the Sovereign Edge With a cryptographically signed, DHI-backed container housing our Gemma SLM, we transition to the deployment phase on GDC air-gapped. Because GDC air-gapped operates a single cluster encompassing bare metal nodes without internet access, deployment relies on a physical handoff. The hardened container image, the validated SBOM, and the deployment manifests are loaded onto a portable storage device. Once transferred to the secure facility, the infrastructure operator pushes the image to the local, disconnected GDC container registry. GKE on GDC orchestrates the container workload. We apply Kubernetes network policies to isolate the SLM namespace, ensuring that the model can only be queried by authorized internal Java backend services. By enforcing kernel isolation at the container level and physical isolation at the hardware level, we construct a true defense-in-depth architecture. The Gemma model processes highly classified proprietary data while satisfying the strictest data sovereignty requirements. The transition from cloud-first to sovereign-edge AI marks a critical maturation in software engineering. Trustworthy intelligence demands predictable execution boundaries. By merging the zero-trust foundations of Docker Hardened Images with the physical isolation of Google Distributed Cloud, engineering teams can deliver state-of-the-art reasoning capabilities directly to the tactical edge without compromising a single byte of sensitive data.

If you've worked with a relational database for any length of time, you know the drill: state lives in a row, and every UPDATE overwrites whatever was there before. It's simple, it's fast to reason about, and it throws away history by design. Update a bank balance from $1,000 to $900, and the fact that a withdrawal happened and why is gone unless some separate audit log happened to catch it. Event sourcing flips that model on its head. Instead of storing current state, you store the sequence of events that produced it, and you derive state on demand by replaying the log. History isn't a side effect you bolt on afterward; it is the write path. This isn't a niche idea anymore. Netflix runs event-sourced pipelines that process over a billion viewer events a day for personalization and health monitoring. Uber pushes millions of events per minute ride requests, location updates, pricing signals through Kafka-backed streams with exactly-once semantics for fraud detection. LinkedIn built Kafka in the first place to serve as an append-only log for activity data. Slack pipes billions of daily messages through event-driven pipelines that carry full metadata so services don't have to make synchronous calls to each other. What's harder to find is a clear-eyed look at what actually breaks in production and how teams fix it. That's what this article is about: the data model, the CQRS split, the consistency trade-offs, and the part that tends to get glossed over: how to keep read models correct when the same event might get delivered twice, three services need to stay in sync, and any one of them might be down when the message arrives. Events as First-Class State In a CRUD system, a withdrawal looks like this: SQL -- Traditional CRUD (destructive) UPDATE accounts SET balance = 900 WHERE id = 123; In an event-sourced system, nothing gets overwritten. You append a new fact: SQL -- Event sourcing (additive) INSERT INTO events (stream_id, event_type, payload, version) VALUES ('account-123', 'MoneyWithdrawn', '{"amount": 100}', 42); Current state is reconstructed by folding over every event in a stream, in version order. It's a pure function — same events in, same state out, and it maps directly onto a left-fold/reduce if you've ever written one: Python def reconstruct_balance(account_id): events = db.query( 'SELECT * FROM events WHERE stream_id = %s ORDER BY version', [f'account-{account_id}'] ) state = {'balance': 0} for event in events: if event.type == 'AccountOpened': state['balance'] = 0 elif event.type == 'MoneyDeposited': state['balance'] += event.payload['amount'] elif event.type == 'MoneyWithdrawn': state['balance'] -= event.payload['amount'] return state['balance'] The Event Store Schema The event store itself is a boring relational table with one architectural rule bolted on: nothing is ever updated or deleted. SQL CREATE TABLE events ( id BIGSERIAL PRIMARY KEY, stream_id UUID NOT NULL, event_type VARCHAR NOT NULL, payload JSONB NOT NULL, version INT NOT NULL, occurred_at TIMESTAMPTZ NOT NULL DEFAULT now(), UNIQUE (stream_id, version) ); That UNIQUE (stream_id, version) constraint is doing more work than it looks like. It's your optimistic concurrency control: if two processes both try to append version 5 to the same stream, the second one gets a unique-constraint violation instead of silently clobbering the first. It's compare-and-swap at the stream level, for free. And to be clear — this table can live in Postgres, MySQL, Oracle, whatever you're already running. Event sourcing is a write strategy, not a database product. Don't let anyone tell you that you need a specialized event store to do this. Snapshots Replaying thousands of events every time you want to read an entity's state gets expensive fast. The standard fix is snapshotting: periodically materialize current state and store it next to the log. On read, load the latest snapshot and replay only what's happened since. SQL SELECT * FROM snapshots WHERE stream_id = 'account-123' ORDER BY version DESC LIMIT 1; -- Then replay only events after snapshot.version SELECT * FROM events WHERE stream_id = 'account-123' AND version > :snapshot_version ORDER BY version; The CQRS Boundary In practice, event sourcing almost always shows up paired with CQRS (Command Query Responsibility Segregation) — splitting the write path (commands that produce events) from the read path (queries against a denormalized read model). This isn't just architectural tidiness. The event log is optimized for append throughput and sequential replay; it's a bad fit for ad-hoc queries, joins, sorting, or full-text search. ComponentRoleTypical technologyCommand handlerValidates commands, appends eventsApplication service layerEvent storeAppend-only source of truthPostgreSQL, EventStoreDB, KafkaProjectionReads events, builds the read modelBackground worker, Kafka consumerRead modelDenormalized query targetPostgreSQL, Redis, ElasticsearchQuery APIServes reads to clientsREST / GraphQL Consistency Models and the Lag Problem Projection Lag Is Real, Design For It Because the projection runs asynchronously, there's a window between an event landing in the store and the read model reflecting it. A user withdraws money and immediately checks their balance and might see the stale number. This is the central consistency problem in any CQRS/event-sourcing setup, and it needs an explicit answer, not a hope that it won't matter. In healthy systems on Kafka or a comparable bus, projection lag typically sits around 50–300ms. Under load spikes, partitions, or consumer restarts, it can stretch to seconds or minutes. Your design needs to be correct under the bad case, not just the median. Pick Your Battles: Selective Consistency The real design question isn't "How do I eliminate eventual consistency?" It's "Where do I actually need strong consistency, and where can I let the lag ride?" Those two buckets look pretty different: Data typeConsistency requiredWhyAccount balance post-withdrawalStrongFinancial correctnessPayment confirmationStrongRegulatory, user expectationRecommendation feedEventualA few seconds of lag is invisibleNotification badge countEventualApproximate is fineProduct inventory on a listing pageEventualMinor lag is tolerableSearch index for a new entityEventualUsers already expect crawl delay Read-Your-Own-Writes For the strong-consistency bucket, the most common mechanism is read-your-own-writes (RYOW). The command API hands back the version number it just appended; the client sends that version back on its next read; and the query service refuses to answer until the read model has caught up. Markdown # 1. Command returns the appended event version POST /accounts/123/withdraw {amount: 100} → HTTP 200 {status: 'ok', event_version: 42} # 2. Client sends version as constraint on next read GET /accounts/123/balance Headers: X-Min-Version: 42 # 3. Query service checks projection position SELECT last_processed_version FROM projection_checkpoints WHERE name = 'balance_projection'; -- If < 42, wait up to 500ms then retry or return 202 Accepted When You Just Bypass the Read Model For a small set of high-stakes flows such as payment confirmation screens, dispute resolution, regulatory audits, plenty of systems skip the read model entirely and rebuild state straight from the event log. It's always consistent, at the cost of latency. Snapshots (see above) keep that latency bounded even for entities with long histories. Three Ways to Keep Projections in Sync The projection needs to process each event exactly once and update the read model atomically with its own checkpoint. In practice, that comes down to three strategies, each suited to a different infrastructure setup. 1. Same-Database Transaction The simplest option: put the read model and the checkpoint table in the same relational database, and wrap the update and the checkpoint advance in one transaction. SQL BEGIN; UPDATE account_balances SET balance = 900 WHERE account_id = 123; UPDATE projection_checkpoints SET last_event_id = 42 WHERE name = 'balance_projection'; COMMIT; Either both writes land or neither does, which kills the two classic failure modes: processing an event without advancing the checkpoint (you'll reprocess it on restart) and advancing the checkpoint without writing the data (permanent, silent data loss). Great when your whole read model fits in one database. 2. Kafka + Idempotent Consumers Once Kafka is your event bus, consumer offsets live inside Kafka and can't be committed in the same transaction as your read-model write. Kafka only promises at-least-once delivery, so on a consumer restart, some events may show up twice. The fix is to make the consumer idempotent, in which case it's safe to run twice on the same event. SQL INSERT INTO account_balances (account_id, balance, last_event_id) VALUES (%s, %s, %s) ON CONFLICT (account_id) DO UPDATE SET balance = EXCLUDED.balance, last_event_id = EXCLUDED.last_event_id WHERE account_balances.last_event_id < EXCLUDED.last_event_id; That WHERE clause is the whole trick. If event version 42 gets replayed after it's already been applied, 42 < 42 is false and the update is a no-op. ScenarioDB state beforeIncoming eventConditionResultNormal processinglast_event_id = 41v = 4241 < 42 → TRUEUpdate appliedCrash replay (duplicate)last_event_id = 42v = 4242 < 42 → FALSESkipped safelyOut-of-order deliverylast_event_id = 42v = 4142 < 41 → FALSEStale event ignored The INSERT branch only ever fires once, when the entity's stream first shows up. Every event after that takes the ON CONFLICT path, and the version guard is what makes it idempotent. 3. The Transactional Outbox Pattern Things get harder when a projection needs to fan out to several different stores at once, say a relational read model, an Elasticsearch index, a Redis cache, and a webhook receiver. No single database transaction can span all of those. This is where the transactional outbox pattern earns its keep. The projection writes its read-model update and an outbox row in one local transaction. A separate relay process picks up pending outbox rows and delivers them to each downstream target, retrying until each one confirms. Because the outbox write is atomic with the read-model write, the relay always knows exactly what still needs to go out. SQL -- Atomic: both rows commit or neither does BEGIN; UPDATE account_balances SET balance = 900 WHERE account_id = 123; INSERT INTO outbox (id, payload, es_sent_at, redis_sent_at, webhook_sent_at) VALUES (gen_random_uuid(), '{"account_id": 123, "balance": 900, "event_version": 42}', NULL, NULL, NULL); COMMIT; Making the Outbox Relay Fault-Tolerant The relay delivers to each target independently and tracks confirmation per target. The scenario worth designing for: the relay writes successfully to target A, then crashes before marking the row delivered, so on restart, target A gets the same write again. Every target needs to handle that gracefully. Elasticsearch: Let the Document ID Do the Work Elasticsearch indexing is idempotent as long as you supply a stable document ID yourself. Index the same ID twice and you just overwrite the first doc with identical content, having no duplicates. Python # Non-idempotent: ES generates a random ID es.index(index='accounts', body={'balance': 900}) # → Two calls produce two documents # Idempotent: stable ID derived from the outbox row es.index( index='accounts', id=outbox_row.id, # same on every retry body={'balance': 900, 'event_version': 42} ) Redis: Absolute Values, Not Deltas SET in Redis is naturally idempotent — setting a key to the same value twice is the same as setting it once. The catch: your relay has to write absolute values, never deltas. INCRBY and LPUSH are not idempotent and have no business being in a relay path. Python # Non-idempotent: applying a delta twice corrupts state redis.incrby('account:123:balance', -100) # Idempotent: writing the derived absolute value redis.set('account:123:balance', 900) Webhooks: Idempotency Keys External webhook receivers are the trickiest case, because you don't control their implementation. The standard answer is the one Stripe documents publicly: an Idempotency-Key header. The relay sends the outbox row's UUID as that key on every attempt; the receiver keeps a table of keys it's already seen and just replays the original response for a repeat. Getting this right on the receiver side means handling three edge cases: Concurrent duplicate requests: Two requests with the same key arrive before either has been saved. Fix it with an atomic INSERT under a unique constraint; the second insert fails, and that thread waits on the first.Key reuse with a different body: Same key, different payload. Reject with a 422 after comparing a hash of the stored body to the incoming one.Key expiry: Keys can't live forever. Stripe uses a 24-hour TTL, which means your relay's retry window (and dead-lettering) has to finish well inside that boundary. The key record and the business-logic effect have to commit in the same transaction. If the business logic runs but the key never gets saved, a retry re-executes the logic and that's not a missed optimization, that's data corruption. Python def handle_request(request): key = request.headers['Idempotency-Key'] req_hash = sha256(request.body) with db.transaction(): row = db.execute(''' INSERT INTO idempotency_keys (key, request_hash, created_at, expires_at) VALUES (%s, %s, now(), now() + interval '24 hours') ON CONFLICT (key) DO UPDATE SET key = EXCLUDED.key RETURNING key, request_hash, response_status, response_body ''', [key, req_hash]).fetchone() if row.request_hash != req_hash: return 422, 'Key reused with different request body' if row.response_status is not None: return row.response_status, row.response_body result = run_business_logic(request) db.execute(''' UPDATE idempotency_keys SET response_status = %s, response_body = %s WHERE key = %s ''', [result.status, result.body, key]) return result.status, result.body Dead-Letter Escalation Transient failures — timeouts, a target being briefly down — get exponential backoff with per-target retry counters on the outbox row. Permanent failures — a schema mismatch, a decommissioned endpoint — shouldn't be allowed to block the relay forever. After a configurable number of attempts (10 is a common default), move the row to a dead-letter table for someone to look at. Who's Actually Running This in Production CompanyEvent volumePatternOutcomeNetflix>1B events/dayEvent sourcing + CQRS on KafkaPersonalization and monitoring for 260M+ subscribersUberMillions/minuteKafka + Flink, exactly-once semanticsFraud detection and dynamic pricing at global scaleLinkedInOrigin of Kafka (2010)Append-only activity stream logUnified data pipeline, replacing point-to-point integrationsSlackBillions of messages/dayKafka event propagation with full metadataDecoupled delivery and indexing under heavy loadStripeDocumented idempotency-key APITransactional idempotency at the webhook layerFormally specified 24-hour idempotency window with body-hash validation Takeaways A few things worth internalizing if you're building (or maintaining) one of these systems: Eventual consistency is a contract, not a bug. RYOW turns the consistency requirement into an explicit versioned header instead of an assumption. Idempotency keys turn "don't double-process this" into a first-class API contract. The outbox pattern turns delivery into a durable, queryable table instead of a fire-and-forget call. Once it's explicit, you can test it and monitor it — that's the whole point.Idempotency is the load-bearing primitive. It's standing in for distributed transactions, which are either unavailable, too slow, or too fragile for high-throughput event-driven systems. At-least-once delivery plus idempotent receivers gets you effectively-once behavior without any distributed coordination.The pattern underneath is always the same: a unique key derived from something stable (an event version, an outbox row's ID), stored atomically next to the effect it protects, checked before that effect is applied, and expired on a TTL that bounds how long you need to keep retrying.None of this locks you into specific technology. Event store in Postgres, EventStoreDB, or Kafka's compacted log — doesn't matter. Read model in MySQL, Redis, or Elasticsearch — doesn't matter. The outbox just needs to live in the same ACID store as the read model it's paired with. That's a low bar, and it's why teams can adopt this incrementally next to whatever they're already running. The hard part of event sourcing was never the event log where appending immutable rows is the easy 20%. The real engineering work is keeping derived state correct across multiple heterogeneous targets when any one of them can fail at any moment. Idempotency, applied consistently, is what makes that tractable without reaching for distributed transactions.

GitHub's contribution calendar gives you an already iconic overview of your activity over the year. Each day has a square, and its color changes with the contribution level. And of course, those days and contributions can also be arranged into buildings, with a week becoming a small part of a city and busier weeks adding height to the skyline. I had some spare minutes and built Quarkus Contribution City around this idea, using Java and Quarkus to generate an SVG from GitHub's contribution data. The visual inspiration came from Pink Pixel's skyline, and you can see a version with my real data on my GitHub profile. A scheduled GitHub Action updates the image at night. For a Java developer, this is a small project you can follow from the API request through to the image. You can use the released action on your own profile, then change the drawing code locally once you have a feel for how the city is put together. For this article, I created a special DZone example that adds a small piece of character art above the buildings. Add the City to Your Profile Your GitHub profile README lives in a public repository with the same name as your account. For my account, that repository is myfear/myfear. Open your own profile repository, or create it with a README if you haven't used this GitHub feature before. Copy the project's complete publishing workflow into .github/workflows/contribution-city.yml on the default branch. This example checks out your profile repository and the action's v1 release line. It then generates the image and publishes it to an output branch. The workflow includes the repository write permission needed for that last step. The generation step uses the repository owner's username, so the same configuration can be copied between personal profile repositories. You can add the layout options to that step: - name: Generate contribution city id: city uses: ./.city-action with: github-token: ${{ secrets.CONTRIBUTION_TOKEN || secrets.GITHUB_TOKEN } username: ${{ github.repository_owner } weeks: '53' height: '14' theme: github-dark The local path refers to the action checkout made earlier in the example workflow. The built-in GITHUB_TOKEN is the starting point for reading your visible contribution data. An optional CONTRIBUTION_TOKEN secret can supply a personal token when additional access is needed. The calendar returned by GitHub depends on the token's permissions and your account's contribution visibility settings. After committing the workflow, open Actions in your profile repository, select publish-contribution-city, and choose Run workflow. A successful run creates contribution-city.svg on the output branch. Add this image reference to your profile README, replacing both occurrences of USERNAME:  The example also has a daily schedule for 03:17 UTC. Subsequent runs create a commit when the generated image changes. For this setup, the workflow and its run history can be found in your profile repository, while the Java implementation is maintained in the action's repository. Give Each Week a Building The city is arranged in time order, with the most recent week on the right. Every building is two character cells wide, so all weeks occupy the same horizontal space. Weekly contribution totals determine the height, and daily contribution levels provide the window colors. Month labels beneath the street help you find your place in the year. Height needs a little extra thought. A very busy week can have many times the contributions of a quiet one. With a linear scale, those quieter weeks would be compressed close to the street. The city uses a logarithmic scale, which gives them more room while preserving the order of weekly totals. For each week, the calculation can be expressed as: int height = weeklyTotal == 0 ? 1 : 1 + (int) Math.round( Math.log1p(weeklyTotal) / Math.log1p(maximumWeeklyTotal) * (maxHeight - 1)); With a maximum of 20 contributions and a height limit of 14, a week with three contributions gets seven window rows. The busiest week gets 14, and a week with zero contributions keeps one row. Quiet periods therefore retain their position in the calendar. An antenna marks the busiest displayed week; when totals are tied, the most recent week receives it. The two columns of windows are filled by cycling through the days of the week. This repeats the daily pattern in taller buildings, so window count is a visual texture. The weekly total controls height, while the window colors carry the daily contribution levels. Together, they give you a view of the pattern across your calendar. You can change weeks to 12 for a shorter view. The tallest building is then calculated from those recent weeks, which lets the selected period use the available height. Keep this in mind when comparing images: the same week can appear taller in a shorter view. The displayed contribution total still covers the full calendar returned by GitHub. Follow the Data Through Java The released project uses Quarkus with the Quarkus GitHub Action extension. That extension connects an @Action method to workflow inputs, outputs, and the job summary. Inside the method, the drawing process comes down to three steps: ContributionCalendar calendar = contributionClient.fetch( configuration.username(), inputs.getGitHubToken().orElseThrow(), context.getGithubGraphQLUrl()); Scene scene = cityBuilder.build(calendar, configuration.options()); String svg = svgRenderer.render(scene, configuration.options().theme()); The client reads the contribution calendar with one GraphQL request and converts the response into Java records for the calendar, weeks, and days. CityBuilder uses these records to place roofs, windows, and the street in a grid. SvgRenderer turns that grid into the final SVG. Changes to the skyline can therefore be made in plain Java classes, with the GitHub connection kept in the client. The SVG contains the shapes and text needed to display the city. Adjacent characters with the same color are grouped during rendering, and their horizontal positions are set explicitly to keep the columns aligned. Identical data and options produce identical file contents, which allows the publisher to compare images before creating a commit. The action sets up Java and JBang, builds the selected source revision, and runs the packaged application. Change the Layout Locally To try a change, install JDK 21 and JBang, with Git and Bash available. The repository includes a Maven wrapper. JBang is also used by the integration tests to run the packaged action through the same launcher used in GitHub Actions. git clone --branch v1.0.0 https://github.com/myfear/quarkus-contribution-city.git cd quarkus-contribution-city ./mvnw verify The tests use local test data and a local GraphQL server, so you can run them without GitHub credentials. They also generate SVG previews under target/previews. Open github-dark.svg, recent-12.svg, and empty.svg in a browser to compare a full calendar, a shorter period, and a calendar with zero contributions. Start with the height calculation in CityBuilder, then run the same command again and compare the previews. The saved data stays the same between runs, which makes the effect of your change easier to see. The project is available under the Apache License 2.0, and the v1.0.0 release gives you a fixed starting point. Give the City a DZone Landmark Version 1.0.0 exposes weeks, height, and the two built-in themes through workflow inputs. For a custom shape, you extend the Java code in your fork. The Scene record gives you a grid of cells to change before the SVG is rendered, and that is where we'll add the DZone lettering. Each cell has a character, a role such as roof or window, and the color information used by the renderer. A character mask is enough to describe a new shape: # marks a filled block, and spaces leave the background visible. By arranging those blocks, you can draw letters, an arch, or a stepped tower. For this example, the lettering is placed above the weekly buildings. It has its own fixed shape, while the buildings below continue to show the contribution calendar. The image uses a snapshot of my real account data, with 534 contributions returned by GitHub on October 5, 2026, when I wrote this article. A DZone-inspired palette and decorative lettering added to my contribution city. The weekly building heights and daily window patterns come from the same calendar used by the regular view. Add this class to the existing city package in your fork, at src/main/java/com/themainthread/contributioncity/city/DZoneLandmark.java: package com.themainthread.contributioncity.city; import java.util.ArrayList; import java.util.Collections; import java.util.List; import com.themainthread.contributioncity.render.Theme; public final class DZoneLandmark { public static final Theme THEME = new Theme("dzone-example", "#161b22", "#242c36", "#303947", "#f58a42", "#9ba9ba", "#edf2f7", List.of("#536070", "#86431f", "#b85b25", "#e57d36", "#ffb365")); private static final List<String> MASK = List.of( "#### ##### ### # # #####", "# # # # # ## # # ", "# # # # # ## # # ", "# # # # # # # # #### ", "# # # # # # ## # ", "# # # # # # ## # ", "#### ##### ### # # #####"); private DZoneLandmark() { } public static Scene addTo(Scene city) { int landmarkWidth = MASK.getFirst().length() * 2; int width = Math.max(city.width(), landmarkWidth); int left = (width - landmarkWidth) / 2; List<List<Cell>> rows = new ArrayList<>(); Cell block = new Cell('█', CellKind.ROOF, 0, -1); for (String line : MASK) { List<Cell> row = new ArrayList<>(Collections.nCopies(width, Cell.SKY)); for (int column = 0; column < line.length(); column++) { if (line.charAt(column) == '#') { row.set(left + column * 2, block); row.set(left + column * 2 + 1, block); } } rows.add(row); } rows.add(Collections.nCopies(width, Cell.SKY)); for (List<Cell> sourceRow : city.rows()) { List<Cell> row = new ArrayList<>(sourceRow); row.addAll(Collections.nCopies(width - city.width(), Cell.SKY)); rows.add(row); } return new Scene(city.username(), city.totalContributions(), rows); } } The theme defines the background, two building shades, roof, street, and label colors, followed by the five window colors. Those window colors follow GitHub's contribution levels from zero through four. Here, the roof color is also used for the lettering, so the decoration has a consistent orange color across all seven rows. Inside addTo, each marked position is expanded to two cells because the renderer's cells are taller than they are wide. The lettering is centered, a blank row is added beneath it, and the original city rows are copied below. Short calendars receive extra background space on the right to make room for the lettering. The dates, counts, and order of the weekly buildings are preserved. To use the decoration in your fork, import DZoneLandmark from the city package in ContributionCityAction. Replace the existing scene-building and rendering lines with: Scene scene = cityBuilder.build(calendar, configuration.options()); Scene decorated = DZoneLandmark.addTo(scene); String svg = svgRenderer.render(decorated, DZoneLandmark.THEME); This selects the example's palette in Java. The published @v1 action continues to accept github-dark and mono; your fork contains the additional drawing code. For a local preview, add the following after the existing Files.writeString call in PreviewTest.write, with the same import for DZoneLandmark: Scene decorated = DZoneLandmark.addTo(builder.build(calendar, options)); Files.writeString(directory.resolve(name.replace(".svg", "-dzone.svg")), renderer.render(decorated, DZoneLandmark.THEME)); Run ./mvnw verify again and open target/previews/github-dark-dzone.svg. The build also writes the shorter and empty versions with the -dzone.svg suffix, so you can check how the decoration fits several calendars. When you're ready to use your fork on GitHub, update the second checkout in the publishing workflow to your fork's repository and a commit containing your changes. The generation step then runs the customized action from that checkout. Try It With Your Own Contributions Try Quarkus Contribution City with your own GitHub account and add the generated SVG to your profile. Start with the released action, then fork the project and change the character mask to give your city a landmark of its own. You could draw a stepped tower from your hometown or add your team's initials above the street, using the DZone version as a starting point. Share your version in a GitHub issue, or open a pull request if you'd like to contribute a new shape.

The most consequential security problem in enterprise AI may not be a model giving a wrong answer. It may be what happens when that model is wired to an identity with enough authority to turn the answer into an action. An agent that can query a database, call an infrastructure API, update a ticket, and trigger a workflow is no longer an interface to a model. It is an executing software principal. Most enterprises are still securing it like a web app with a clever text box, and I think that mismatch is where the next wave of incidents will come from. The Question Nobody Logs Picture an incident response agent. A pager fires, the agent pulls telemetry, queries a database, opens a ticket, calls a cloud API, and proposes a fix. No human clicked anything between steps two and five. Now ask the forensic question. Which principal performed each of those operations? Not who typed the prompt, but whose credential hit the database, under what scope, and who approved it. Microsoft's own security guidance names this problem plainly. Its least privilege pattern for agents lists "identity ambiguity" as a core challenge: agents running under shared secrets, service principals, or mixed "on behalf of user" modes, which makes it unclear which principal is responsible for an action. It also notes that logs often capture the chat response rather than the underlying tool actions and downstream authorization decisions. Source: Least privilege for AI agents with Microsoft Entra Agent ID (page updated July 15, 2026). That is the gap. Authentication tells you the agent is who it claims to be. It says nothing about whether each action should be allowed. From Application to Principal In a traditional application, the logic is fixed at design time. A user acts, the app calls an API, the API touches data. You can enumerate the paths. An agent chooses its path at runtime. It selects tools, chains calls, reads results that influence the next call, and may cross system boundaries. Microsoft describes this directly: agents plan and execute multistep workflows, often chain actions across multiple systems, and can take delegated actions with minimal or no human intervention. Once software can decide and act, it needs what every other actor in the enterprise already has: its own identity, an accountable owner, scoped permissions, a lifecycle and a way to be shut off. Microsoft's Entra Agent ID work reflects the same direction, modeling agents as a dedicated identity type rather than reusing ordinary application identities. See the overview of agent identities in Microsoft Entra. My position is blunt. The LLM must never be the security boundary. A model can be persuaded, confused, or hijacked by text inside a retrieved document. The boundary has to sit around the identity, the tools, and the downstream systems, where it can be enforced deterministically. Authentication Is Not Authorization An agent can authenticate perfectly and still be dangerously overprivileged. Take an incident agent that legitimately needs to read logs, metrics, tickets, and configuration, and to write one thing: an incident ticket. It should not be able to delete infrastructure, modify IAM roles, create administrators, export customer records, or change security policy. Microsoft's pattern turns this into concrete guidance. Separate the read role used for evidence gathering from the write role used for ticket creation. Allowlist only the "create or update ticket" actions. Block delete and admin actions. Require human approval for bulk updates. Aggregate Privilege: The Number Nobody Computes Permission creep is old news. Agents give it a new shape. Microsoft calls out "permission creep and excessive aggregate privileges": teams grant broad roles to unblock a pilot and never narrow them, or they layer many narrow roles without task separation, and the combined roles create overly broad effective access. Without aggregate analysis, the agent's true capability across systems is easy to underestimate. Each grant looks defensible in isolation. CRM read, mailbox access, ticketing, a cloud role, a deployment pipeline. Together they describe a high-privilege operator that never took a day of onboarding. So the review question changes from "is this permission safe?" to "what can this agent do end to end when every role and tool is combined?" Microsoft's first implementation step is exactly that: inventory deployed agents and tool integrations and identify effective aggregate permissions. The Tool Surface Every tool an agent can call is attack surface, and the dangerous failures are boring ones. Microsoft's pattern warns that if an agent can invoke any available tool, a single prompt injection or workflow bug can trigger high-impact operations such as export, delete, or privilege changes, or chain actions across services. The industry has started to formalize this. The OWASP GenAI Security Project announced its Top 10 for Agentic Applications on December 9, 2025. The list covers ten risk classes, including goal hijack, tool misuse, and identity and privilege abuse. Its ASI03 entry describes agents operating with borrowed, over-scoped, or long-lived credentials, so a compromised agent inherits far more access than its task needs. A readable summary is in Teleport's breakdown of the OWASP list. The design rule that follows: a tool is not trusted because an agent asked for it. Microsoft's guidance is to deny unreviewed tools, plugins, integrations, and cross-tenant or guest paths by default, and to verify authorization at the downstream system on each call rather than relying on the orchestrator alone. It describes revalidating identity, roles, and scope at every hop, from orchestrator to tool to downstream service, so one weak integration cannot bypass the rest. The trade-off is real, and Microsoft states it: this depends on downstream systems correctly re-checking authorization, and gaps there reintroduce the risk. Delegation Is a Security Boundary When an agent acts as itself, the audit trail is clean. When it acts on behalf of Alice, a harder question appears: does it inherit everything Alice can do? It should not. Alice may hold broad rights that a single task never needs. The fix is explicit "on behalf of" context, narrowed scopes, and a named owner for the agent. Microsoft recommends logging the "on behalf of" user wherever it applies. OWASP's ASI03 framing is the mirror image: delegated credentials, cached sessions, and implicit identity let an attacker use an agent to do what the original owner never intended. Authorization Cannot End at Login Traditional IAM answers "may this identity reach this resource?" Agentic systems need a second question: "should this specific action happen right now?" Microsoft's recommended pattern is a stable, lifecycle-managed agent identity combined with time-limited privileges through just-in-time entitlements, meaning temporary role activation, short-lived tokens, or approvals, so higher privilege exists only for the duration of a specific workflow. For remediation agents, it suggests JIT elevation or approval gates, scoping to specific resource groups, denying subscription-wide rights, keeping rollback procedures, and requiring step-up controls for destructive actions. Here is the decision shape I would want in front of every tool call: Python def authorize(agent_id, on_behalf_of, tool, action, resource, context): scope = effective_scope(agent_id, on_behalf_of) # computed, never assumed if tool not in allowlist(agent_id): return deny("tool not allowlisted") if action not in scope.allowed_actions(resource): return deny("action outside effective scope") if is_high_impact(action): # delete, export, privilege change return require_approval_or_jit(agent_id, action, resource) return allow(ttl_seconds=300, correlation_id=context.correlation_id) The point is not the code. The decision happens outside the model, per action, with a short lifetime on the grant. Log the Action Chain, Not the Conversation A transcript is not an audit trail. Microsoft's guidance lists what a useful record contains: agent identity, role, effective scope, action, resource, correlation ID, and the "on behalf of" user where applicable. Its success metrics include the percentage of agent actions emitting those required fields and the percentage of workflows traceable end to end across orchestrator, tools, and downstream systems. If your SIEM can show that an agent "answered a question" but cannot show which API it called under which scope, you have chat history, not telemetry. Revocation Is a Metric Most agent programs I read about spend pages on capability and a sentence on shutdown. Microsoft treats the stop path as a first-class concern: test revocation by disabling the agent, rotating credentials, invalidating tokens, and removing stale permissions. Its listed success factors include mean time to revoke or disable an agent identity, including token invalidation, and the percentage of environments validated with a tested kill switch and recovery procedure. I would put mean time to revoke on the same dashboard as mean time to detect. If disabling an agent takes a ticket, an approval, and two days, then that agent's autonomy was never bounded. It was only unobserved. Why Legacy Estates Make This Harder Agents cannot safely automate what an enterprise cannot understand or control. Legacy environments tend to hide undocumented business logic, old APIs, fragmented data, inconsistent identity models, and weak observability. Point an agent at that, and you hand a fast, tireless actor a map nobody has drawn. That is why identity, integration, observability and governance now sit next to model selection in modernization work. It is also why vendors in the modernization space are repositioning around agents. On March 3, 2026, Sonata Software said it had been recognized as a Microsoft Frontier Partner, describing an approach that combines AI agents with human engineering across cloud and AI platforms, AI business solutions and security. On October 7, 2026, Sonata Software announced that it had earned the 2026–2027 Microsoft AI Business Solutions Inner Circle award, marking the company’s sixth Inner Circle recognition. Microsoft's Niels Jensen, AI Business Solutions ERP Lead in Enterprise Partner Solutions, was quoted in the release saying, "Inner Circle partners represent our top-selling partners worldwide." (Sonata's press release) Both items are vendor announcements, so read them as signals about where the services market is heading, not as proof of any specific control. What matters for this argument is the shift itself: modernization is being reframed as preparation for governed autonomy, and governance starts with the identity layer. What to Do on Monday Inventory every agent and tool integration, including guest and cross-tenant paths, and compute each agent's aggregate effective permissions.Give each production agent a unique identity and a named owner. Retire shared credentials.Replace broad roles with task-scoped ones. Split read from write.Allowlist tools and actions. Gate delete, export, and privilege changes behind approval or JIT elevation.Enforce authorization at the downstream system, not only in the orchestrator.Emit identity, role, scope, action, resource, and correlation ID for every action, and ship it to your SIEM.Run a revocation drill and record the time it takes. The Argument in One Line The goal is not to stop enterprises from giving agents autonomy. It is to make that autonomy bounded, attributable, and revocable. If an agent can act like an employee, an application, and an automation system at once, the question worth asking is why we still secure it as if it were only an application.

By Igboanugo David Ugochukwu CORE

Need a break from reading and want to check out some video content? This episode of The Neuron AI Explained is worth watching! Hosts Corey and Grant take a closer look at GPT-6 Astra and Google's Gemini 3.8 Flash, sharing their thoughts on how these models perform beyond the benchmark scores. Sourced from The Neuron AI Explained Youtube Page If you enjoy the video and want to keep learning, check out The Neuron Academy. You'll find hands-on lessons, structured learning paths, and practical examples to help you build on what you already know about AI. Whether you're looking to improve your prompting skills or get more experience with AI agents and workflows, there's plenty to explore. Well, that's enough from me! Happy watching.

I was working on a content extraction pipeline for a RAG implementation that ingests regulatory filings and legal documents to create a knowledge repository. The pipeline processes ingested files, runs them through an LLM, and produces structured summaries and extracts key insights. The output looked well-formatted and presented in a professional tone. Everything looked great to pass the unit test or integration test. Then I decided to measure the output. The median F1 score across our initial test set was 0.611, which is well below our 0.80 target. The summaries read correctly. They just weren't complete. The system was confidently presenting a polished version of half the truth. This article is about what I learned building the evaluation approach that uncovered this, which metrics actually matter, what they miss, and the single biggest insight that changed how I think about RAG pipelines. The Failure Mode Here's what our initial numbers looked like: Precision: 0.844 (median) — high. What the model said was almost always correct.Recall: 0.481 (median) — low. It was missing more than half the important content.F1: 0.611 — the harmonic mean told the real story. A system with high Precision but low Recall is the most dangerous failure mode in regulated domains. It never said anything wrong; it just doesn't say enough. Stakeholders see a well-written summary and assume completeness. They don't know what's missing because they didn't read the 160-page original. In my experience, this pattern is common in summarization pipelines that haven't been properly evaluated. If your team hasn't measured Recall independently from Precision, you're probably in the same boat. Picking the Right Metrics I tried every metric in the toolkit. Here's what actually proved useful: What Worked Metric Threshold I Used What It Actually Told Us Token-level F1 > 0.80 target Best single signal for "is this summary complete and accurate?" ROUGE-1 > 0.40 = acceptable Caught cases where the summary used completely different vocabulary BERTScore > 0.85 = acceptable Caught the semantic drift when the summary paraphrased into something different What Disappointed Us Metric Why It Underperformed ROUGE-2 (bigram) Punished valid paraphrases heavily. A perfect summary written in different words scored 0.06. METEOR Theoretically handles synonyms. In practice, scores were so low across the board (0.01–0.04) that they provided no discriminative signal. TF-IDF Cosine Similarity Fooled by topical similarity. Two documents about the same regulation scored high even when the facts differed. Automated Metrics Pitfall None of them catch: Logical errors: Correct facts in the wrong relationshipAttribution/Citation mistakes: "Article 7 states..." when it was actually Article 12Temporal confusion: Mixing up current provisions with repealed onesImportant omissions that aren't in the reference either What I recommend now: Use F1 + ROUGE-1 + BERTScore as automated gates. Then human evaluation on 20–30 samples before any production decision. There's no shortcut past that second step. How Document Length Impacts Scores This was the finding that made everything else make sense. We ran evaluation across documents ranging from 10 to 162 pages. Here's what the data showed: Document Pages Precision Recall F1 Short regulation 10 0.85 0.62 0.71 Medium directive 28 0.83 0.53 0.65 Long regulation 58 0.80 0.42 0.55 Very long regulation 162 0.89 0.04 0.07 That last row surprised me the most. Precision of 0.89 with Recall of 0.04. The system produced an accurate summary of pages 1–5 and completely ignored the other 157 pages. When I ran correlation analysis: Recall vs. document length: r = -0.71 (p < 0.05)F1 vs. document length: r = -0.78 (p < 0.05)Precision vs. document length: r = +0.05 (not significant) The pattern was consistent: as documents got longer, the system's coverage collapsed while its confidence stayed high. It became a confident partial summarizer, which is never wrong, just incomplete. Why This Happens My initial design was to process the first ~5 pages of each document. This was done to reduce token count and keep the cost in control. Usually, you will find the abstract, purpose, and high-level detail of a regulation in most of the regulatory content for any domain. So I thought this would work well, and all stakeholders agreed on this approach. This worked well during demos and the initial testing phase. We tested for a 10-page document; the first 5 pages here mean 50% coverage, which is very good. However, for a 162-page document, that's just 3%. The system literally could not see 97% of the content. If you only test on short documents (which most teams do during development), this never shows up. The evaluation looks great, and we implement it. Then a compliance team relies on a summary that silently drops the critical clause on page 47. How I Fixed It What I Tried Hierarchical summarization: We split the full document into segments or chunks, summarize each one, then produce a final summary from the segment summaries. This guarantees every section gets seen. This improved Recall from 0.48 to ~0.65 on our test set. Not perfect, but a massive improvement for one architectural change.Contextual segmentation: Instead of fixed-page splits, we explored semantic segmentation. Embedding sentences and splitting where topic similarity drops. Better preservation of logical sections, but more complex to implement and tune.Adaptive chunk sizing: Make the chunk size proportional to document length so the entire document always gets processed. Simple, and ensures nothing is skipped. What I'd Recommend Starting With Hierarchical summarization or Summary or Summaries. It's the fastest fix with the highest impact. You can implement it in a day, split into 10-page segments, summarize each, store it in your index, then summarize the summaries. It won't solve everything, but it eliminates the catastrophic "invisible 97%" failure mode on long documents. Building an Evaluation Approach That Actually Works Here's the process I'd set up again if starting from scratch: 1. Get a Stratified Golden Dataset Include documents at 2x, 5x, and 10x your median document length. This is where we failed initially. We tested on short documents and missed the length problem entirely. Minimum 30 document summary pairs. Stratify by length, domain, and complexity. Here's what a single entry looked like in ours: JSON { "doc_id": "REG_2019_1156", "source_file": "regulation_2019_1156.pdf", "pages": 12, "domain": "environmental_compliance", "reference_summary": "This regulation establishes emission standards for...", "key_provisions": [ "Article 4: Emission limits for Category 3 machinery", "Article 7: Penalties up to $10M for non-compliance", "Article 12: Transitional period ending 2024-01-01" ], "difficulty": "medium", "notes": "Critical penalty clause is on page 9 - good test for coverage" } If you notice the field key_provisions, this field was the most valuable addition. It lets us measure not just vocabulary overlap but whether specific critical facts/insights were captured. If Article 7's penalty clause doesn't appear in the summary, that's a material failure regardless of what ROUGE says. I implemented a scoring pipeline that ran nightly: a Lambda function pulled new summaries, computed F1 and BERTScore against the golden references, and pushed results to CloudWatch with alerts on threshold breaches. 2. Score at Multiple Levels Don't just compute one number. Break it down: By document length: This is where you'll find the cliffBy document type: Some content is inherently harderBy section position: Are conclusions (typically at the end) being missed? 3. Set Honest Thresholds Metric Minimum (don't ship below) Target Action if below minimum F1 0.60 0.80 Revisit your chunking/coverage approach Recall 0.50 0.70 You're missing content — increase document coverage Precision 0.75 0.85 Check for hallucination 4. Don't Skip Human Evaluation I've seen cases where ROUGE was 0.45 (acceptable) and the summary was completely wrong because it used the right vocabulary to say the wrong things. Automated metrics are necessary but not sufficient without human supervision. I know it's expensive, and it might slow things down, but don’t skip human evaluation. For faithfulness specifically, checking whether the summary only says things that are actually in the source, we used an LLM-as-judge with this prompt pattern: Markdown “You are evaluating a summary for faithfulness to its source document. SOURCE DOCUMENT (or relevant excerpt): {source_text} GENERATED SUMMARY: {generated_summary} For each factual claim in the summary: 1. State the claim 2. Is it SUPPORTED, PARTIALLY SUPPORTED, or NOT SUPPORTED by the source? 3. If supported, quote the relevant passage. Then give an overall faithfulness score (0.0 - 1.0): - 1.0 = every claim is directly supported - 0.5 = some claims lack source backing - 0.0 = the summary fabricates or contradicts the source Output your assessment as structured JSON.” This caught things ROUGE never could. Cases where the summary stated the penalty was "$5M" when the regulation actually said "$10M." The words overlapped; the meaning was wrong. Sample 20–30 summaries. Have domain experts rate completeness and accuracy on a 1–5 scale. This is your production readiness gate. Common Mistakes to Avoid Testing only on short documents: Our development set was mostly 10–20 page documents. Everything looked great. The length problem was invisible until we deliberately tested on 100+ page documents.Trusting Precision alone: When we found "Precision: 0.84", I thought we were in good shape. I should have immediately asked, "and what's the Recall?" High Precision with unknown Recall means nothing.No baseline comparison: We didn't know if our pipeline was better than simply taking the first 3 paragraphs. Turns out, for short documents, "first 3 paragraphs" was a surprisingly strong baseline.Evaluating once, not continuously: The first evaluation revealed the length problem. We fixed it, but we didn't set up ongoing monitoring. Six months later, new document types crept in that the pipeline struggled with, and nobody noticed until a stakeholder complained. Key Takeaways Measure Recall independently. If your Precision is high but you haven't checked Recall, you might have a confident liar. Test on long documents: Your pipeline's real weakness only appears at 5x your median document length. If you're not testing there, you don't know your system.The length/recall correlation is real: r = -0.71 in our case. If you're using any fixed-size processing approach, expect the same pattern.Fix coverage first: If Precision is already 0.80+, your problem is almost certainly Recall. Hierarchical summarization is the fastest fix.Automated metrics get you 80%. The last 20% requires human eyes. Evaluate continuously: A continuous automated evaluation pipeline ensures the system remains reliable over time. The goal isn't a perfect score, it's knowing exactly where your pipeline fails so you can make informed decisions about what to deploy, and what to hold back. Build the evaluation pipeline before you need it, because by the time a compliance team finds the gap, you've already shipped a confident liar.

A few months ago, our RAG (Retrieval-Augmented Generation) setup did what most first versions do: the Streamlit app called OpenAI directly, pulled chunks from MongoDB, and stitched an answer together. It worked. It also had three problems nobody wants to discover in production. First, the OpenAI bill. Users ask the same question in ten different ways: "How many leave days do I get?" and "What's the annual leave entitlement?" are the same question, but if you're calling the LLM for every single one of them, you're paying for the same answer over and over. Second, if OpenAI had a slow minute, every single user felt it. Nothing was cached anywhere, so every request, no matter how routine, went all the way to the model. Third, and honestly the scarier one: an LLM will happily answer a question even when the answer isn't in your documents. It doesn't say "I don't know"; it just generates something plausible. For an internal knowledge assistant, that's not a bug you can shrug off — that's a wrong HR policy answer someone acts on. So we built a gateway. Nothing exotic — a single FastAPI service that sits between the UI and everything else (OpenAI, Gemini, MongoDB). The client never talks to those systems directly anymore; it only ever talks to the gateway, and the gateway decides what happens next. Why a Gateway and Not Just “Add Caching Somewhere” The tempting shortcut is to bolt caching onto the existing call site and move on. We didn't do that, mainly because caching, grounding, routing, rate limiting, and logging all need to see the same request at the same point in its lifecycle. Splitting that logic across the UI and a dozen helper functions means every future change touches five files. Put it all in one place instead, and it's one file you read to understand the entire request path. Nothing here is framework magic. No LangChain, no LlamaIndex. It's the OpenAI SDK and MongoDB's native $vectorSearch called directly. That was a deliberate choice — when something behaves oddly, we want to be able to step through our own code, not a framework's abstraction layer. What Actually Happens on a /Query Call This is the part that matters most, so here's the full path a question takes: Each of those boxes guards against one of the three problems above. Rate limiting protects against abuse. The cache lookup protects against paying twice for the same question. The grounding gate protects against confidently wrong answers. The budget check protects the invoice. And the last step means none of this is a black box — every request is logged with its cache status, provider, tokens, and cost, and shows up live in the Analytics tab. Semantic Caching: Matching What People Mean, Not What They Typed Here's the part people usually find counterintuitive. We don't do exact-string caching. We embed the incoming query the same way we embed document chunks, and compare it against previously answered queries stored in their own cache collection, using cosine similarity. If the match is above 90%, we return the stored answer straight away — no LLM call at all. What that looks like in practice: "How many annual leave days?" -- miss (first time anyone's asked)Same question again -- hit (exact repeat)"What's the yearly vacation entitlement...?" -- hit (different words, same question)"Meal allowance during travel?" -- miss (genuinely different topic) That third line is the whole point. An exact-match cache would have missed it and burned an LLM call for a question we'd already answered five minutes earlier. Cosine similarity catches the paraphrase. We also cache at ingestion time, which is easy to overlook but saves real money. Before embedding a document chunk, we hash it. If that hash already exists, we skip it — no embedding call, no reprocessing. Re-uploading a document you already ingested costs nothing. Only the chunks that actually changed get new embeddings. Making Sure the Model Doesn’t Just Make Something Up This is the guardrail we care about most, and it's deliberately built as two separate layers rather than one clever one: A confidence gate before generation. We retrieve the top-k chunks and check the best similarity score. If it's below our grounding threshold, we never even call the LLM — we return a "not in the documents" response instantly. Cheap, fast, and it can't hallucinate because it never ran.A strict system prompt during generation. Even when the gate passes, the model is explicitly instructed to refuse if the retrieved context doesn't actually contain the answer. Neither layer alone is bulletproof. Together, they catch most of the ways an LLM can wander off and answer a question you never asked it to answer with information it invented. Routing on Cost, Not Just on Uptime Our primary model is gpt-4o-mini. The gateway keeps a running total of what we've spent since the last budget reset, and once that total hits the configured limit, new requests get routed to Gemini 2.5 Flash instead — called through OpenRouter's OpenAI-compatible API, so it's the same SDK, just a different base URL and model name. No separate integration to maintain. The part worth calling out: the budget number is a runtime setting, not a constant baked into the code. Whoever owns the budget can change it without anyone redeploying anything. And if the fallback provider is ever misconfigured, we fail back to the primary rather than failing the request outright. What We Get to See None of this is useful if it's invisible. Every request - cache status, grounding result, provider used, tokens, cost, latency — gets logged, and it's all visible in the Analytics screen. There's also a Routing screen that shows the currently active provider alongside spend versus budget, in real time. Rate limiting runs on a 60-second sliding window per client IP, so one runaway script can't drown out everyone else. Where That Leaves Us PatternWhat it doesWhy it mattersGateway / ProxyOne FastAPI service in front of all RAG trafficOne place to change behavior, not fiveSemantic CachingCosine similarity match against past queriesLower cost, faster answers on repeat/paraphrased questionsGrounding GateConfidence threshold + strict system promptFewer hallucinated answersCost-Based RoutingAuto-switch provider once budget is hitSpend stays predictable, no manual babysittingEmbedding DedupHash chunks, skip re-embedding unchanged onesCheaper, faster re-ingestionRate Limits + LoggingSliding window per IP, per-request metricsAbuse protection and actual visibility into cost None of this required a heavyweight framework — it's FastAPI, MongoDB Atlas, and the OpenAI SDK, wired together deliberately. The bigger lesson for us wasn't any single pattern; it was that caching, grounding, and routing all want to live at the same layer, because they're all answering the same underlying question: Should this request even reach the model, and if so, which one? If you're running a RAG app that talks to the LLM directly today, that question is worth asking before the bill or a bad answer asks it for you.

This article outlines how Delta Lake tables managed in Databricks and those managed in Snowflake can be shared in both directions without duplicating data. This is achieved through the use of shared object storage (ADLS) and open table formats. The article discusses the benefits, tradeoffs, business and cost advantages, the approach to implementation, and main technical aspects to consider. Overview Interoperability enables one platform to access data that is owned by another platform, without moving the data.Both Databricks and Snowflake store a single copy of the underlying Parquet files in ADLS, and they share only table metadata through catalog APIs. Databricks stores data in Delta (or Delta Uniform), which Snowflake can read via catalog integration.On the other hand, Snowflake-managed Iceberg tables can be accessed by Databricks through catalog federation. Metadata is the Key to Interoperability Key Advantages No data duplication: One physical copy of data in ADLS serves both engines, eliminating duplicate storage and copy pipelines.Meet users where they are: Producers publish once; consumers query from Databricks or Snowflake with their own compute.Best-of-breed & open formats: Open Parquet + Delta/Iceberg formats avoid lock-in and support future OLTP/OLAP convergence.Fresher data: Consumers read the latest committed metadata rather than a stale nightly extract. Pros and Cons Points to consider Pros Cons / Limitations No storage duplication — single copy in ADLS. Cross-engine writes are limited; the consuming engine is typically read-only. Lower latency than nightly copy / ETL jobs. Metadata refresh must be configured; a lag exists between write and visibility. Uses each engine's native compute and cost model. Converting Snowflake tables to Iceberg can surface datatype mismatches. Open formats reduce vendor lock-in. Governance must be re-applied per engine—policies do not travel with the table. Business and Cost Benefits Lower cost: Removes redundant storage and the compute cost of copy/sync pipelines between the two platforms.Speed: Lower data and process latency means data products reach consumers sooner (faster GTM).Compliance: A single governed copy supports data minimization and retention obligations.Agility: Teams pick the right tool for each workload without re-platforming the underlying data. Databricks to Snowflake: Implementation Approach Databricks owns the Delta/Uniform table, while Snowflake retrieves metadata through the Catalog Integration API and reads the data directly from ADLS. Catalog Integration Flow Snowflake requests metadata from Unity Catalog, receives the table location, and reads Parquet data directly from ADLS using an external volume. Databricks-Managed Delta Read by Snowflake (Catalog Integration) Databricks remains the owner; Snowflake reads in place. 1. Create external Delta tables with UniForm (Iceberg) enabled in ADLS, registered in Unity Catalog. SQL CREATE TABLE Catalog.SchemaName.TableName ( ID INT, Name STRING, CreatedAt TIMESTAMP ) TBLPROPERTIES ( 'delta.columnMapping.mode'='name', 'delta.enableIcebergCompatV2'='true', 'delta.universalFormat.enabledFormats'='iceberg' ); 2. Create an external volume in Snowflake pointing to the ADLS path (Managed Identity authentication). SQL CREATE OR REPLACE EXTERNAL VOLUME DEVTEAMADLS_EXE_VOL STORAGE_LOCATIONS = ( ( NAME='DEVTEAMADLS_EXE_VOL', STORAGE_PROVIDER='AZURE', STORAGE_BASE_URL='azure://devteamadls.blob.core.windows.net/Data/', AZURE_TENANT_ID='<tenant id>' ) ); 3. Create a catalog integration so Snowflake reads current metadata from Unity Catalog's Iceberg REST endpoint. SQL CREATE OR REPLACE CATALOG INTEGRATION DEVADLS_CATALOG_REST CATALOG_SOURCE = ICEBERG_REST TABLE_FORMAT = ICEBERG CATALOG_NAMESPACE = 'SchemaName' REST_CONFIG = ( CATALOG_URI = 'https://<workspace>.azuredatabricks.net/api/2.1/unity-catalog/iceberg', WAREHOUSE = 'DevDeltaLake_Warehouse' ) REST_AUTHENTICATION = ( TYPE = OAUTH, OAUTH_TOKEN_URI = 'https://<workspace>.azuredatabricks.net/oidc/v1/token', OAUTH_CLIENT_ID = '<shared via secure exchange>', OAUTH_CLIENT_SECRET = '<shared via secure exchange>', OAUTH_ALLOWED_SCOPES = ('all-apis','sql') ) ENABLED = TRUE; 4. Create the Iceberg table in Snowflake and apply RBAC + masking on the Snowflake side. SQL CREATE ICEBERG TABLE SFDBName.SFSchema.SFTableName EXTERNAL_VOLUME = 'DEVTEAMADLS_EXE_VOL' CATALOG = 'DEVADLS_CATALOG_REST' CATALOG_TABLE_NAME = 'DeltaTableName' CATALOG_NAMESPACE = 'DeltaSchemaName' AUTO_REFRESH = TRUE; ALTER ICEBERG TABLE ... MODIFY COLUMN <col> SET MASKING POLICY <policy>; Snowflake to Databricks: Implementation Approach Snowflake remains the owner; Databricks reads in place. Query federation is the lighter alternative when Iceberg conversion is not desired. Snowflake owns the Iceberg table; Databricks federates the catalog and reads data in place from ADLS. Catalog Federation Flow Databricks federates the Snowflake catalog to retrieve the latest metadata, receives the updated table location, and reads Parquet data directly from ADLS through an external location using Managed Identity. Snowflake-Managed Data Read by Databricks (Catalog Federation) 1. Create Iceberg tables in the Snowflake-managed catalog on an external volume path. SQL CREATE OR REPLACE ICEBERG TABLE WareHouse.Schema.TableName ( c_custkey INTEGER, c_name STRING, c_acctbal INTEGER ) CATALOG = 'SNOWFLAKE' EXTERNAL_VOLUME = 'my_ext_vol' BASE_LOCATION = 'schema/tablename'; 2. Provision a resource account (key pair) and grant Databricks' role read access to the Iceberg tables. SQL ALTER USER UserName SET RSA_PUBLIC_KEY='<public key>'; GRANT USAGE ON DATABASE DatabseName TO ROLE MY_DATABRICKS_ROLE; GRANT USAGE ON SCHEMA DatabseName.SchemaName TO ROLE MY_DATABRICKS_ROLE; GRANT SELECT ON ALL TABLES IN SCHEMA DatabseName.SchemaName TO ROLE MY_DATABRICKS_ROLE; GRANT SELECT ON FUTURE TABLES IN SCHEMA DatabseName.SchemaName TO ROLE MY_DATABRICKS_ROLE; 3. In Unity Catalog, create the Snowflake connection, an external location for the ADLS path, and a foreign catalog with the authorized path. SQL CREATE CONNECTION SnowFlakeConn TYPE snowflake OPTIONS ( host '<acct>.snowflakecomputing.com', port '443', sfWarehouse '<wh>', user 'UserName', <key-pair auth options> ); CREATE EXTERNAL LOCATION snf_iceberg_loc URL 'abfss://<container>@<storage>.dfs.core.windows.net/' WITH (STORAGE CREDENTIAL <cred>); CREATE FOREIGN CATALOG SnowFlakeCatalog USING CONNECTION SnowFlakeConn OPTIONS ( database '<snf_db>', authorized_paths 'abfss://<container>@<storage>.dfs.core.windows.net/' ); 4. Refresh and query. Databricks reads Iceberg data directly from ADLS using its own compute. Apply RBAC and masking policies in Databricks. SQL REFRESH FOREIGN CATALOG SnowFlakeCatalog; SELECT * FROM SnowFlakeCatalog.SchemaName.TableName; Conclusion Through the use of interoperability, both Databricks and Snowflake will be able to share data bidirectionally through a single version of the data that is present in ADLS, but with metadata exchange. This will lead to no duplication of data and will always provide access to fresh data. There are several advantages for businesses in terms of cost savings through storage and ETL cost reduction, faster access to data, increased flexibility, and vendor lock-in reduction.