DZone Spotlight

A transaction reads one row and then remains open. Later, a deployment adds a column using ALTER TABLE ... ALGORITHM=INSTANT. The migration waits. A new request issues an ordinary SELECT against the same table, and that request waits too. The original query has already finished. The schema change requires no table rebuild. The new query accesses a different row. Yet all three sessions participate in the same blocking sequence. The explanation sits above InnoDB’s row-lock layer. The open transaction retains a metadata lock, the schema change requests an incompatible lock, and that pending request can make later queries queue behind it. Instant DDL reduces the work required to change a table. It does not eliminate the coordination required to begin or complete that change. Reproduce the Queue With Three Connections Use a disposable database and three independent MySQL connections. A tiny table keeps data volume out of the explanation: MySQL CREATE DATABASE mdl_demo; CREATE TABLE mdl_demo.orders ( id BIGINT PRIMARY KEY, status VARCHAR(20) NOT NULL ) ENGINE=InnoDB; INSERT INTO mdl_demo.orders VALUES (1, 'NEW'), (2, 'NEW'), (3, 'DONE'); In Session 1, start a transaction and read one row: MySQL START TRANSACTION; SELECT * FROM mdl_demo.orders WHERE id = 1; -- Leave the transaction open. The query returns immediately. Do not commit yet. In Session 2, request the schema change: MySQL ALTER TABLE mdl_demo.orders ADD COLUMN source VARCHAR(32) NULL, ALGORITHM=INSTANT; After the ALTER begins waiting for a metadata lock, issue another read in Session 3: MySQL SELECT * FROM mdl_demo.orders WHERE id = 2; This later read can also wait, despite accessing a different row. Committing Session 1 allows the migration to proceed, followed by the queued read. MySQL’s verified Bug #106480 documents this three-connection behavior for MySQL 8.0. Test the sequence against your deployed version rather than assuming every DDL operation has identical locking details. Separate Execution Cost From Lock Admission For supported operations, ALGORITHM=INSTANT changes metadata without rebuilding the table. Whether a particular alteration qualifies depends on the operation, table characteristics, and server version. MySQL publishes these restrictions in its online DDL operation matrix. That optimization removes potentially expensive data movement. It does not promise a maximum elapsed time. The migration’s duration includes time spent acquiring required locks. A metadata change that executes quickly after admission can still wait behind an existing transaction for minutes. This distinction should influence deployment planning. Estimating migration risk from table size alone misses the failure demonstrated here. A three-row table can experience the same admission problem as a much larger one. The operational question is therefore twofold: how expensive is the change once admitted, and how long may it wait while affecting other requests? The Blocker Can Be Idle For tables used within a transaction, MySQL retains transactional metadata locks until the transaction ends. Finishing a SELECT does not necessarily release them. See MySQL’s metadata-lock lifetime documentation. Consequently, searching only for slow queries can miss the session responsible. Application code commonly stretches transactions beyond their SQL work. An endpoint reads data, calls a remote service, performs additional processing, and commits afterward. The database query takes milliseconds, but the transaction remains open throughout the external work. A connection can therefore appear idle while still preventing a schema change. Measure transaction age separately from query duration. Review transaction boundaries around network calls, user interaction, and background work. Moving unrelated operations outside a transaction can reduce exposure without changing the migration itself. Preflight checks help identify existing blockers, but they cannot guarantee that another transaction will not start immediately afterward. Why Later Readers Wait Without the pending ALTER, the two reads in the example could coexist. The DDL request changes the scheduling situation. MySQL normally prioritizes metadata write-lock requests over read-lock requests, with documented exceptions. That policy prevents a continuous stream of readers from indefinitely starving a writer. The priority rules appear in the metadata-lock acquisition documentation. On a busy table, this creates a widening queue. One open transaction delays the migration, and the waiting migration delays newly arriving work. Those requests can occupy application connections until the pool fills. Subsequent requests may then wait for a connection before reaching MySQL at all. This is why a migration incident can look like general database saturation even when CPU and storage activity remain modest. The database is coordinating access, not necessarily executing expensive queries. Inspect Ownership and Pending Requests Start with a diagnostic query that connects waiting sessions to potential blockers: MySQL SELECT object_schema, object_name, waiting_pid, waiting_lock_type, blocking_pid, blocking_lock_type FROM sys.schema_table_lock_waits WHERE object_schema = 'mdl_demo' AND object_name = 'orders'; The sys view documentation describes these session and lock relationships. Use the result alongside transaction age and application ownership. Identify which connection belongs to the migration runner and which belongs to the application. For lower-level investigation, inspect performance_schema.metadata_locks and correlate its owner thread identifiers with performance_schema.threads. Distinguish granted locks from pending requests rather than treating every displayed lock as an active blocker. Do not automatically terminate the oldest connection. It may hold legitimate work whose rollback is expensive. When the migration is deferrable, removing its waiting request may be the less disruptive intervention. Set the Correct Timeout The familiar innodb_lock_wait_timeout controls InnoDB row-lock waits. It does not bound the metadata-lock wait responsible for this incident. Configure the migration connection instead: MySQL SET SESSION lock_wait_timeout = 3; ALTER TABLE mdl_demo.orders ADD COLUMN source VARCHAR(32) NULL, ALGORITHM=INSTANT; In MySQL 8.4, lock_wait_timeout governs metadata-lock acquisition attempts and defaults to 31,536,000 seconds. The timeout applies separately to each acquisition attempt, so it is not a universal three-second statement deadline. See the server variable reference. Three seconds is illustrative. Choose a budget based on acceptable application disruption. Apply the setting on the actual connection executing the migration. Setting it on a separate administrative connection has no effect on the migration session. A dedicated connection also avoids leaving unexpected timeout settings in an application pool. Verify That Timeout Releases the Queue Extend the reproduction using a fresh table definition. Keep Session 1’s transaction open. Start the migration with a short metadata-lock timeout, then issue Session 3’s read while the migration is waiting. When the ALTER times out and its pending request is removed, the later read should proceed in this isolated scenario even though Session 1 remains open. That observation identifies the queue boundary. The two reads were compatible; the waiting schema change prevented the later one from advancing. A failed migration must remain visible to deployment automation. Do not mark it complete merely because application traffic recovered. Retry with backoff and a bounded attempt count rather than repeatedly recreating the same queue. After a client disconnect or ambiguous execution result, inspect the actual schema before retrying. Make Admission Part of the Deployment Contract A production migration policy should specify the permitted algorithm, metadata-lock wait budget, retry behavior, and conditions for postponement. Monitor pending DDL alongside transaction age, connection-pool usage, and request latency. Alerting only on migration duration can detect the problem after application capacity has already been consumed. Test both successful admission and blocked admission before deployment. The latter proves that the migration yields when production conditions are unfavorable. Instant DDL remains valuable because it avoids unnecessary table reconstruction. Its safety depends on the surrounding lifecycle: transactions must finish, lock requests must be admitted, and waiting migrations must have a bounded opportunity to interfere with traffic.

Most of the SaaS teams get the scaling question backward. They spend around six months debating Kubernetes clusters and microservices for a product that has limited customers. And when the same team gets an enterprise client and suddenly has 500 concurrent users, the database falls. This is because no one thought about connection pooling. The architecture decisions that actually matter for a SaaS product aren't about picking the trendiest stack. They're about sequencing. Which parts of the system need to scale on day one, and which parts can wait until there's real traffic to justify the complexity. A payments startup I've seen referenced in postmortems rebuilt its entire auth layer twice in eighteen months, first to add multi-tenancy, then again to fix a data leak between tenant accounts that never should have been possible in the first place. This piece walks through the architectural decisions that hold up as a SaaS product goes from ten users to ten thousand: how to structure the application layer, what multi-tenancy actually requires, and where security has to be built in rather than patched on later. What SaaS Architecture Actually Means SaaS architecture is the technical blueprint for how a product gets built, deployed, and shared across every customer using it. A desktop app or an internal company tool typically serves one user or one organization. SaaS application architecture is different because it needs to run different customer accounts on the same codebase and the same database. It has a shared setup, and that is the reason SaaS works as a business model. One codebase, sold repeatedly, instead of a fresh install for every buyer. But it changes what a bad query or a bad index actually costs you. On a single-tenant system, a slow query is one customer's problem. On a multi-tenant platform, that same query can drag down everyone who happens to be logged in when it runs. Solid saas architecture plans for this early. Teams need to figure out how many concurrent users the system can handle, how tenant data stays separated on shared infrastructure, and how new capacity gets added as signups grow. Updates need a path to production that doesn't take the whole platform offline. None of this requires unusual technology. It requires making these calls while the codebase is still small enough to change your mind cheaply. Start With a Modular Monolith, Not Microservices The architecture decision that shapes everything downstream is whether to start with microservices or keep things simple. Most teams get this wrong in the same direction. The Cost of Splitting Too Early A lot of SaaS product development advice treats microservices as step one, and that's backward for most teams. Once you split a product into a dozen independently deployed services, someone has to manage network calls between them, handle service discovery, and debug failures that span multiple systems. A five-person engineering team rarely has the bandwidth for that on top of building the actual product. What a Modular Monolith Looks Like Most SaaS teams don't need thirty microservices. They need one codebase that's organized well. A modular monolith ships as a single app, but internally, billing code lives in a billing module, user accounts live in their own module, and the core product logic stays separate from both. Each piece owns its own data and only talks to the others through a defined interface. Splitting things out later isn't painful if you set it up this way. Say billing starts getting heavier traffic than the rest of the app. You already have a clean boundary around it, so turning it into its own service is a deployment problem, not a rewrite. Real Companies That Stayed Monolithic Basecamp built on Rails and stayed on a monolith for over twenty years while serving millions of users. Shopify did the same thing on a modular Rails codebase, well past the point where a lot of engineering blogs said they should have moved to microservices. They eventually split off specific pieces, but only once those pieces had scaling needs the rest of the platform didn't. When Splitting Actually Makes Sense Forget team size or how much funding you've raised. Look at whether different parts of your SaaS application architecture need separate scaling, separate release schedules, or separate infrastructure. If billing and your core product feature grow together and the same engineers maintain both, there's no reason to pull them apart. Microservices fix a coordination problem. That problem shows up once a system gets big. Building for it before you're big just adds latency and more moving parts to maintain. What Is Multi-Tenant Architecture in SaaS Basic Definition Multi-tenant architecture in SaaS means multiple customers, or tenants, share the same application and often the same database, but each tenant's data stays walled off from the others. One codebase, one running instance, thousands of accounts on it. Nobody sees anyone else's data. That's different from giving each customer their own separate deployment, which is still how a fair amount of older enterprise software works. Multi-tenancy is what makes it possible to support a few thousand customers without babysitting a few thousand installations. The Three Isolation Models There are three common ways teams set up tenant isolation, and I've watched companies pick the wrong one for their stage more than once. Siloed isolation gives each tenant a fully separate database. It's the safest option and the easiest one to explain to a security auditor. It's also expensive past a certain point. I know a healthcare SaaS company that ran siloed databases for its first 40 enterprise clients and then spent three months just automating the database provisioning process because doing it by hand had become a full-time job for someone. Pooled isolation is the opposite extreme. Every tenant lives in the same tables, tagged with a tenant ID column. Cheap, scales fine, and it puts nearly all the isolation burden on your application code getting every single query right. Forget a WHERE clause once, and you've leaked another company's data. This is where most of the ugly security incidents in SaaS come from, not from hackers, but from a developer who missed a filter on a Tuesday afternoon. Bridge isolation, also called schema-per-tenant, splits the difference. Same database, separate schemas per tenant. Costs less than siloing, still gives you a real boundary instead of relying purely on application logic. Most products that get past a few hundred customers land on pooled or bridge, then add row-level security at the database level as a backstop, so a missed query filter doesn't turn into a headline. Multi-Tenant SaaS Architecture Best Practices Picking an isolation model is step one. Making it actually hold up under real traffic is a different problem, and it's the one that bites teams later. 1. Enforce Isolation at the Database Layer Row-level security should sit at the database layer, not just in your application code. Postgres has native RLS policies for exactly this reason. Enforce tenant filtering at the query engine, and even a buggy application query can't accidentally cross tenant lines. Relying on application code alone means every engineer who ever touches the codebase has to remember the tenant filter, on every query, indefinitely. Sooner or later somebody forgets. 2. Watch for Noisy Neighbors One tenant running a bulk export or a heavy report can drag down query times for everyone else sharing that database. I've seen a single client's 2-million-row export slow the whole platform to a crawl during business hours. Connection pooling limits per tenant help. So does routing heavy jobs like exports and report generation into a separate queue, away from regular user traffic. 3. Tag Every Cache Key With a Tenant ID Caching gets trickier with multiple tenants sharing infrastructure. A cache key without a tenant identifier baked into it is a bug waiting to happen. I've seen a caching layer serve Tenant A's dashboard data to Tenant B because the cache key was just dashboard_summary instead of dashboard_summary:tenant_492. Every cache key, every session token, every background job payload needs the tenant ID attached, no exceptions. 4. Automate Tenant Provisioning Before You Need To Tenant provisioning also needs to be automated early, even if the first version is a single script. Manually setting up a new tenant's schema or database entry works fine for your first ten customers. It falls apart around customer thirty, and by then you're doing it under pressure because sales already promised a launch date. The Core Infrastructure Layers Once the application logic and tenant model are settled, the rest of the system comes down to a few layers that each need to do one job well, without leaning on each other to cover for gaps. 1. Application Layer Session data goes into Redis, not local server memory. This is what makes horizontal scaling actually work. When sessions live centrally, a load balancer can route a user's next request to any server in the pool, and it doesn't matter which one handled their login. Skip this step, and users get logged out at random whenever a server gets cycled out during autoscaling. Kubernetes and ECS solve the scaling mechanics. They don't fix a stateful app. 2. Data and Queue Layer PostgreSQL or Aurora with read replicas is the standard setup past a few thousand users. Writes go to the primary, reads spread across replicas, and dashboard queries stop competing with transactional writes for the same connections. Queues handle everything that doesn't need to finish before a response goes back to the user. Sending an email, generating a PDF, processing an uploaded file. Drop these into SQS or a Redis-backed job runner, return the response immediately, and let a background worker pick it up. Report generation is the classic case where teams skip this early and pay for it later, since a report that takes eight seconds to build will tie up a web server for eight seconds per request if it's running inline. 3. Automation and Observability Terraform for infrastructure, CI/CD for deployments, Grafana or Datadog for monitoring. Set these up from the start rather than after the first outage. Dashboards catch a database running near capacity days before it turns into customer complaints about slow page loads. Security Considerations for SaaS Architecture Security in a multi-tenant system isn't a feature you bolt on before a compliance audit. It's a property of the architecture itself, and it either works because of how the system is built or fails because of how it wasn't. 1. Tenant Data Isolation Under Load Every isolation model discussed earlier, siloed, pooled, or bridge, needs isolation enforced at more than one layer. Application-level checks catch most mistakes. Database-level row security catches the ones that slip through code review. Relying on a single layer means one bug becomes a breach instead of a near miss. 2. Authentication and Authorization at Scale Auth gets more complicated once you're dealing with tenant-level roles on top of user-level permissions. A user might be an admin for one tenant and a regular user on a different account if your product allows multi-org membership. Token scopes need to carry tenant context, not just user identity, or a valid session for one account can end up touching data in another. 3. Encryption Boundaries Data needs encryption at rest and in transit, but the boundary that actually matters for SaaS is key management per tenant, particularly for enterprise customers who ask about it directly during procurement. Some products use a shared encryption key across all tenants. Others generate a unique key per tenant so a compromised key doesn't expose the whole customer base at once. The second approach costs more engineering time up front and saves you from a much worse conversation later. 4. Compliance Triggers as You Grow SOC 2 and GDPR requirements don't show up on day one, but the architectural decisions that make compliance possible do. Audit logging, data retention controls, and the ability to fully delete a tenant's data on request are much easier to build in from the start than to retrofit once an enterprise prospect asks for a SOC 2 report during contract negotiation. I've watched a sales cycle stall for two months while an engineering team scrambled to add audit logs that should have existed already. Common Scaling Mistakes to Avoid Before Building a SaaS Architecture Most of the problems that show up later trace back to a handful of decisions made early, usually for reasonable-sounding reasons at the time. 1. Splitting Into Microservices Before There's a Reason To This one keeps coming up because it's the most common piece of bad advice in SaaS engineering circles. A team with six engineers and no paying customers yet decides to build fifteen microservices because that's what they read scales. Then they spend more time debugging service-to-service network calls than building the product features that would actually get them customers. 2. Storing Session State on the Application Server Local session storage feels harmless until you try to scale horizontally and discover half your users are getting logged out every time a server restarts. This is usually caught late, often during the first real traffic spike, because it works fine at low volume and only breaks once load balancing starts routing users unpredictably across servers. 3. Skipping Read Replicas Until the Database Is Already Struggling Adding a read replica takes an afternoon when your database is healthy. It takes a stressful weekend when your primary database is already choking under load, and every dashboard query is competing with checkout transactions. Teams tend to wait until performance is visibly bad before setting this up, which is exactly backward. 4. Treating Security as Something to Add Before the Compliance Deadline Row-level security, audit logging, and tenant-scoped encryption keys are all much cheaper to build into the initial schema than to retrofit onto a database with millions of existing rows. I've seen a data migration to add proper tenant isolation take longer than the original feature that needed it. When to Bring in Outside Architecture Expertise Not every team needs to hire this out or figure it out alone. The right software development team structure can provide the expertise needed to scale efficiently. 1. Signs You Need Outside Help A saas architecture consultant earns their fee when a team is making irreversible decisions without the experience to know which ones are irreversible. Choosing a multi-tenant isolation model is one of those. So is picking a database that can't handle your projected write volume two years out. If nobody on the team has taken a SaaS product through a scaling event before, an outside consultant who has seen the same mistakes across a dozen other companies can save months of rebuilding. The opposite signal matters too. If your team already has someone who's built multi-tenant systems before, bringing in a consultant to re-litigate decisions that person already knows how to make just slows things down and adds a layer of translation between the person with the answer and the person implementing it. 2. Build In-House vs. Bring in Specialized Support Full SaaS product development makes more sense when a company needs to move fast on a product with no existing engineering team, not when an established team needs a second opinion on one architectural decision. Hiring a fractional consultant for a two-week architecture review costs a fraction of what a bad multi-tenancy decision costs to unwind eighteen months later, after customer data is already sitting in the wrong shape across a production database. The decision comes down to a specific question, not a general one: is the problem "we don't know what to build" or "we don't have enough hands to build it"? Those need different kinds of outside help, and a lot of engagements go sideways because a team hired for one problem while actually having the other. Conclusion Building a SaaS product that holds up under real usage comes down to sequencing decisions correctly, not chasing the most sophisticated stack available. Start with a modular monolith. Get multi-tenant isolation right at the database layer, not just in application code. Keep application servers stateless so horizontal scaling actually works when you need it. None of this is exotic engineering. It's a set of ordinary decisions made at the right time, before the codebase gets big enough that reversing course means months of migration work instead of an afternoon of refactoring. If you're mapping out architecture for a new SaaS product, it's worth getting a second set of eyes on the isolation model and data layer decisions before they're locked into a production schema.

When you move messaging onto a third-party platform, you don't hand off the authorization problem. You split it across two systems. Most incidents live in the gap. A simulation harness, and a documentation audit of QuickBlox as a worked example. The Revoked User Who Kept Reading Say a clinic's app removes a contractor's access to a patient thread at 14:02. The app's database says so, the UI hides the thread, and the audit table has the row. The contractor's phone keeps receiving new messages until 15:00, and then keeps going. Nothing in the application is wrong. The membership removal went to the chat platform's API as a single best-effort call, and the API was having a bad ten minutes. The call failed, and the exception was swallowed. The contractor's session token is still valid, and the contractor's client keeps making requests that keep it valid. This is a hypothetical, but every piece is routine: a removal path with no retry, an admin API that can be unavailable, a session whose lifetime depends on activity. The useful question is how long the window lasts, and it becomes measurable once you stop treating the integration as an API call and start treating it as a consistency boundary. This article builds a small deterministic harness, seamlab.py, to measure that window along with seven other failures at the same boundary, including an AI agent acting after a human has taken over a conversation. It then audits one real platform, QuickBlox, against the same questions using its public documentation. QuickBlox is the worked example because its developer documentation is public and detailed enough to check. The harness measures a class of platforms and nothing about any vendor. The documentation audit reads documentation and tests nothing. I keep those two kinds of evidence apart throughout. The Flawed Assumption The usual framing of "build vs. buy" for chat or calling is about transport: delivery, ordering, presence, media. Delegating those is reasonable. But "who may read this thread, who may post, who may trigger an action" is not a transport concern, and delegating transport forks it: Your application holds the relationships: this clinician treats that patient, this contractor was removed.The platform holds a projection of them: dialog memberships, session tokens, message history. The projection is only as fresh as your last successful write to it, and the platform enforces the projection, not your database. So at least two places answer "is this allowed," and they can disagree. Each failure below is a case of that disagreement, or of one side believing a claim the other never checked. Two quantities make this concrete. Revocation window W: the time between the app committing a revocation and the last successful read by the revoked principal. If the platform checks live membership on every read, W is bounded by the time until the removal call lands. If it authorizes from the session token alone, W is bounded by the token's remaining life, and whether that life is fixed or extended by activity decides everything.Authority provenance: for any action in the system, whose authority is it exercising? For a human action, the answer is trivial. For an AI agent in a multi-party conversation, it usually isn't. Threat Model Six attacker positions, all realistic for a chat integration: #AttackerCapabilityAExternalKnows the webhook URLBAuthenticated low-privilege participantCan post messages into a thread the agent readsCSelf-registered platform userPlatform allows signup, no verified binding to your user modelDRevoked participantHolds a valid token from before revocationEReplayerHas a captured, validly signed event (logs, proxy, compromised sink)FWriter on the audit storeCan modify rows Trust boundaries: client to app, app to platform (admin API), platform to app (event delivery), agent to tools, and the audit store. ClassPreconditionMechanismTestControlForged eventAUnauthenticated event endpoint acceptedforgedSignature verificationIdentity confusionCSender resolved by mutable attribute (email)spoofed identityImmutable-ID binding tableReplayEDedupe TTL shorter than replay horizonreplayTimestamp window plus retention ≥ 2×windowDuplicate side effectNormal at-least-once deliveryNon-atomic check-then-actraceAtomic claim before actingStale accessDToken and membership outlive revocationrevocation sweepShort TTL, outbox retry, token killAgent confused deputyBInjected text drives a privileged toolA1–A4Parameter-bound approvalPost-handoff actionRace with human takeoverAgent call in flight at handoffA5–A6Epoch fencingScope overreachBAgent reads history beyond sessionA7Capability-scoped history Audit tampering (F) is covered separately below. The Harness seamlab.py is about 480 lines of stdlib Python on a logical clock, with no network and no randomness. It contains: A platform simulator: at-least-once event envelopes signed with HMAC, expiring tokens with scopes frozen at mint, optional idle (sliding) expiry, self-service registration, and an admin API that can be switched off. Whether reads check live membership, and whether a server-side token kill exists, are switches.Two app implementations: a naive one and a hardened one.An agent world with a tool proxy configurable per control, so the table shows which control closes which attack.A worst-case agent: it obeys every CALL tool key=value line in its context. That is deliberate. The tests target the authorization layer and say nothing about how robust any model is to injection. If your model resists injection, that's a bonus. These controls assume it doesn't. The duplicate-delivery test enumerates every distinct interleaving of two workers handling the same event, rather than sampling a few. Naive Design, Then the Attacks The naive ingress handler is a shape I'd expect to find in plenty of codebases: no authentication on the endpoint, dedupe with a TTL, check then act then record, sender resolved by email. def handle(self, env): t = self.seen.get(env["id"]); yield # check if t is not None and self.clock.t - t < self.DEDUPE_TTL: # 300s return "dup" uid = self.resolve(env["body"]["sender"]) # email match if uid: self.run_command(uid, env["body"]["text"]) # act yield self.seen[env["id"]] = self.clock.t; yield # record AttackNaiveHardenedForged unsigned eventsucceedsblockedValidly signed event, spoofed identitysucceedsblockedReplay at t+225 sblockedblockedReplay at t+3600 ssucceedsblockedDuplicate delivery: interleavings producing a double effect60 of 660 of 12 The spoofed-identity row is the interesting one. The event carries a correct platform signature. Nothing was forged. The attacker registered a platform account with email [email protected], unverified. The naive app lowercases and matches against its user table, so a perfectly authentic message from the wrong person runs with the clinician's authority. Signature verification can't help, because the signature is telling the truth about who sent it. The control is a binding table keyed on the platform's immutable user ID, written only by your backend when it provisions the platform user for an authenticated app user. The attacker's ID has no binding, so the event is rejected as unbound. The race result is worse than it looks. 60 of 66 distinct schedules double-fire. Only interleavings where the first worker finishes recording before the second starts checking are safe. This is the ordinary case for any retry storm. The fix is one atomic operation (SET NX EX, or an insert against a unique constraint) before any side effect. The hardened handler: def handle(self, env): if not hmac.compare_digest(env.get("sig", ""), sig_of(env)): return "bad_sig" if abs(self.clock.t - env["ts"]) > self.TOL: return "stale" ok = self.claim(env["id"]); yield # atomic set-if-absent, retention 2*TOL if not ok: return "dup" uid = self.bindings.get(env["body"]["sender"]) if uid is None: return "unbound" self.run_command(uid, env["body"]["text"]); yield return "ok" Mutation-Testing the Control The suite should be able to fail the hardened design; otherwise, the green column is decoration. So I mutated one parameter: dedupe retention set to half the timestamp tolerance. The replay at t+225 s then succeeds, because the event is still inside the accepted time window while its dedupe record has expired. The replay at t+3600 s is still blocked by the timestamp check. That yields an invariant worth writing into code review: dedupe retention must be at least twice the accepted timestamp skew (once for events slightly in the future, once for slightly in the past). Timestamp validation and dedupe are two halves of one control, and shortening either silently opens the gap. One caveat: claim-before-act gives at-most-once side effects if the process dies between claim and effect. For effects that must not be lost, the downstream operation needs its own idempotency key derived from the event ID. The harness does not model crash recovery. Revocation Window, Measured The harness drives a revoked user who keeps polling and refreshing, with the token's age at revocation sampled across its lifetime. The scenario that matters: the admin API is down when the revocation is issued and comes back 600 s later. In the first table, the platform checks live membership on every read: ConfigWorst W (s)Mean W (s)Token mints per user per hourNaive, TTL 3600, single attempt360018301Hardened, TTL 303016120Hardened, TTL 60603060Hardened, TTL 30030015212Hardened, TTL 9006014054Hardened, TTL 36006015561 "Hardened" means a transactional outbox retrying the removal with capped backoff, a broker that re-checks the app's authorization on every token mint, and a short TTL. Once TTL exceeds the outage, W is capped by the outage plus up to one backoff interval (601 s in this run). Below the outage length, TTL is the binding constraint. With a healthy admin API, every configuration, naive included, shows W = 0. The naive design only fails when the platform is degraded, which is when nobody is watching. When the Platform Doesn’t Check Membership, and Tokens Slide Two platform properties change the picture, and I only added the second after reading the vendor documentation described below, which is why the harness changed between drafts. If reads are authorized from the token alone, removal calls don't matter, and the TTL is the only built-in lever. If the token's expiry is idle-based, so each request pushes it out, then a revoked client that keeps making requests never lets the token expire, and the TTL stops being a bound at all. The broker's re-check at refresh never runs, because refresh never happens. The last control available is a server-side token kill, if the platform offers one. Results for a token-only platform (hardened TTL 60, naive TTL 3600; "never ends" means access persisted to the end of the simulated horizon): Expiry model, admin APINaiveHardened, no token killHardened + token killFixed TTL, healthy3600 s60 s0 sFixed TTL, 600 s outage3600 s60 s60 sIdle expiry, healthynever endsnever ends0 sIdle expiry, 600 s outagenever endsnever ends601 s Read the third row twice. With idle expiry and no kill, the short TTL and the broker re-check buy nothing against a client that keeps talking. The only controls that end access are live-membership enforcement on the platform or an invalidation you can trigger from the server. So the most valuable hour you can spend before integrating a platform is finding out which of those you have. At TTL 60, each active user generates 60 token mints per hour. Whether that is trivial depends on your user count and the platform's rate limits, which the harness does not model. The AI Participant: Confused Deputy, Not “Authorization for Bots” A multi-party conversation is a prompt authored by several principals with different privileges. The agent reading it holds one set of tool credentials. Any principal who can put text in the thread can attempt to spend authority they don't have. This is the classic confused deputy, with the deputy now parsing natural language. The harness has a dialog with a patient and a clinician, and three tools: get_my_record (low privilege), approve_refill (clinician only), and read_history. The proxy runs in three configurations: naive (the agent's standing credential decides everything), role-check (the call executes with the authority of whoever authored the triggering message), and hardened (role check plus parameter-bound approval, epoch fencing, and history clamping). AttackNaiveRole-check onlyHardenedA1 Patient's message steers agent into approve_refillsucceedsblockedblockedA2 Clinician's benign turn; patient's injected text sits in contextsucceedssucceedsblockedA3 Approval for rx42 reused to approve rx99succeedssucceedsblockedA4 One approval consumed twicesucceedssucceedsblockedA5 In-flight agent call executes after human takeoversucceedssucceedsblockedA6 New agent turn minted after takeoversucceedssucceedsblockedA7 Agent reads history outside session scopesucceedssucceedsblockedB1 Clinician approves refill through explicit approvalworksworksworksB2 Patient reads own record through agentworksworksworks The B rows make sure the hardened column isn't green merely because it denies everything. A2 is the finding that changed my plan. My pre-run design was authority as the intersection of the acting principals' rights. Role-check is that rule, and it stops the obvious attack (A1). It fails A2 completely. The clinician sends "please summarize the thread." The agent's capability is legitimately the clinician's. The patient's earlier message, still in context, asks for a refill approval, and the agent obliges, spending authority that was never granted for that purpose. Intersection scopes who the agent acts for. It says nothing about what that authority was meant to cover. The fix is to stop treating conversation turns as grants for privileged operations. A privileged call needs an approval bound to the exact call, issued out-of-band by an authenticated UI session of a privileged human: def _approval_ok(self, tool, args, ap): b = ap["body"] good = hmac.new(APP_KEY, json.dumps(b, sort_keys=True).encode(), hashlib.sha256).hexdigest() if not hmac.compare_digest(good, ap["mac"]): return False if (b["tool"], b["args"], b["epoch"]) != (tool, args, self.epoch): return False if self.clock.t > b["exp"] or b["nonce"] in self.used: return False if self.users.get(b["principal"]) != "clinician": return False self.used.add(b["nonce"]); return True The approval binds tool, arguments, epoch, expiry, and a single-use nonce. A3 fails because the arguments differ, and A4 because the nonce is spent. Injected text can still request the call. It cannot carry the authority to make it. The system doesn't try to detect injection. It makes sure injected text can't hold a capability. Fenced handoff (A5, A6). When a human takes over, the app increments a per-dialog epoch. Every capability carries the epoch it was minted under, and the tool proxy rejects stale ones. A5 is the race: a turn starts at t=0, the human takes over at t=5, and the agent's call arrives at t=8. Without the fence it executes. This is a fencing token, and it has a fencing token's limit: it only works if the resource performing the effect checks the epoch. If the agent can also post through a path that bypasses your proxy, such as the platform's own API with the agent's own credentials, the fence is decorative. The platform-side complement is removing the agent from the dialog at handoff, which is another admin call, subject to the same outage window measured above, and which not every dialog type allows (see below). History scope (A7). The capability carries the index where the current session begins, and read_history refuses anything earlier. The harness checks the proxy's enforcement. It does not check that the agent's context assembly respects the boundary, which is separate code you would need to test. The audit log, and what a hash chain does not do The hardened path records each agent tool call (allowed or denied) and each handoff in a hash-chained log with actor type, model version, tool, arguments, result, and principal: Clean chain, verified against an externally stored head hash: passes.Row modified without recomputing the chain: detected.Row modified and the chain recomputed from that row forward (an attacker with write access): verifies clean unless you compare against the externally stored head, in which case it is detected. The chain alone defends against accidental corruption and clumsy tampering. It does not defend against someone who can write the store and knows the scheme. What does is anchoring the head hash somewhere that an attacker can't write: a separate account, an append-only object store, a periodic signed checkpoint. If your threat model includes a malicious insider with database access, the anchor is the control, and the chain is the mechanism that makes anchoring cheap. On compliance: in a regulated setting, "who did what, when, under whose authority" is what this log makes answerable. The HIPAA Security Rule's audit-control standard is 45 CFR §164.312(b) (cited from memory; confirm before relying on it). A platform's compliance posture does not discharge your obligations, and a log design is one input to them, not compliance itself. A Documentation Audit: The Seam on Quickblox Everything above is simulation. This section asks the checklist questions of a real platform using its public developer documentation, retrieved on 4 October 2026. Nothing here was tested against a live QuickBlox account. Documentation can be incomplete or out of date, and where I could not find an answer, I say so instead of guessing. Sources are listed at the end. QuestionWhat the documentation saysConsequence for the seam1. Are server-to-app events authenticated?I found no first-party documentation of outbound server-to-server event webhooks. A competitor's comparison page asserts QuickBlox doesn't support webhooks. That is a competitor's claim and I haven't confirmed it.If there is no signed event feed, the forged/replay/race tests apply to whatever you build instead. The rule that survives is to never let your backend act on events reported by clients, because client-reported events are attacker-controlled. If you poll, poll with a server credential and treat the response as the source.2. Does a read check live membership, or only the token?Not stated in the pages I read.Unknown. This decides which row of the revocation tables applies. It is a two-minute sandbox test: remove an occupant, then read with that user's still-valid token.3. Token lifetimeBoth user and application session tokens expire 2 hours after the last REST request, and each REST response carries an expiry header. I found no configuration for it.This is idle expiry, which is the dangerous row of the table above. A client that keeps making REST calls keeps the session alive. I don't know whether live chat-connection activity also extends it, and I haven't verified an absolute cap.4. How is membership removed?Removal goes through the dialog update call. A dialog's owner can remove any occupant (the REST reference says admins can too), and an occupant can remove only themselves. The SDK and dashboard docs show removing other users working for group dialogs, not for private (one-to-one) or public ones. Deleting a user removes them from group dialogs but not from private dialogs' occupant lists.Private dialogs have no documented remove-another-user operation. If revocation matters, a clinician-patient conversation modeled as a private dialog can't be revoked by removal. You'd have to delete the dialog for everyone (which needs the owner and loses history). Model sensitive conversations as group dialogs with an app-controlled owner. This is a design decision to make before launch, not after.5. Immutable IDs, self-registrationUser creation with an application session is documented as acceptable for prototypes. For production the docs recommend creating users from your backend with the API key and disabling application-session user creation. Users carry an external_user_id field intended to link to an outside system, and login, email and external ID must be unique.The identity-confusion attack depends on that production setting. Bind your app user to the platform's own user ID, written only by your backend, and don't resolve senders by login or email. external_user_id is a convenient mapping field, but the binding should still be enforced on your side.6. Can the agent be a distinct principal?Dialogs have owner, admin, and occupant roles.Make the agent a separate user and an occupant, never an admin or owner. To fence it at handoff by removing it, the dialog must be a group dialog, per row 4. I did not verify whether the agent's session can be invalidated server-side.7. Can you reconcile after a missed event?Dialog and message REST endpoints exist, and the update and delete references show them taking an API key. I did not verify listing and filter semantics for drift detection.Plan the reconciler as a poll of dialogs and occupants compared with your database. Confirm pagination and filter support in a sandbox first. Three observations from the table, none of them about vendor quality. Every platform has choices like these, and every one is a place the seam can fail: The documented 2-hour idle expiry changes the revocation answer. In the harness's token-only, idle-expiry rows, short TTLs and broker re-checks buy nothing, and the only thing that ends access is membership enforcement or a server-side kill. Whether QuickBlox enforces membership at read time is the first thing to test, and whether you can invalidate a session from the server is the second.The private-dialog limitation is the most actionable finding. It is a design constraint you can only avoid by choosing group dialogs up front.The absence of a documented signed event feed moves work to you. The ingress hardening in this article becomes a design for whatever sync mechanism replaces it. On the vendor's compliance statements: QuickBlox's own pages state that it offers HIPAA-ready hosting with a signed BAA, and that it is SOC 2 certified. I haven't verified those claims, and a BAA does not make your application compliant. The seam problems in this article are exactly the part a BAA doesn't cover. Cost and trade-offs Ingress overhead. Hardened ingress costs about 8.8 µs per event against 2.0 µs for the naive handler, in-process, on one machine (Python 3.12). Indicative only. In production, the dominant cost is one atomic store write per event.Token churn. Short TTLs convert revocation latency into mint load: 60 mints per user-hour at TTL 60, 120 at TTL 30. And if expiry is idle-based, TTL tuning may not be available to you at all.Dependence on the admin API. Forced removal and token kill are only as available as that API. The outbox hides outages from the revoking caller. It doesn't shorten them.Approval friction. Every privileged action needs a human click bound to specific arguments. The agent can't autonomously perform privileged operations. That is the intended outcome, and it will frustrate some product teams. Mitigate it with a good approval surface: show the exact arguments, make approval one tap, and expire quickly.Complexity. The hardened design adds a broker, an outbox, an approval service, a per-dialog epoch, and an anchored log. For an internal tool with a few trusted users, it is probably more than you need. Limitations of this work The simulator is simple: one dialog, instantaneous admin success when up, no clock skew, no partial failures inside the platform, a single-node claim store.The worst-case agent models the threat, not any LLM.I did not implement or test a membership reconciler, although I recommend one.The QuickBlox audit is documentation-only. Docs may be incomplete, inconsistent (the owner-versus-admin removal rule is stated differently in different pages), or out of date. Every "unknown" in the table is a sandbox test I haven't run.Remaining exposure none of this addresses: injected text can still influence unprivileged tools (what the agent says, what it summarizes), exfiltration through in-scope reads, and approval fatigue.Nothing here shows that any real platform is vulnerable to any attack in the harness. Conclusion The integration boundary is a consistency boundary, and the interesting failures are state disagreements and authority crossing it unchecked. The revocation window gives you a number to argue about, instead of a feeling, and the number depends on three platform properties you can look up before writing code: whether reads check live membership, whether token expiry is fixed or idle-based, and whether you can remove a member or kill a session from the server. For the AI participant, the real question is whose authority a tool call exercises. Intersecting principals' rights is necessary but not sufficient. A privileged call needs an approval bound to its exact arguments, or conversation text will eventually carry authority it shouldn't. Sources (QuickBlox documentation, retrieved 4 October 2026): Key Concepts: https://docs.quickblox.com/docs/key-conceptsAuthentication (JS): https://docs.quickblox.com/sdks/js-authenticationUpdate Dialog: https://docs.quickblox.com/reference/update-dialogiOS Dialogs: https://docs.quickblox.com/sdks/ios-chat-dialogsDashboard, Chat: https://docs.quickblox.com/docs/chat-dashboardDashboard, Users: https://docs.quickblox.com/docs/users-dashboardCreate User: https://docs.quickblox.com/reference/create-userDelete Dialog: https://docs.quickblox.com/reference/delete-dialog

By Igboanugo David Ugochukwu CORE

The enterprise paradigm for artificial intelligence has fractured. While massive cloud-hosted large language models continue to dominate consumer applications, highly regulated sectors like defense, aerospace, and high-tier finance are actively blocking external API calls to public AI services due to strict data sovereignty mandates. The solution is migrating toward small language models (SLMs) localized at the tactical edge. Deploying an SLM into an air-gapped, "sovereign cloud in a box" environment requires absolute cryptographic certainty over the supply chain. A compromised base image running deep inside a disconnected network invalidates the entire security perimeter. The recent release of Docker Hardened Images (DHI) and the expansion of Google Distributed Cloud (GDC) air-gapped appliances give backend engineers the exact toolchain required to host models like Google's Gemma with zero-trust enforcement. This technical guide details an end-to-end framework for curating a zero-CVE container, packing a fine-tuned Gemma SLM, and pushing it to an isolated GDC cluster. The Target Infrastructure: GDC Air-Gapped Google Distributed Cloud air-gapped is an integrated hardware and software platform designed to operate physically disconnected from the internet. It allows enterprises to deploy container-based workloads and managed services like Vertex AI in completely isolated environments. The architecture supports rigorous compliance frameworks, including NIST, ISO 27001, and SOC II. Notably, GDC offers packaged generative AI solutions enabled by Gemma, providing local, conversational search capabilities on-premises. However, when deploying custom fine-tuned checkpoints or building the Java microservices that handle backend inference orchestration- a core focus of my daily work as an IEEE Senior Member and backend developer — we cannot rely on standard public container registries. The container must be hardened from the base layer up. Base Image Curation With Docker DHI To build our localized SLM container, we start with Docker Hardened Images. DHI provides minimal, secure, production-ready container images maintained by Docker. Rather than blindly pulling a generic Alpine or Debian base image and running a vulnerability scanner after the fact, DHI guarantees a hardened foundation. Using the docker dhi CLI, available in Docker Desktop 4.65 and later, we can inspect and mirror these hardened artifacts directly to our enterprise registries. Shell # Verify local DHI CLI installation docker dhi --help # Mirror a hardened Python or Java base image to your local org docker dhi mirror start --org enterprise-registry \ dhi/python,enterprise-registry/dhi-python By switching our Dockerfile FROM instruction to enterprise-registry/dhi-python, we structurally eliminate the attack surface before installing our inference engine dependencies. Generating the SLM Artifact and SBOM As I detailed in my Springer book chapter Introduction to Trustworthy AI, accountability in machine learning relies heavily on technical robustness and traceability. We cannot deploy a black-box container into an air-gapped network. We must generate a cryptographic Software Bill of Materials (SBOM) that catalogs the exact SLM weight hashes, the inference engine binaries, and the OS packages. The DHI CLI handles this transparently. Once you package your Gemma model weights into the hardened container, you can retrieve the exact attestations and SPDX SBOMs attached to the base images. Shell # Retrieve the SLSA provenance and SPDX SBOM for audit docker dhi attestation get dhi/python:3.11 sha256:<digest> -o provenance.json This JSON payload is ingested into the local CI/CD pipeline. The pipeline validates that no unauthorized dependencies were injected during the model packaging phase. If the cryptographic signature fails, the build halts immediately. Deploying to the Sovereign Edge With a cryptographically signed, DHI-backed container housing our Gemma SLM, we transition to the deployment phase on GDC air-gapped. Because GDC air-gapped operates a single cluster encompassing bare metal nodes without internet access, deployment relies on a physical handoff. The hardened container image, the validated SBOM, and the deployment manifests are loaded onto a portable storage device. Once transferred to the secure facility, the infrastructure operator pushes the image to the local, disconnected GDC container registry. GKE on GDC orchestrates the container workload. We apply Kubernetes network policies to isolate the SLM namespace, ensuring that the model can only be queried by authorized internal Java backend services. By enforcing kernel isolation at the container level and physical isolation at the hardware level, we construct a true defense-in-depth architecture. The Gemma model processes highly classified proprietary data while satisfying the strictest data sovereignty requirements. The transition from cloud-first to sovereign-edge AI marks a critical maturation in software engineering. Trustworthy intelligence demands predictable execution boundaries. By merging the zero-trust foundations of Docker Hardened Images with the physical isolation of Google Distributed Cloud, engineering teams can deliver state-of-the-art reasoning capabilities directly to the tactical edge without compromising a single byte of sensitive data.

If you've worked with a relational database for any length of time, you know the drill: state lives in a row, and every UPDATE overwrites whatever was there before. It's simple, it's fast to reason about, and it throws away history by design. Update a bank balance from $1,000 to $900, and the fact that a withdrawal happened and why is gone unless some separate audit log happened to catch it. Event sourcing flips that model on its head. Instead of storing current state, you store the sequence of events that produced it, and you derive state on demand by replaying the log. History isn't a side effect you bolt on afterward; it is the write path. This isn't a niche idea anymore. Netflix runs event-sourced pipelines that process over a billion viewer events a day for personalization and health monitoring. Uber pushes millions of events per minute ride requests, location updates, pricing signals through Kafka-backed streams with exactly-once semantics for fraud detection. LinkedIn built Kafka in the first place to serve as an append-only log for activity data. Slack pipes billions of daily messages through event-driven pipelines that carry full metadata so services don't have to make synchronous calls to each other. What's harder to find is a clear-eyed look at what actually breaks in production and how teams fix it. That's what this article is about: the data model, the CQRS split, the consistency trade-offs, and the part that tends to get glossed over: how to keep read models correct when the same event might get delivered twice, three services need to stay in sync, and any one of them might be down when the message arrives. Events as First-Class State In a CRUD system, a withdrawal looks like this: SQL -- Traditional CRUD (destructive) UPDATE accounts SET balance = 900 WHERE id = 123; In an event-sourced system, nothing gets overwritten. You append a new fact: SQL -- Event sourcing (additive) INSERT INTO events (stream_id, event_type, payload, version) VALUES ('account-123', 'MoneyWithdrawn', '{"amount": 100}', 42); Current state is reconstructed by folding over every event in a stream, in version order. It's a pure function — same events in, same state out, and it maps directly onto a left-fold/reduce if you've ever written one: Python def reconstruct_balance(account_id): events = db.query( 'SELECT * FROM events WHERE stream_id = %s ORDER BY version', [f'account-{account_id}'] ) state = {'balance': 0} for event in events: if event.type == 'AccountOpened': state['balance'] = 0 elif event.type == 'MoneyDeposited': state['balance'] += event.payload['amount'] elif event.type == 'MoneyWithdrawn': state['balance'] -= event.payload['amount'] return state['balance'] The Event Store Schema The event store itself is a boring relational table with one architectural rule bolted on: nothing is ever updated or deleted. SQL CREATE TABLE events ( id BIGSERIAL PRIMARY KEY, stream_id UUID NOT NULL, event_type VARCHAR NOT NULL, payload JSONB NOT NULL, version INT NOT NULL, occurred_at TIMESTAMPTZ NOT NULL DEFAULT now(), UNIQUE (stream_id, version) ); That UNIQUE (stream_id, version) constraint is doing more work than it looks like. It's your optimistic concurrency control: if two processes both try to append version 5 to the same stream, the second one gets a unique-constraint violation instead of silently clobbering the first. It's compare-and-swap at the stream level, for free. And to be clear — this table can live in Postgres, MySQL, Oracle, whatever you're already running. Event sourcing is a write strategy, not a database product. Don't let anyone tell you that you need a specialized event store to do this. Snapshots Replaying thousands of events every time you want to read an entity's state gets expensive fast. The standard fix is snapshotting: periodically materialize current state and store it next to the log. On read, load the latest snapshot and replay only what's happened since. SQL SELECT * FROM snapshots WHERE stream_id = 'account-123' ORDER BY version DESC LIMIT 1; -- Then replay only events after snapshot.version SELECT * FROM events WHERE stream_id = 'account-123' AND version > :snapshot_version ORDER BY version; The CQRS Boundary In practice, event sourcing almost always shows up paired with CQRS (Command Query Responsibility Segregation) — splitting the write path (commands that produce events) from the read path (queries against a denormalized read model). This isn't just architectural tidiness. The event log is optimized for append throughput and sequential replay; it's a bad fit for ad-hoc queries, joins, sorting, or full-text search. ComponentRoleTypical technologyCommand handlerValidates commands, appends eventsApplication service layerEvent storeAppend-only source of truthPostgreSQL, EventStoreDB, KafkaProjectionReads events, builds the read modelBackground worker, Kafka consumerRead modelDenormalized query targetPostgreSQL, Redis, ElasticsearchQuery APIServes reads to clientsREST / GraphQL Consistency Models and the Lag Problem Projection Lag Is Real, Design For It Because the projection runs asynchronously, there's a window between an event landing in the store and the read model reflecting it. A user withdraws money and immediately checks their balance and might see the stale number. This is the central consistency problem in any CQRS/event-sourcing setup, and it needs an explicit answer, not a hope that it won't matter. In healthy systems on Kafka or a comparable bus, projection lag typically sits around 50–300ms. Under load spikes, partitions, or consumer restarts, it can stretch to seconds or minutes. Your design needs to be correct under the bad case, not just the median. Pick Your Battles: Selective Consistency The real design question isn't "How do I eliminate eventual consistency?" It's "Where do I actually need strong consistency, and where can I let the lag ride?" Those two buckets look pretty different: Data typeConsistency requiredWhyAccount balance post-withdrawalStrongFinancial correctnessPayment confirmationStrongRegulatory, user expectationRecommendation feedEventualA few seconds of lag is invisibleNotification badge countEventualApproximate is fineProduct inventory on a listing pageEventualMinor lag is tolerableSearch index for a new entityEventualUsers already expect crawl delay Read-Your-Own-Writes For the strong-consistency bucket, the most common mechanism is read-your-own-writes (RYOW). The command API hands back the version number it just appended; the client sends that version back on its next read; and the query service refuses to answer until the read model has caught up. Markdown # 1. Command returns the appended event version POST /accounts/123/withdraw {amount: 100} → HTTP 200 {status: 'ok', event_version: 42} # 2. Client sends version as constraint on next read GET /accounts/123/balance Headers: X-Min-Version: 42 # 3. Query service checks projection position SELECT last_processed_version FROM projection_checkpoints WHERE name = 'balance_projection'; -- If < 42, wait up to 500ms then retry or return 202 Accepted When You Just Bypass the Read Model For a small set of high-stakes flows such as payment confirmation screens, dispute resolution, regulatory audits, plenty of systems skip the read model entirely and rebuild state straight from the event log. It's always consistent, at the cost of latency. Snapshots (see above) keep that latency bounded even for entities with long histories. Three Ways to Keep Projections in Sync The projection needs to process each event exactly once and update the read model atomically with its own checkpoint. In practice, that comes down to three strategies, each suited to a different infrastructure setup. 1. Same-Database Transaction The simplest option: put the read model and the checkpoint table in the same relational database, and wrap the update and the checkpoint advance in one transaction. SQL BEGIN; UPDATE account_balances SET balance = 900 WHERE account_id = 123; UPDATE projection_checkpoints SET last_event_id = 42 WHERE name = 'balance_projection'; COMMIT; Either both writes land or neither does, which kills the two classic failure modes: processing an event without advancing the checkpoint (you'll reprocess it on restart) and advancing the checkpoint without writing the data (permanent, silent data loss). Great when your whole read model fits in one database. 2. Kafka + Idempotent Consumers Once Kafka is your event bus, consumer offsets live inside Kafka and can't be committed in the same transaction as your read-model write. Kafka only promises at-least-once delivery, so on a consumer restart, some events may show up twice. The fix is to make the consumer idempotent, in which case it's safe to run twice on the same event. SQL INSERT INTO account_balances (account_id, balance, last_event_id) VALUES (%s, %s, %s) ON CONFLICT (account_id) DO UPDATE SET balance = EXCLUDED.balance, last_event_id = EXCLUDED.last_event_id WHERE account_balances.last_event_id < EXCLUDED.last_event_id; That WHERE clause is the whole trick. If event version 42 gets replayed after it's already been applied, 42 < 42 is false and the update is a no-op. ScenarioDB state beforeIncoming eventConditionResultNormal processinglast_event_id = 41v = 4241 < 42 → TRUEUpdate appliedCrash replay (duplicate)last_event_id = 42v = 4242 < 42 → FALSESkipped safelyOut-of-order deliverylast_event_id = 42v = 4142 < 41 → FALSEStale event ignored The INSERT branch only ever fires once, when the entity's stream first shows up. Every event after that takes the ON CONFLICT path, and the version guard is what makes it idempotent. 3. The Transactional Outbox Pattern Things get harder when a projection needs to fan out to several different stores at once, say a relational read model, an Elasticsearch index, a Redis cache, and a webhook receiver. No single database transaction can span all of those. This is where the transactional outbox pattern earns its keep. The projection writes its read-model update and an outbox row in one local transaction. A separate relay process picks up pending outbox rows and delivers them to each downstream target, retrying until each one confirms. Because the outbox write is atomic with the read-model write, the relay always knows exactly what still needs to go out. SQL -- Atomic: both rows commit or neither does BEGIN; UPDATE account_balances SET balance = 900 WHERE account_id = 123; INSERT INTO outbox (id, payload, es_sent_at, redis_sent_at, webhook_sent_at) VALUES (gen_random_uuid(), '{"account_id": 123, "balance": 900, "event_version": 42}', NULL, NULL, NULL); COMMIT; Making the Outbox Relay Fault-Tolerant The relay delivers to each target independently and tracks confirmation per target. The scenario worth designing for: the relay writes successfully to target A, then crashes before marking the row delivered, so on restart, target A gets the same write again. Every target needs to handle that gracefully. Elasticsearch: Let the Document ID Do the Work Elasticsearch indexing is idempotent as long as you supply a stable document ID yourself. Index the same ID twice and you just overwrite the first doc with identical content, having no duplicates. Python # Non-idempotent: ES generates a random ID es.index(index='accounts', body={'balance': 900}) # → Two calls produce two documents # Idempotent: stable ID derived from the outbox row es.index( index='accounts', id=outbox_row.id, # same on every retry body={'balance': 900, 'event_version': 42} ) Redis: Absolute Values, Not Deltas SET in Redis is naturally idempotent — setting a key to the same value twice is the same as setting it once. The catch: your relay has to write absolute values, never deltas. INCRBY and LPUSH are not idempotent and have no business being in a relay path. Python # Non-idempotent: applying a delta twice corrupts state redis.incrby('account:123:balance', -100) # Idempotent: writing the derived absolute value redis.set('account:123:balance', 900) Webhooks: Idempotency Keys External webhook receivers are the trickiest case, because you don't control their implementation. The standard answer is the one Stripe documents publicly: an Idempotency-Key header. The relay sends the outbox row's UUID as that key on every attempt; the receiver keeps a table of keys it's already seen and just replays the original response for a repeat. Getting this right on the receiver side means handling three edge cases: Concurrent duplicate requests: Two requests with the same key arrive before either has been saved. Fix it with an atomic INSERT under a unique constraint; the second insert fails, and that thread waits on the first.Key reuse with a different body: Same key, different payload. Reject with a 422 after comparing a hash of the stored body to the incoming one.Key expiry: Keys can't live forever. Stripe uses a 24-hour TTL, which means your relay's retry window (and dead-lettering) has to finish well inside that boundary. The key record and the business-logic effect have to commit in the same transaction. If the business logic runs but the key never gets saved, a retry re-executes the logic and that's not a missed optimization, that's data corruption. Python def handle_request(request): key = request.headers['Idempotency-Key'] req_hash = sha256(request.body) with db.transaction(): row = db.execute(''' INSERT INTO idempotency_keys (key, request_hash, created_at, expires_at) VALUES (%s, %s, now(), now() + interval '24 hours') ON CONFLICT (key) DO UPDATE SET key = EXCLUDED.key RETURNING key, request_hash, response_status, response_body ''', [key, req_hash]).fetchone() if row.request_hash != req_hash: return 422, 'Key reused with different request body' if row.response_status is not None: return row.response_status, row.response_body result = run_business_logic(request) db.execute(''' UPDATE idempotency_keys SET response_status = %s, response_body = %s WHERE key = %s ''', [result.status, result.body, key]) return result.status, result.body Dead-Letter Escalation Transient failures — timeouts, a target being briefly down — get exponential backoff with per-target retry counters on the outbox row. Permanent failures — a schema mismatch, a decommissioned endpoint — shouldn't be allowed to block the relay forever. After a configurable number of attempts (10 is a common default), move the row to a dead-letter table for someone to look at. Who's Actually Running This in Production CompanyEvent volumePatternOutcomeNetflix>1B events/dayEvent sourcing + CQRS on KafkaPersonalization and monitoring for 260M+ subscribersUberMillions/minuteKafka + Flink, exactly-once semanticsFraud detection and dynamic pricing at global scaleLinkedInOrigin of Kafka (2010)Append-only activity stream logUnified data pipeline, replacing point-to-point integrationsSlackBillions of messages/dayKafka event propagation with full metadataDecoupled delivery and indexing under heavy loadStripeDocumented idempotency-key APITransactional idempotency at the webhook layerFormally specified 24-hour idempotency window with body-hash validation Takeaways A few things worth internalizing if you're building (or maintaining) one of these systems: Eventual consistency is a contract, not a bug. RYOW turns the consistency requirement into an explicit versioned header instead of an assumption. Idempotency keys turn "don't double-process this" into a first-class API contract. The outbox pattern turns delivery into a durable, queryable table instead of a fire-and-forget call. Once it's explicit, you can test it and monitor it — that's the whole point.Idempotency is the load-bearing primitive. It's standing in for distributed transactions, which are either unavailable, too slow, or too fragile for high-throughput event-driven systems. At-least-once delivery plus idempotent receivers gets you effectively-once behavior without any distributed coordination.The pattern underneath is always the same: a unique key derived from something stable (an event version, an outbox row's ID), stored atomically next to the effect it protects, checked before that effect is applied, and expired on a TTL that bounds how long you need to keep retrying.None of this locks you into specific technology. Event store in Postgres, EventStoreDB, or Kafka's compacted log — doesn't matter. Read model in MySQL, Redis, or Elasticsearch — doesn't matter. The outbox just needs to live in the same ACID store as the read model it's paired with. That's a low bar, and it's why teams can adopt this incrementally next to whatever they're already running. The hard part of event sourcing was never the event log where appending immutable rows is the easy 20%. The real engineering work is keeping derived state correct across multiple heterogeneous targets when any one of them can fail at any moment. Idempotency, applied consistently, is what makes that tractable without reaching for distributed transactions.

GitHub's contribution calendar gives you an already iconic overview of your activity over the year. Each day has a square, and its color changes with the contribution level. And of course, those days and contributions can also be arranged into buildings, with a week becoming a small part of a city and busier weeks adding height to the skyline. I had some spare minutes and built Quarkus Contribution City around this idea, using Java and Quarkus to generate an SVG from GitHub's contribution data. The visual inspiration came from Pink Pixel's skyline, and you can see a version with my real data on my GitHub profile. A scheduled GitHub Action updates the image at night. For a Java developer, this is a small project you can follow from the API request through to the image. You can use the released action on your own profile, then change the drawing code locally once you have a feel for how the city is put together. For this article, I created a special DZone example that adds a small piece of character art above the buildings. Add the City to Your Profile Your GitHub profile README lives in a public repository with the same name as your account. For my account, that repository is myfear/myfear. Open your own profile repository, or create it with a README if you haven't used this GitHub feature before. Copy the project's complete publishing workflow into .github/workflows/contribution-city.yml on the default branch. This example checks out your profile repository and the action's v1 release line. It then generates the image and publishes it to an output branch. The workflow includes the repository write permission needed for that last step. The generation step uses the repository owner's username, so the same configuration can be copied between personal profile repositories. You can add the layout options to that step: - name: Generate contribution city id: city uses: ./.city-action with: github-token: ${{ secrets.CONTRIBUTION_TOKEN || secrets.GITHUB_TOKEN } username: ${{ github.repository_owner } weeks: '53' height: '14' theme: github-dark The local path refers to the action checkout made earlier in the example workflow. The built-in GITHUB_TOKEN is the starting point for reading your visible contribution data. An optional CONTRIBUTION_TOKEN secret can supply a personal token when additional access is needed. The calendar returned by GitHub depends on the token's permissions and your account's contribution visibility settings. After committing the workflow, open Actions in your profile repository, select publish-contribution-city, and choose Run workflow. A successful run creates contribution-city.svg on the output branch. Add this image reference to your profile README, replacing both occurrences of USERNAME:  The example also has a daily schedule for 03:17 UTC. Subsequent runs create a commit when the generated image changes. For this setup, the workflow and its run history can be found in your profile repository, while the Java implementation is maintained in the action's repository. Give Each Week a Building The city is arranged in time order, with the most recent week on the right. Every building is two character cells wide, so all weeks occupy the same horizontal space. Weekly contribution totals determine the height, and daily contribution levels provide the window colors. Month labels beneath the street help you find your place in the year. Height needs a little extra thought. A very busy week can have many times the contributions of a quiet one. With a linear scale, those quieter weeks would be compressed close to the street. The city uses a logarithmic scale, which gives them more room while preserving the order of weekly totals. For each week, the calculation can be expressed as: int height = weeklyTotal == 0 ? 1 : 1 + (int) Math.round( Math.log1p(weeklyTotal) / Math.log1p(maximumWeeklyTotal) * (maxHeight - 1)); With a maximum of 20 contributions and a height limit of 14, a week with three contributions gets seven window rows. The busiest week gets 14, and a week with zero contributions keeps one row. Quiet periods therefore retain their position in the calendar. An antenna marks the busiest displayed week; when totals are tied, the most recent week receives it. The two columns of windows are filled by cycling through the days of the week. This repeats the daily pattern in taller buildings, so window count is a visual texture. The weekly total controls height, while the window colors carry the daily contribution levels. Together, they give you a view of the pattern across your calendar. You can change weeks to 12 for a shorter view. The tallest building is then calculated from those recent weeks, which lets the selected period use the available height. Keep this in mind when comparing images: the same week can appear taller in a shorter view. The displayed contribution total still covers the full calendar returned by GitHub. Follow the Data Through Java The released project uses Quarkus with the Quarkus GitHub Action extension. That extension connects an @Action method to workflow inputs, outputs, and the job summary. Inside the method, the drawing process comes down to three steps: ContributionCalendar calendar = contributionClient.fetch( configuration.username(), inputs.getGitHubToken().orElseThrow(), context.getGithubGraphQLUrl()); Scene scene = cityBuilder.build(calendar, configuration.options()); String svg = svgRenderer.render(scene, configuration.options().theme()); The client reads the contribution calendar with one GraphQL request and converts the response into Java records for the calendar, weeks, and days. CityBuilder uses these records to place roofs, windows, and the street in a grid. SvgRenderer turns that grid into the final SVG. Changes to the skyline can therefore be made in plain Java classes, with the GitHub connection kept in the client. The SVG contains the shapes and text needed to display the city. Adjacent characters with the same color are grouped during rendering, and their horizontal positions are set explicitly to keep the columns aligned. Identical data and options produce identical file contents, which allows the publisher to compare images before creating a commit. The action sets up Java and JBang, builds the selected source revision, and runs the packaged application. Change the Layout Locally To try a change, install JDK 21 and JBang, with Git and Bash available. The repository includes a Maven wrapper. JBang is also used by the integration tests to run the packaged action through the same launcher used in GitHub Actions. git clone --branch v1.0.0 https://github.com/myfear/quarkus-contribution-city.git cd quarkus-contribution-city ./mvnw verify The tests use local test data and a local GraphQL server, so you can run them without GitHub credentials. They also generate SVG previews under target/previews. Open github-dark.svg, recent-12.svg, and empty.svg in a browser to compare a full calendar, a shorter period, and a calendar with zero contributions. Start with the height calculation in CityBuilder, then run the same command again and compare the previews. The saved data stays the same between runs, which makes the effect of your change easier to see. The project is available under the Apache License 2.0, and the v1.0.0 release gives you a fixed starting point. Give the City a DZone Landmark Version 1.0.0 exposes weeks, height, and the two built-in themes through workflow inputs. For a custom shape, you extend the Java code in your fork. The Scene record gives you a grid of cells to change before the SVG is rendered, and that is where we'll add the DZone lettering. Each cell has a character, a role such as roof or window, and the color information used by the renderer. A character mask is enough to describe a new shape: # marks a filled block, and spaces leave the background visible. By arranging those blocks, you can draw letters, an arch, or a stepped tower. For this example, the lettering is placed above the weekly buildings. It has its own fixed shape, while the buildings below continue to show the contribution calendar. The image uses a snapshot of my real account data, with 534 contributions returned by GitHub on October 5, 2026, when I wrote this article. A DZone-inspired palette and decorative lettering added to my contribution city. The weekly building heights and daily window patterns come from the same calendar used by the regular view. Add this class to the existing city package in your fork, at src/main/java/com/themainthread/contributioncity/city/DZoneLandmark.java: package com.themainthread.contributioncity.city; import java.util.ArrayList; import java.util.Collections; import java.util.List; import com.themainthread.contributioncity.render.Theme; public final class DZoneLandmark { public static final Theme THEME = new Theme("dzone-example", "#161b22", "#242c36", "#303947", "#f58a42", "#9ba9ba", "#edf2f7", List.of("#536070", "#86431f", "#b85b25", "#e57d36", "#ffb365")); private static final List<String> MASK = List.of( "#### ##### ### # # #####", "# # # # # ## # # ", "# # # # # ## # # ", "# # # # # # # # #### ", "# # # # # # ## # ", "# # # # # # ## # ", "#### ##### ### # # #####"); private DZoneLandmark() { } public static Scene addTo(Scene city) { int landmarkWidth = MASK.getFirst().length() * 2; int width = Math.max(city.width(), landmarkWidth); int left = (width - landmarkWidth) / 2; List<List<Cell>> rows = new ArrayList<>(); Cell block = new Cell('█', CellKind.ROOF, 0, -1); for (String line : MASK) { List<Cell> row = new ArrayList<>(Collections.nCopies(width, Cell.SKY)); for (int column = 0; column < line.length(); column++) { if (line.charAt(column) == '#') { row.set(left + column * 2, block); row.set(left + column * 2 + 1, block); } } rows.add(row); } rows.add(Collections.nCopies(width, Cell.SKY)); for (List<Cell> sourceRow : city.rows()) { List<Cell> row = new ArrayList<>(sourceRow); row.addAll(Collections.nCopies(width - city.width(), Cell.SKY)); rows.add(row); } return new Scene(city.username(), city.totalContributions(), rows); } } The theme defines the background, two building shades, roof, street, and label colors, followed by the five window colors. Those window colors follow GitHub's contribution levels from zero through four. Here, the roof color is also used for the lettering, so the decoration has a consistent orange color across all seven rows. Inside addTo, each marked position is expanded to two cells because the renderer's cells are taller than they are wide. The lettering is centered, a blank row is added beneath it, and the original city rows are copied below. Short calendars receive extra background space on the right to make room for the lettering. The dates, counts, and order of the weekly buildings are preserved. To use the decoration in your fork, import DZoneLandmark from the city package in ContributionCityAction. Replace the existing scene-building and rendering lines with: Scene scene = cityBuilder.build(calendar, configuration.options()); Scene decorated = DZoneLandmark.addTo(scene); String svg = svgRenderer.render(decorated, DZoneLandmark.THEME); This selects the example's palette in Java. The published @v1 action continues to accept github-dark and mono; your fork contains the additional drawing code. For a local preview, add the following after the existing Files.writeString call in PreviewTest.write, with the same import for DZoneLandmark: Scene decorated = DZoneLandmark.addTo(builder.build(calendar, options)); Files.writeString(directory.resolve(name.replace(".svg", "-dzone.svg")), renderer.render(decorated, DZoneLandmark.THEME)); Run ./mvnw verify again and open target/previews/github-dark-dzone.svg. The build also writes the shorter and empty versions with the -dzone.svg suffix, so you can check how the decoration fits several calendars. When you're ready to use your fork on GitHub, update the second checkout in the publishing workflow to your fork's repository and a commit containing your changes. The generation step then runs the customized action from that checkout. Try It With Your Own Contributions Try Quarkus Contribution City with your own GitHub account and add the generated SVG to your profile. Start with the released action, then fork the project and change the character mask to give your city a landmark of its own. You could draw a stepped tower from your hometown or add your team's initials above the street, using the DZone version as a starting point. Share your version in a GitHub issue, or open a pull request if you'd like to contribute a new shape.

Every incident review has a moment where someone asks how the thing was even possible. For agentic AI, the uncomfortable answer is often that everything worked as designed. The credentials were real. The API accepted them. The identity system confirmed the agent was allowed to call that endpoint. And the organization still ended up with an action nobody wanted. That gap is what Salt Security's State of Agentic AI and API Security 2H 2026 puts numbers on. The report surveys more than 380 security and technology professionals and adds telemetry from Salt Labs, the company's research arm. I read it as a security argument, not a scare statistic, so here is what the data supports and where it needs care. The Headline, Read Carefully The number everyone will quote is 49.8%. That share of respondents confirmed or suspected that an AI agent in their organization took an action in the past 12 months that the organization did not intend, expect, or authorize. Read the fine print and the picture gets more interesting. Only 25.3% report confirmed instances. Another 24.5% suspect it happened but could not prove it. A further 12.3% said they would not currently be able to determine whether it happened at all, and 15.4% did not know. Only 22.5% said it had not happened. So the cautious reading is that one in four organizations has a confirmed case, and a large share cannot rule one out. That second group may be the more important finding, because an organization that cannot answer the question has a different problem from one that answered yes. What These Incidents Actually Look Like Among respondents with a confirmed or suspected event, the report breaks down what the action involved. Respondents could pick several outcomes, so the figures overlap: 53.9% reported an external API call or transaction.49.7% reported a configuration or permission change.41.9% reported modified or deleted data.37.2% reported contact with a customer, partner, or third party.32.5% reported access to data the agent should not have had.Only 9.9% said there was no material impact. These are changes in state, made in real systems. When a chatbot gets something wrong, someone reads a wrong answer. When an agent gets something wrong, money moves, records change, or a customer receives a message. I think this is the line security teams should draw: the risk is no longer in what the model says but in what the system around it is allowed to do. Why Valid Access Is Not the Same as Authorization Authentication tells you who is calling. Permission tells you what that caller may touch. Neither tells you whether this specific action, at this moment, serves what the humans actually asked for. Salt Security's report makes a related point about identity. Only 26.1% of respondents say their agents get dedicated identities with scoped permissions. Another 24.3% use dedicated identities with broad permissions. The rest is a patchwork: 8.1% use shared service accounts, 5.2% use human user credentials, and 7.6% rely on embedded credentials or API keys. About 17.5% say it varies by application. The report is blunt that a dedicated identity is necessary but not sufficient. A clearly labeled agent with too much privilege can still do serious damage. The coverage data sharpens the point. Respondents report dedicated controls for AI agents (56.4%) and APIs (55.9%), but only 26.6% for agent identities and 24.8% for agent tools. Those two are exactly where an agent gains the ability to act. The Chain Nobody Designed Part of why this is hard is architectural. The report finds that 76.8% of organizations have agents that invoke other agents or tools that themselves call APIs. For 20.4% this happens routinely. By the time a request reaches the system that changes something, the original instruction can be several steps behind it. A security control that evaluates each API call in isolation sees a legitimate request from a legitimate caller. It cannot see that the intent behind it drifted three hops ago. This is why the report defines the agentic path as the full sequence from instruction, through the model's reasoning and the tools and servers it invokes, to the API call and the outcome. I find that framing useful because it gives defenders a unit of analysis that matches how the system actually behaves. Can You Reconstruct What Happened? Here the numbers are hard to wave away. Only 12.5% can consistently trace an agent's complete path across all agents, and 38.6% can do so for most. Asked how long it would take to determine every action a specific agent took yesterday, only 15.7% said minutes with automated tooling. 36% said hours of manual log correlation, and 28.5% said days and multiple teams. That means 64.5% are looking at hours or days. Hours matter when the actor is software. An agent can repeat an action, call several services, and change configurations without waiting for anyone's workday to start. Accountability looks somewhat better than visibility: 20.9% can name the responsible human or team for all agents, and 51.2% for most. But as the report notes, knowing whose agent it was does not tell you what it did. Stopping It Is Mostly a Human Job Only 17.5% can automatically stop an agent that begins behaving unexpectedly in production. About 56.1% can stop one manually within minutes. 12% can only take the whole application down, 3.1% cannot stop it, and 11.2% do not know. So 85.6% have some way to intervene, but most of that way runs through a person noticing first. The report's recommendation here is the part I would underline. A kill switch is useful, but it should not be the only thing standing between an odd decision and an irreversible action. The stronger design is preventive and layered: least privilege on the agent identity, explicit policy at the tool and API boundary, step-up human approval for sensitive operations, transaction limits, auditable action records, and fast credential or session revocation. The Connective Layer Is Now an Attack Surface The Model Context Protocol, or MCP, gives AI applications a standard way to discover and use external tools. The report is careful to say the protocol is not inherently unsafe. The concern is that it concentrates powerful capabilities behind an interface an agent can invoke. In the survey, 32.9% confirmed finding an unauthorized, shadow, or misconfigured MCP server in the past year, and 54.3% confirmed or suspected one. Salt Labs also reports detecting MCP-related infrastructure in 21.6% of customer environments in August 2026, up from 16.7% in January. The report itself cautions that the customer population grew between those measurements, so this is a change in detected prevalence rather than growth in a fixed group. The same restraint applies to the attack data. Salt Labs observed attacker activity on agentic connected API paths in 26.3% of the environments where it looked, but calls the analysis experimental, limited, and directional. I would treat it as a signal that these endpoints are already on attackers' radar, not as a market-wide rate. Separately, the broader telemetry covers 5,319,583 API endpoints and 104,339 attacker instances over twelve months, and 95.8% of those attacker instances were authenticated. The report warns against assuming those are agents. The takeaway is narrower and still important: authentication alone does not establish trust. A Framework I Would Use: Sort Actions by Consequence This part is my analysis, not the report's. Do not treat every agent action as one risk class. I would sort them into three: Read. Retrieving information. Often fine to run autonomously with logging.Write. Changing a record, a configuration, or a permission. Needs policy checks tied to the resource and the sensitivity of the change.Destructive or external. Deleting data, moving money, altering access, or contacting a third party. Needs the strongest gate, which may mean human approval or hard transaction limits. Notice that the survey's own breakdown maps onto these tiers. External transactions, permission changes, deletions, and third-party contact are all in the top half of the list. A read-only assistant and a write-capable agent should never share the same trust assumptions just because they share a model. What I Would Do on Monday The report's recommendations point in a consistent direction, and I would compress them into five moves: Build a living inventory that maps agents, MCP servers, tools, APIs, and data flows together, not just a flat list of endpoints.Give each production agent its own non-human identity with narrow scope, a named owner, and credentials that can be revoked quickly.Put policy checks at the action boundary, before the high-impact call executes, instead of relying on detection after the fact.Preserve a correlated trail from the initiating instruction to the final state change, and aim to make reconstruction a minutes-long task.Measure the controls: scoped identity coverage, path trace coverage, time to reconstruct, time to contain, and how quickly unknown MCP servers are discovered. A Note on Sourcing This is a vendor survey. Salt Security sells agentic and API security products, and the report's own methodology states that it is not a fixed respondent panel, so changes from the prior report are directional. Treat the figures as informed signal rather than audited fact. The two executive comments in the release reflect that vendor perspective. Salt CEO Roey Eliyahu said the data shows a gap "already producing real operational consequences." Yaniv Balmas, Salt's Head of Research, said the company's Manus research and the survey are "two views of the same problem." Per the release, Salt Labs showed that a single malicious email could hijack the Manus agentic platform and reach connected accounts, and that the platform's guardrail detected the attack but did not stop it before the malicious code ran. I have not independently verified that research, so I am attributing it to Salt. The Real Question The report closes on a test I find hard to improve: can your organization say what an agent did, why it was allowed, and whether you can stop it? Most cannot yet answer all three. The sensible response is not to slow agents down out of fear. It is to treat their authority as a privileged interface, engineered and monitored like every other one. Source: Salt Security, State of Agentic AI and API Security 2H 2026

By Igboanugo David Ugochukwu CORE

A few months ago, our RAG (Retrieval-Augmented Generation) setup did what most first versions do: the Streamlit app called OpenAI directly, pulled chunks from MongoDB, and stitched an answer together. It worked. It also had three problems nobody wants to discover in production. First, the OpenAI bill. Users ask the same question in ten different ways: "How many leave days do I get?" and "What's the annual leave entitlement?" are the same question, but if you're calling the LLM for every single one of them, you're paying for the same answer over and over. Second, if OpenAI had a slow minute, every single user felt it. Nothing was cached anywhere, so every request, no matter how routine, went all the way to the model. Third, and honestly the scarier one: an LLM will happily answer a question even when the answer isn't in your documents. It doesn't say "I don't know"; it just generates something plausible. For an internal knowledge assistant, that's not a bug you can shrug off — that's a wrong HR policy answer someone acts on. So we built a gateway. Nothing exotic — a single FastAPI service that sits between the UI and everything else (OpenAI, Gemini, MongoDB). The client never talks to those systems directly anymore; it only ever talks to the gateway, and the gateway decides what happens next. Why a Gateway and Not Just “Add Caching Somewhere” The tempting shortcut is to bolt caching onto the existing call site and move on. We didn't do that, mainly because caching, grounding, routing, rate limiting, and logging all need to see the same request at the same point in its lifecycle. Splitting that logic across the UI and a dozen helper functions means every future change touches five files. Put it all in one place instead, and it's one file you read to understand the entire request path. Nothing here is framework magic. No LangChain, no LlamaIndex. It's the OpenAI SDK and MongoDB's native $vectorSearch called directly. That was a deliberate choice — when something behaves oddly, we want to be able to step through our own code, not a framework's abstraction layer. What Actually Happens on a /Query Call This is the part that matters most, so here's the full path a question takes: Each of those boxes guards against one of the three problems above. Rate limiting protects against abuse. The cache lookup protects against paying twice for the same question. The grounding gate protects against confidently wrong answers. The budget check protects the invoice. And the last step means none of this is a black box — every request is logged with its cache status, provider, tokens, and cost, and shows up live in the Analytics tab. Semantic Caching: Matching What People Mean, Not What They Typed Here's the part people usually find counterintuitive. We don't do exact-string caching. We embed the incoming query the same way we embed document chunks, and compare it against previously answered queries stored in their own cache collection, using cosine similarity. If the match is above 90%, we return the stored answer straight away — no LLM call at all. What that looks like in practice: "How many annual leave days?" -- miss (first time anyone's asked)Same question again -- hit (exact repeat)"What's the yearly vacation entitlement...?" -- hit (different words, same question)"Meal allowance during travel?" -- miss (genuinely different topic) That third line is the whole point. An exact-match cache would have missed it and burned an LLM call for a question we'd already answered five minutes earlier. Cosine similarity catches the paraphrase. We also cache at ingestion time, which is easy to overlook but saves real money. Before embedding a document chunk, we hash it. If that hash already exists, we skip it — no embedding call, no reprocessing. Re-uploading a document you already ingested costs nothing. Only the chunks that actually changed get new embeddings. Making Sure the Model Doesn’t Just Make Something Up This is the guardrail we care about most, and it's deliberately built as two separate layers rather than one clever one: A confidence gate before generation. We retrieve the top-k chunks and check the best similarity score. If it's below our grounding threshold, we never even call the LLM — we return a "not in the documents" response instantly. Cheap, fast, and it can't hallucinate because it never ran.A strict system prompt during generation. Even when the gate passes, the model is explicitly instructed to refuse if the retrieved context doesn't actually contain the answer. Neither layer alone is bulletproof. Together, they catch most of the ways an LLM can wander off and answer a question you never asked it to answer with information it invented. Routing on Cost, Not Just on Uptime Our primary model is gpt-4o-mini. The gateway keeps a running total of what we've spent since the last budget reset, and once that total hits the configured limit, new requests get routed to Gemini 2.5 Flash instead — called through OpenRouter's OpenAI-compatible API, so it's the same SDK, just a different base URL and model name. No separate integration to maintain. The part worth calling out: the budget number is a runtime setting, not a constant baked into the code. Whoever owns the budget can change it without anyone redeploying anything. And if the fallback provider is ever misconfigured, we fail back to the primary rather than failing the request outright. What We Get to See None of this is useful if it's invisible. Every request - cache status, grounding result, provider used, tokens, cost, latency — gets logged, and it's all visible in the Analytics screen. There's also a Routing screen that shows the currently active provider alongside spend versus budget, in real time. Rate limiting runs on a 60-second sliding window per client IP, so one runaway script can't drown out everyone else. Where That Leaves Us PatternWhat it doesWhy it mattersGateway / ProxyOne FastAPI service in front of all RAG trafficOne place to change behavior, not fiveSemantic CachingCosine similarity match against past queriesLower cost, faster answers on repeat/paraphrased questionsGrounding GateConfidence threshold + strict system promptFewer hallucinated answersCost-Based RoutingAuto-switch provider once budget is hitSpend stays predictable, no manual babysittingEmbedding DedupHash chunks, skip re-embedding unchanged onesCheaper, faster re-ingestionRate Limits + LoggingSliding window per IP, per-request metricsAbuse protection and actual visibility into cost None of this required a heavyweight framework — it's FastAPI, MongoDB Atlas, and the OpenAI SDK, wired together deliberately. The bigger lesson for us wasn't any single pattern; it was that caching, grounding, and routing all want to live at the same layer, because they're all answering the same underlying question: Should this request even reach the model, and if so, which one? If you're running a RAG app that talks to the LLM directly today, that question is worth asking before the bill or a bad answer asks it for you.

This article outlines how Delta Lake tables managed in Databricks and those managed in Snowflake can be shared in both directions without duplicating data. This is achieved through the use of shared object storage (ADLS) and open table formats. The article discusses the benefits, tradeoffs, business and cost advantages, the approach to implementation, and main technical aspects to consider. Overview Interoperability enables one platform to access data that is owned by another platform, without moving the data.Both Databricks and Snowflake store a single copy of the underlying Parquet files in ADLS, and they share only table metadata through catalog APIs. Databricks stores data in Delta (or Delta Uniform), which Snowflake can read via catalog integration.On the other hand, Snowflake-managed Iceberg tables can be accessed by Databricks through catalog federation. Metadata is the Key to Interoperability Key Advantages No data duplication: One physical copy of data in ADLS serves both engines, eliminating duplicate storage and copy pipelines.Meet users where they are: Producers publish once; consumers query from Databricks or Snowflake with their own compute.Best-of-breed & open formats: Open Parquet + Delta/Iceberg formats avoid lock-in and support future OLTP/OLAP convergence.Fresher data: Consumers read the latest committed metadata rather than a stale nightly extract. Pros and Cons Points to consider Pros Cons / Limitations No storage duplication — single copy in ADLS. Cross-engine writes are limited; the consuming engine is typically read-only. Lower latency than nightly copy / ETL jobs. Metadata refresh must be configured; a lag exists between write and visibility. Uses each engine's native compute and cost model. Converting Snowflake tables to Iceberg can surface datatype mismatches. Open formats reduce vendor lock-in. Governance must be re-applied per engine—policies do not travel with the table. Business and Cost Benefits Lower cost: Removes redundant storage and the compute cost of copy/sync pipelines between the two platforms.Speed: Lower data and process latency means data products reach consumers sooner (faster GTM).Compliance: A single governed copy supports data minimization and retention obligations.Agility: Teams pick the right tool for each workload without re-platforming the underlying data. Databricks to Snowflake: Implementation Approach Databricks owns the Delta/Uniform table, while Snowflake retrieves metadata through the Catalog Integration API and reads the data directly from ADLS. Catalog Integration Flow Snowflake requests metadata from Unity Catalog, receives the table location, and reads Parquet data directly from ADLS using an external volume. Databricks-Managed Delta Read by Snowflake (Catalog Integration) Databricks remains the owner; Snowflake reads in place. 1. Create external Delta tables with UniForm (Iceberg) enabled in ADLS, registered in Unity Catalog. SQL CREATE TABLE Catalog.SchemaName.TableName ( ID INT, Name STRING, CreatedAt TIMESTAMP ) TBLPROPERTIES ( 'delta.columnMapping.mode'='name', 'delta.enableIcebergCompatV2'='true', 'delta.universalFormat.enabledFormats'='iceberg' ); 2. Create an external volume in Snowflake pointing to the ADLS path (Managed Identity authentication). SQL CREATE OR REPLACE EXTERNAL VOLUME DEVTEAMADLS_EXE_VOL STORAGE_LOCATIONS = ( ( NAME='DEVTEAMADLS_EXE_VOL', STORAGE_PROVIDER='AZURE', STORAGE_BASE_URL='azure://devteamadls.blob.core.windows.net/Data/', AZURE_TENANT_ID='<tenant id>' ) ); 3. Create a catalog integration so Snowflake reads current metadata from Unity Catalog's Iceberg REST endpoint. SQL CREATE OR REPLACE CATALOG INTEGRATION DEVADLS_CATALOG_REST CATALOG_SOURCE = ICEBERG_REST TABLE_FORMAT = ICEBERG CATALOG_NAMESPACE = 'SchemaName' REST_CONFIG = ( CATALOG_URI = 'https://<workspace>.azuredatabricks.net/api/2.1/unity-catalog/iceberg', WAREHOUSE = 'DevDeltaLake_Warehouse' ) REST_AUTHENTICATION = ( TYPE = OAUTH, OAUTH_TOKEN_URI = 'https://<workspace>.azuredatabricks.net/oidc/v1/token', OAUTH_CLIENT_ID = '<shared via secure exchange>', OAUTH_CLIENT_SECRET = '<shared via secure exchange>', OAUTH_ALLOWED_SCOPES = ('all-apis','sql') ) ENABLED = TRUE; 4. Create the Iceberg table in Snowflake and apply RBAC + masking on the Snowflake side. SQL CREATE ICEBERG TABLE SFDBName.SFSchema.SFTableName EXTERNAL_VOLUME = 'DEVTEAMADLS_EXE_VOL' CATALOG = 'DEVADLS_CATALOG_REST' CATALOG_TABLE_NAME = 'DeltaTableName' CATALOG_NAMESPACE = 'DeltaSchemaName' AUTO_REFRESH = TRUE; ALTER ICEBERG TABLE ... MODIFY COLUMN <col> SET MASKING POLICY <policy>; Snowflake to Databricks: Implementation Approach Snowflake remains the owner; Databricks reads in place. Query federation is the lighter alternative when Iceberg conversion is not desired. Snowflake owns the Iceberg table; Databricks federates the catalog and reads data in place from ADLS. Catalog Federation Flow Databricks federates the Snowflake catalog to retrieve the latest metadata, receives the updated table location, and reads Parquet data directly from ADLS through an external location using Managed Identity. Snowflake-Managed Data Read by Databricks (Catalog Federation) 1. Create Iceberg tables in the Snowflake-managed catalog on an external volume path. SQL CREATE OR REPLACE ICEBERG TABLE WareHouse.Schema.TableName ( c_custkey INTEGER, c_name STRING, c_acctbal INTEGER ) CATALOG = 'SNOWFLAKE' EXTERNAL_VOLUME = 'my_ext_vol' BASE_LOCATION = 'schema/tablename'; 2. Provision a resource account (key pair) and grant Databricks' role read access to the Iceberg tables. SQL ALTER USER UserName SET RSA_PUBLIC_KEY='<public key>'; GRANT USAGE ON DATABASE DatabseName TO ROLE MY_DATABRICKS_ROLE; GRANT USAGE ON SCHEMA DatabseName.SchemaName TO ROLE MY_DATABRICKS_ROLE; GRANT SELECT ON ALL TABLES IN SCHEMA DatabseName.SchemaName TO ROLE MY_DATABRICKS_ROLE; GRANT SELECT ON FUTURE TABLES IN SCHEMA DatabseName.SchemaName TO ROLE MY_DATABRICKS_ROLE; 3. In Unity Catalog, create the Snowflake connection, an external location for the ADLS path, and a foreign catalog with the authorized path. SQL CREATE CONNECTION SnowFlakeConn TYPE snowflake OPTIONS ( host '<acct>.snowflakecomputing.com', port '443', sfWarehouse '<wh>', user 'UserName', <key-pair auth options> ); CREATE EXTERNAL LOCATION snf_iceberg_loc URL 'abfss://<container>@<storage>.dfs.core.windows.net/' WITH (STORAGE CREDENTIAL <cred>); CREATE FOREIGN CATALOG SnowFlakeCatalog USING CONNECTION SnowFlakeConn OPTIONS ( database '<snf_db>', authorized_paths 'abfss://<container>@<storage>.dfs.core.windows.net/' ); 4. Refresh and query. Databricks reads Iceberg data directly from ADLS using its own compute. Apply RBAC and masking policies in Databricks. SQL REFRESH FOREIGN CATALOG SnowFlakeCatalog; SELECT * FROM SnowFlakeCatalog.SchemaName.TableName; Conclusion Through the use of interoperability, both Databricks and Snowflake will be able to share data bidirectionally through a single version of the data that is present in ADLS, but with metadata exchange. This will lead to no duplication of data and will always provide access to fresh data. There are several advantages for businesses in terms of cost savings through storage and ETL cost reduction, faster access to data, increased flexibility, and vendor lock-in reduction.

I moved a project from Java to Kotlin in a single pass, back when the codebase was small enough that doing so cost almost nothing. The system I work on now is a different animal: hundreds of thousands of lines of Java, a monolith rather than a set of services. Converting it wholesale is the kind of gamble nobody signs off on. The option still exists technically, but the cost is now measured in quarters of engineering time rather than days. How much code existed when someone first asked about Kotlin decided everything that followed. I have spent more than a decade building and scaling high-load distributed systems on the JVM, across fintech, security, and media infrastructure, and the pattern holds: you pay for a language migration at the start, well before anyone puts it on a roadmap. The Easy Migration The small project was a handful of Spring Boot microservices with JPA and PostgreSQL, still short of production at a few thousand lines of code. A colleague who spent his evenings in the Kotlin community suggested we try it on this one. The team was small and curious, so we agreed within a day or two. Spring compatibility was the thing we worried about, and it never became a problem. Because nothing had shipped yet, we converted the entire codebase at once. IntelliJ IDEA can rewrite a Java file to Kotlin with a single keystroke, and the converter handles about 80 percent of the conversion on its own. The rest of the week went into manual cleanup. Adding nullability annotations stripped out the platform types, and the compiler immediately flagged a dozen possible null-pointer exceptions that Java had been hiding inside Optional. We moved from field injection to constructor injection, which fits Kotlin far better. Data classes replaced our POJOs and trimmed about a quarter of the boilerplate, making the model easier to read. Static members needed more attention. After conversion, our public static final constants ended up inside a companion object, which changed the bytecode that Java callers expected, so we annotated them with @JvmStatic and @JvmField to keep the modules we hadn't touched yet compiling. By the end of the week, the tests were green, and Spring Boot loaded the new classes without complaint. The whole exercise took one week because the codebase was small, which is the entire point: the same cleanup across a large system is measured in quarters, not days. The Parser That Sold Me The message parser is what convinced me the move was worth defending to the rest of the team. Our services received data, validated it against an Avro schema, and returned a result. Parsing under load burns resources, and many Java libraries rely on runtime reflection during serialization and deserialization. Jackson can match the performance without reflection, to be fair: its Afterburner and Blackbird modules generate bytecode at runtime for field and method access, which is a real optimization. The cost shows up per format. JSON, XML, YAML, TOML, and Avro protocols each want their own modules and manual wiring, and a new format usually means finding a library or writing one. Kotlinx.serialization works differently. Its compiler plugin generates every serialization routine at build time and never calls reflection. We standardized most formats around Kotlinx.serialization, using it as the common model layer for JSON, XML, YAML, and TOML, then bolted on Avro as a separate plugin module. The schemas were plain Kotlin classes with the @Serializable annotation, and code generation handled the rest. We had to implement Avro's default values in the plugin by hand, but only once. A smaller dependency surface, typing that held end to end, one approach instead of five. That early switch also bought time to question the rest of the stack while it stayed light: jOOQ for queries that grew complicated, smaller web frameworks for lightweight services, and test tooling with more expressive assertions. Experiments are cheap on a compact codebase, and you can settle choices that would turn into prolonged arguments once the system has weight and a dozen people depending on it. The Monolith, and What to Do About Yours The same migration on the monolith I work on now is a different proposition, and I would push back on anyone who claimed it could happen in one pass. The window there has thinned to a slot rather than shut. Java and Kotlin still compile side by side in a single project, leaving an incremental approach open. You can write new tests in Kotlin against the existing Java, peel bounded contexts off into separate Kotlin services, and drop the idea that migration has to happen all at once. None of that demands the big-bang rewrite that scares management off, and each piece keeps the window from closing further. Postponing the conversation makes it more expensive every quarter, which is the cost the easy migration above never had to pay. If your codebase is still small, bring it up now, while the call is for engineers, not a budget committee. The migration window rarely closes dramatically. One quarter, it simply stops being practical. It just gets heavier to lift, and in one planning session, you realize the topic stopped coming up a while ago.