DZone Spotlight

For the past decade, the gold standard of Site Reliability Engineering (SRE) was "all your telemetry in one place, beautifully rendered." Centralized platforms allowed teams to build massive grids of charts tracking CPU usage, memory allocation, request latencies, and error rates. Yet this approach masks a critical flaw: traditional dashboards show only symptoms, forcing humans to act as data archaeologists to find the root cause. With microservices handling highly distributed workloads, a simple user request might cross a dozen internal boundaries. When latency spikes, a static dashboard tells you what is broken, but it cannot tell you why. The modern engineering directive has fundamentally shifted from reactive visibility to deep system understanding. The conventional approach of isolating metrics, logs, and traces into discrete dashboard tabs or separate tools has broken down under cloud-native complexity: The Contextual Chasm: When an alert fires, engineers are forced to pivot between systems, manually trying to correlate a metric spike in one graph with a raw text error in a log file. This manual synchronization creates a high mean time to resolution (MTTR).Pre-Aggregation Destruction: Traditional monitoring relies on pre-aggregated data averages to control cloud storage costs. This destroys fine-grained details, making it impractical to debug highly specific "unknown-unknown" incidents affecting rare user cohorts.Anticipation Bias: Dashboards assume that a software engineering team can accurately predict every failure mode ahead of time to build a chart for it. Complex systems fail in entirely unpredictable ways that static, predefined views cannot capture. The Observability Revolution: Core Paradigms Replacing the Dashboard Rather than forcing engineers to stare at screens waiting for a line to turn red, modern observability treats system telemetry as a unified, exploratory analytics problem: High-Cardinality Unified Data Layers: Modern architectures store unaggregated, raw event streams. By preserving trace IDs, customer IDs, and container headers, engines can instantly pinpoint the precise thread or database index responsible for a system failure.Contextual Topology and Causality: Advanced systems continuously map software dependencies and traffic patterns. Instead of sending a dozen isolated alert notifications, the system assembles an incident narrative, proving how a configuration drift upstream actively broke a payment processing gate downstream.Agentic AI Remediation: The industry is moving from monitoring toward autonomous operations. Specialized AI agents consume telemetry streams directly to execute preventative actions like scaling production clusters or rolling back buggy code deployments. The traditional dashboard view: application health, load, response times, and error rates show that something is wrong, but not why. The engineer still has to hunt for the cause. The high-cardinality view: instead of scanning charts, the failing dimension is isolated directly — here, errors tied to a specific service version (v350.10) and HTTP 401 status — pointing straight at the root cause. Adapting to the Age of Non-Deterministic Infrastructure The massive expansion of large language model (LLM) agents and generative software layers has further accelerated the demise of legacy dashboards. Traditional systems check whether an API endpoint returns a 200 OK HTTP status code. An AI-driven microservice can operate perfectly within technical boundaries, response times, and system limits, while still delivering entirely hallucinated or factually incorrect output to an enterprise client. Observability has adapted by monitoring semantic tokens, prompt guardrails, and vector pipeline drift, complex factors that cannot be displayed on a standard bar chart. The metric of success for a modern technology infrastructure is no longer the number of operational dashboards under management. Success is defined by autonomous, rapid understanding and instantaneous path-to-resolution, which is why the static visual graph has been displaced as the primary debugging surface. The Strategic Dilemma Transitioning from static, pre-aggregated dashboards to high-cardinality observability (tracking raw, un-aggregated event data across highly granular dimensions like user_id, container_id, and ip_address) is essential for modern debugging and reducing MTTR. However, storing raw, multi-dimensional event streams exponentially inflates cloud infrastructure and data ingest costs. Without strict architectural guardrails, observability spend can grow to consume an unsustainable percentage of total cloud budgets. The True Drivers of High-Cardinality Cost Exploding Volume of Unique Dimensions: Cloud-native elements (ephemeral containers, serverless functions) spin up and down constantly, generating billions of unique metric combinations that saturate traditional relational structures.Data Ingestion Overhead: Commercial observability vendors charge based on data volume ingested and index size, and self-hosted open-source clusters (e.g., OpenSearch, or Prometheus with Cortex/Thanos) carry the same cost pressure through the infrastructure required to store and index that volume.Hot Storage Reliance: Keeping raw, unaggregated logs and traces instantly queryable requires expensive, high-throughput SSD cloud storage, driving up baseline cloud infrastructure costs. Architectural Framework for Cost Mitigation To balance the technical necessity of high-cardinality data with strict fiscal control, tech leaders should mandate a tiered storage approach: Implement Intelligent Data Sampling Not all telemetry is created equal. SRE teams should configure tail-based sampling rules. For instance, store 100% of traces that result in an HTTP 5xx error or high latency, but sample only 1% to 5% of healthy, predictable HTTP 200 OK traces. Adopt Tiered Storage Horizons Hot Tier (0–7 Days): Keep full, high-cardinality raw telemetry in expensive, high-performance storage for immediate active incident response.Warm Tier (8–30 Days): Move data to aggregated metrics blocks or lower-cost search-optimized tiers for historical trend analysis.Cold Tier (31+ Days): Compress and offload raw data into highly economical object storage (such as Amazon S3, Google Cloud Storage, or Azure Blob) strictly for long-term compliance and audit requirements. Tiered storage keeps high-cardinality observability affordable: recent data stays hot for fast incident response, then ages into progressively cheaper, slower storage for trend analysis and compliance. Enforce Strict Lifecycle Policies. Automate the purging or down-sampling of high-cardinality dimensions as data ages. A specific container ID from three weeks ago holds little operational value for daily SRE workflows. Verdict Do not scale back observability goals to save on infrastructure bills. Instead, optimize how and where that telemetry lives. By implementing data sampling at ingestion and automated lifecycle workflows, organizations can unlock the full power of high-cardinality root-cause analysis while keeping data storage costs linear and predictable.

Rebuilding an Elasticsearch index from PostgreSQL sounds simple: Create a new index.Copy the rows that belong in search.Replay anything that changed during the copy.Verify the result.Move the search alias. The difficult part is deciding where the bulk copy ends and the replay begins. A transaction can update a row before the rebuild starts and commit after the copy has taken its database snapshot. With the wrong watermark, that change may appear in neither the copy nor the replay. The rebuild finishes successfully, but the new index is wrong. I built a controlled recovery drill to measure this problem. The test used synthetic data rather than information from a client or production system. The setup included: 500,000 PostgreSQL rows449,917 rows initially matching the search predicate2,000 updates, 250 deletes, and 120 inserts during each rebuildEight concurrent writer connectionsA two-node Elasticsearch 8.19.3 cluster I ran the timestamp-based strategy five times. It produced the same 366 measurable inconsistencies in every run. The corrected transaction-aware strategy was tested in 30 runs across several configurations. All 30 passed the implemented pre-cutover checks. The most surprising result was not the total number of errors. It was that a field-level comparison of 5,000 documents reported no stale content in any of the five defective runs. First, Confirm That a Rebuild Is Necessary Before rebuilding from PostgreSQL, confirm that the existing index cannot be recovered. Resolve the application alias, identify the physical index behind it, and inspect the affected shards. Elasticsearch’s cluster allocation explain API can show whether an unassigned shard is caused by disk pressure, a missing node, an allocation rule, or the absence of a valid shard copy. If a usable snapshot exists, restoring it into a separate versioned index may also be faster than rebuilding the full projection. The rest of this article assumes that allocation repair and snapshot restoration are not available. Unsafe Watermark 1: updated_at The first rebuild strategy used a timestamp as the replay boundary: SQL -- Record the replay boundary. SELECT clock_timestamp(); -- Copy the current search projection. SELECT id, title, body, status, version, updated_at FROM searchable_content WHERE status = 'PUBLISHED' ORDER BY id; -- Replay rows changed after the boundary. SELECT id, title, body, status, version, updated_at FROM searchable_content WHERE status = 'PUBLISHED' AND updated_at > :rebuild_started_at ORDER BY id; This approach assumes that updated_at represents commit order. In the drill, the value was assigned when the update statement ran, not when the transaction committed. Consider this sequence: Plain Text Session A: UPDATE row 1 updated_at = 17:23:46.252262 transaction remains open Session B: reads the rebuild watermark watermark = 17:23:46.465777 Session A: COMMIT The transaction committed after the watermark, but the row carried a timestamp from before the watermark. The bulk scan did not include the update because its PostgreSQL snapshot was created while the transaction was still open. The replay did not include it because the row failed this condition: SQL updated_at > :rebuild_started_at The change fell between the two phases. This issue is not limited to clock_timestamp(). A timestamp assigned by an application, database statement, or trigger may still be created before the surrounding transaction commits. If application servers assign the value, differences between their clocks introduce another source of error. Timestamp replay also has two simpler problems. First, a deleted row no longer exists in the source table. A query against the current rows cannot tell Elasticsearch that the old document must be removed. Second, a row may remain in PostgreSQL but stop matching the search predicate. In this experiment, a row could change from PUBLISHED to DRAFT. The replay query selected only published rows, so it did not remove the old Elasticsearch document. At the end of each timestamp-based run, the defects were: Defect classCountMissing from Elasticsearch3Deleted from PostgreSQL but still indexed228Outside the search predicate but still indexed131Present in both systems but stale4Total366 PostgreSQL contained 449,863 rows matching the final predicate, while Elasticsearch contained 450,219 documents. The net count difference was 356, which understated the real problem. Three missing documents offset part of the 359 unexpected documents, and the four stale documents did not change the count at all. Unsafe Watermark 2: MAX(seq) A transactional outbox solves part of the problem. A trigger can write an outbox event in the same transaction as every insert, update, or delete. Deletes now leave tombstones, so the replay process knows which Elasticsearch documents to remove. A first outbox design might use a BIGSERIAL sequence. Before starting the copy, the rebuild records the largest visible sequence value: SQL SELECT MAX(seq) FROM reindex_outbox; After the copy, it replays everything above that value: SQL SELECT seq, content_id, op FROM reindex_outbox WHERE seq > :starting_seq ORDER BY seq; This avoids clock problems, but it still does not provide a commit boundary. Consider two transactions: Plain Text T1: UPDATE row 1 outbox seq 51 allocated transaction remains open T2: UPDATE row 2 outbox seq 52 allocated COMMIT Rebuild process: SELECT MAX(seq) returns 52 T1: COMMIT When the rebuild reads MAX(seq), T1 is still invisible. T2 has committed, so the largest visible sequence is 52. T1 later commits with sequence 51. A replay using seq > 52 will never return the T1 event. PostgreSQL sequences generate distinct values, but they do not represent commit order. The PostgreSQL sequence documentation also explains that a value returned by nextval() is not reclaimed if its transaction later aborts. Sequence allocation and transaction visibility are separate mechanisms. Replacing the timestamp with a counter did not fix the underlying problem because both values were assigned before transaction visibility was settled. Figure 1. Timestamp and sequence-maximum watermarks can both miss a transaction that starts before the watermark and commits afterward. A Transaction-Aware Replay Window The corrected implementation records the PostgreSQL transaction ID in every outbox event: SQL CREATE TABLE reindex_outbox ( seq bigserial PRIMARY KEY, content_id bigint NOT NULL, op text NOT NULL CHECK (op IN ('upsert', 'delete')), xact_id bigint NOT NULL DEFAULT pg_current_xact_id()::text::bigint, captured_at timestamptz NOT NULL DEFAULT clock_timestamp() ); The outbox event is inserted in the same transaction as the source change. Before beginning the bulk scan, the rebuild records the current PostgreSQL snapshot’s xmin: SQL SELECT pg_snapshot_xmin( pg_current_snapshot() )::text::bigint; PostgreSQL defines snapshot xmin as the lowest transaction ID that was still active when the snapshot was taken. This gives the rebuild a conservative lower bound for transactions that may not be represented in the bulk-copy snapshot. The replay population begins with: SQL SELECT DISTINCT content_id FROM reindex_outbox WHERE xact_id >= :snapshot_xmin; This may include changes that the bulk scan already copied. That is acceptable because the Elasticsearch operations use stable document IDs and are designed to be idempotent. The snapshot tells the rebuild where replay should begin. It does not tell the rebuild when replay is complete. A Final Boundary Is Still Needed An outbox consumer may catch up while writes continue. It may even report zero lag for a moment. That does not close the recovery window because another transaction can commit immediately afterward. The drill therefore uses a final quiescent boundary: Capture the transaction-aware replay lower bound.Bulk-copy the source projection while writes continue.Drain visible outbox events.Stop accepting new source mutations.Allow transactions already in progress to finish.Drain the remaining events.Verify the new index.Move the alias and resume writes. The synthetic writer had a finite plan and had already completed by the time the final barrier was applied. The experiment therefore measured finalization after the writer became idle. It did not measure the delay that production requests would experience while blocked by an active write barrier. A system that cannot pause writes needs a durable stream handoff. PostgreSQL logical decoding, for example, can combine an exported snapshot with a persistent replication position. Do Not Use One Sequence Cursor for the Drain My first outbox drain repeated the sequence problem in another place. It used: SQL WHERE seq > :last_applied_seq Suppose sequence 52 becomes visible while the transaction holding sequence 51 remains open. If the consumer advances its cursor to 52, sequence 51 is skipped when it later commits. The corrected implementation records applied events individually: SQL SELECT o.seq, o.content_id FROM reindex_outbox o LEFT JOIN reindex_progress p ON p.rebuild_id = :rebuild_id AND p.seq = o.seq WHERE o.xact_id >= :snapshot_xmin AND p.seq IS NULL ORDER BY o.seq; This is an applied-event ledger rather than one high-water mark. A late commit cannot appear below a cursor that has already moved past it. The replay also reads the current PostgreSQL state instead of trusting an old document payload stored in the outbox: If the row currently matches the search predicate, it is upserted.If the row was deleted, the Elasticsearch document is deleted.If the row still exists but no longer matches the predicate, the document is also deleted. Several events for the same source ID therefore converge on its current state. An outbox event is marked as applied only after its Elasticsearch operation succeeds. Unresolved operations remain unapplied and cause the cutover to fail. The Elasticsearch Bulk API also requires item-level checking. A request can return HTTP 200 while individual operations fail and the response contains errors: true. In a separate overload test, 525 rejected document operations appeared inside 22 HTTP-200 responses. Checking only the HTTP status would have reported those requests as successful. Why a 5,000-Document Sample Reported a Clean Result I tested three lightweight reconciliation methods against the timestamp-based rebuild: Counts divided into six ID rangesChecks against the 250 IDs selected for deletionA field-level comparison over 5,000 randomly selected source records The first two checks found problems immediately. All six count ranges differed, and 228 of the 250 deleted IDs were still present in Elasticsearch. The sampled content comparison found none of the four stale documents in any of the five runs. The full-corpus comparison found all four. For a population of N comparable documents containing s stale records, and a sample of n documents selected without replacement, the probability of finding at least one stale record is: Plain Text P(detect at least one) = 1 - C(N - s, n) / C(N, n) For this experiment: Plain Text N = 449,860 documents present in both systems s = 4 stale documents n = 5,000 sampled documents The probability of finding at least one stale document was approximately 4.37%. To reach a 95% detection probability, the sample would need 237,134 documents, or 52.7% of the comparable corpus. Reaching 99% would require 307,601 documents, or 68.4%. Figure 2. Detection probability as the sample size increases when four of 449,860 comparable documents are stale. This does not mean sampling is useless. A 5,000-document sample can quickly reveal a widespread transformation error. Sampling can also be useful as a regular monitoring check between rebuilds. It is weak as a cutover gate for rare defects. Transaction-boundary races may affect only a few records inside an otherwise reasonable-looking index. That is exactly the situation in which a modest random sample has little statistical power. What the Full-Corpus Check Compared The full comparison checked: The complete expected set of document IDsA digest over id, title, body, status, and version Each field was length-prefixed before hashing so that field boundaries could not be confused. PostgreSQL calculated the source-side digest. On the Elasticsearch side, the verifier read the actual fields from _source and calculated the digest again. The drill uses MD5 as a compact equality checksum, not as a security mechanism. The digest excludes updated_at, whose representation was not canonicalized in the same way on both sides. The result was: Reconciliation resultTimestamp rebuildTransaction-aware rebuildMissing from Elasticsearch30Deleted source rows still indexed2280Predicate-excluded rows still indexed1310Stale in 5,000-document sample00Stale in full-corpus comparison40Total measured inconsistencies3660 The digest check verifies the document fields included in this projection. It does not verify mappings, analyzers, synonyms, autocomplete, facets, relevance, or authorization rules. Those require functional search tests before the alias moves. For a much larger index, the comparison should stream sorted ID-and-digest pairs or compare deterministic partitions instead of loading everything into memory. A Practical Pre-Cutover Gate A rebuild should not move the alias until all of these conditions are true: The expected and actual document IDs match.The full-corpus content digests match.No committed outbox event in the recovery window remains unapplied.No Elasticsearch operation remains unresolved.The target index has the required active shards.Representative search canaries pass. The canaries should test the behavior users depend on, not only whether one document exists. Depending on the application, they may include: A known search queryAn autocomplete requestA filter or facetSpecial-character or multilingual contentA tenant or authorization boundary Elasticsearch supports removing the old alias target and adding the new target in one alias update operation. The response should still be checked, and the previous index should be retained until the rollback window has passed. It is also important to measure through verification. In this experiment, final drain, shard preparation, and full-corpus verification added roughly 6.5 to 13.8 seconds after the main rebuild work. A benchmark that stops at the last bulk request does not measure when the index is actually ready. Limitations This was a controlled experiment using synthetic data. The source contained one table, one search predicate, and no joins or enrichment. The main benchmark used one primary shard. Both Elasticsearch nodes ran inside one Podman virtual machine on an Apple M4 Max laptop. This allowed a replica to be allocated, but it did not represent separate machines, storage devices, network paths, or failure domains. The workload plan was seeded, so the operations and target IDs repeated across runs. Thread scheduling, timing, and throughput did not. The writer completed before the final barrier was activated, so the experiment measured quiescent finalization rather than the delay experienced by blocked production writes. The content digest covered five fields and excluded updated_at. It verified document content, not mappings or search behavior. Finally, snapshot xmin is a conservative recovery boundary, not a replacement for a mature change data capture system. Long-running transactions can widen the replay range. Outbox events must be retained for the full rebuild window, and every relevant source mutation path must participate in the same transactional capture process. Conclusion The timestamp watermark failed because the row timestamp was assigned before its transaction committed. MAX(seq) failed for the same reason: sequence allocation did not represent commit order. The reconciliation test exposed a separate weakness. Checking 5,000 documents sounded substantial, but it provided only a 4.37% chance of finding the four stale records created by this workload. A reliable rebuild needs two guarantees: A replay window based on transaction visibility rather than statement timingA complete pre-cutover comparison capable of finding rare inconsistencies Without both, the rebuild can complete, the document count can look reasonable, and the new index can still disagree with PostgreSQL. Companion Materials The synthetic workload, implementation, figures, and per-run results are available in the dzone-recovery-drill-v1.1 companion repository.

Every Kafka tutorial covers idempotency the same way: "Store the event ID. Before processing, check if it exists. If it does, skip. Done." That advice is correct. It is also incomplete in a way that will surface only under load — and only in production, at the moment you least want a surprise. Here is what actually happens inside a multi-replica consumer group when two delivery attempts arrive at the same instant. The Pattern You Know A Kafka topic configured with acks=all and enable.idempotence=true on the producer side still delivers messages at least once. Network partitions, broker leader elections, retry storms — any of these can cause a consumer to receive the same message twice. The standard fix is to deduplicate on the consumer side using a stable event ID included in every message payload. The implementation looks like this: Java // Application-layer idempotency check if (processedEventRepository.existsByEventId(eventId)) { log.warn("Duplicate event — skipping | eventId: {}", eventId); return false; } // Process and record processedEventRepository.save(ProcessedEvent.builder() .eventId(eventId) // ... .build()); Under normal operation, this is correct. One consumer thread, one message, one check, one save. The second delivery hits the existsByEventId guard and is dropped cleanly. The problem is that a Kafka consumer group under load does not serialize delivery at the event level. The Race Condition You Don't See in the Tutorial Kafka's partition assignment guarantees that only one consumer in a group receives messages from a given partition at a given time. What it does not guarantee is that the same event ID cannot arrive via two different partitions — for example, during a producer retry that sends to a different partition — or that two messages with the same semantics cannot appear if upstream logic emits duplicates before the broker sees them. More concretely: if you run two replicas of your consumer service (and in Kubernetes, you will), two pods can receive the same event in the same wall-clock window. The sequence looks like this: Plain Text T=0 Pod A receives event e1, calls existsByEventId("e1") → false (not yet in DB) T=0 Pod B receives event e1, calls existsByEventId("e1") → false (not yet in DB) T=1 Pod A passes the guard, begins processing T=1 Pod B passes the guard, begins processing T=2 Pod A saves ProcessedEvent("e1") → INSERT succeeds T=2 Pod B saves ProcessedEvent("e1") → ??? Both pods read false from the application-layer check before either has written. Both proceed. Without a database-level guardrail, both complete, and you have sent the welcome email — or charged the card, or incremented the counter — twice. The Fix: The Database as the Serialization Point The application-layer check is an optimization — it avoids a write attempt for the obvious duplicate case. It is not, and cannot be, a correctness guarantee, because it is not atomic with the write. The correctness guarantee has to live at the layer that is atomic: the database. The schema includes a UNIQUE constraint on event_id: SQL CREATE TABLE processed_events ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), event_id VARCHAR(255) NOT NULL UNIQUE, -- the guarantee is here event_type VARCHAR(100) NOT NULL, processed_at TIMESTAMP NOT NULL DEFAULT now(), topic VARCHAR(255) NOT NULL, partition_number INT NOT NULL, offset_number BIGINT NOT NULL ); That UNIQUE constraint is enforced at the storage engine level — not at the application level, not at the ORM level. No two rows with the same event_id can be committed, regardless of how many concurrent writers attempt it. The database serializes the race. What changes in the application is how you handle the loser of that race: Java @Transactional public boolean sendWelcomeNotification(UserCreatedEvent event, ...) { String eventId = event.getEventId().toString(); // Fast path: skip if already recorded (optimization, not guarantee) if (processedEventRepository.existsByEventId(eventId)) { log.warn("Duplicate event detected — skipping | eventId: {}", eventId); notificationMetricsService.incrementDuplicate(); return false; } try { // ... process the event ... processedEventRepository.save(ProcessedEvent.builder() .eventId(eventId) .eventType("USER_CREATED") .processedAt(LocalDateTime.now()) .topic(topic) .partitionNumber(partition) .offsetNumber(offset) .build()); return true; } catch (DataIntegrityViolationException ex) { // Two threads passed the application check simultaneously. // The DB constraint caught one of them. This is correct behavior. log.warn("Duplicate event caught at DB level — skipping | eventId: {} | userId: {}", eventId, event.getId()); notificationMetricsService.incrementDuplicate(); return false; } } The DataIntegrityViolationException is Spring's translation of a constraint violation from any supported database (PostgreSQL, MySQL, Oracle). Catching it and suppressing it is not an error condition — it is the correct outcome of a race that the DB resolved correctly. Note that the exception is caught in the service layer, not in the Kafka listener. If it propagated up to the listener, Spring Kafka's retry mechanism would re-deliver the event, triggering more unnecessary attempts. Catching it at the service boundary terminates the race cleanly. Two Layers, One Correct Answer The full pattern has two distinct roles: LayerRoleWhat it stopsApplication check (existsByEventId)OptimizationThe non-concurrent duplicate: the obvious second delivery hours or days laterDB UNIQUE constraint + DataIntegrityViolationExceptionCorrectness guaranteeThe concurrent duplicate: two threads, two replicas, same event, same instant Removing either layer has a cost. Without the application check, every retry hits a write attempt that will fail, generating unnecessary DB load and constraint violations in your logs. Without the DB constraint, the application check is a promise with no enforcement — and distributed systems do not honor promises. The right posture is to treat the application check as a performance optimization and the database constraint as the correctness invariant. What This Looks Like Under Fault Injection I verified this pattern against deliberate duplicate delivery in a Kafka consumer running two Spring Boot replicas. Injecting two simultaneous deliveries of the same eventId produced exactly one SENT notification log and one SKIPPED log (the DataIntegrityViolationException path). No double side effects, no exceptions visible to the retry mechanism, no compensating logic required. The database caught the race that the application check could not. Key Takeaways Application-layer idempotency checks are not atomic with the writes they guard. Under concurrent delivery, both threads can pass the check before either writes.The database's UNIQUE constraint is the actual serialization point. It is enforced at a level that the application cannot race.DataIntegrityViolationException is a success path, not an error path. It means the DB caught a concurrent duplicate — suppress it, do not propagate it.You need both layers. The application check eliminates unnecessary write attempts. The DB constraint enforces the invariant. Neither is sufficient alone. If your Kafka consumer only has the application check, you have an idempotency aspiration, not an idempotency guarantee. The race condition described here may never surface on a single-replica dev environment. It will surface on a multi-replica production deployment, under the load conditions that make that second replica necessary in the first place. The implementation described in this article is part of a production-grade event-driven microservices platform built with Spring Boot 4, Java 21, and Kafka. The full source is available at github.com/Rummy43/ai-microservices-platform.

Every system that stores sensitive data needs keys that never leave a hardened boundary, an audit trail of every use, and rotation that doesn't mean re-encrypting petabytes. On Google Cloud, that's Cloud KMS; on AWS, it's AWS KMS. Both store symmetric and asymmetric keys, both use HSMs, and both are built for envelope encryption. But AWS KMS vs. Google Cloud KMS comes down to the choices that show up in your code: how keys are named, what rotation means, where authorization lives, and whether the service hands you data keys. This article walks through Cloud KMS in Java and then maps it onto AWS KMS. The Cloud KMS Resource Model Cloud KMS organizes keys in a four-level hierarchy under a project: Location. Where the key material lives: a region such as us-east1, a multi-region such as us, or global. Key material stays in the location you pick for its whole lifecycle.Key ring. A named group of keys in one location. It exists for organization and IAM inheritance.CryptoKey. The key you reference in code. It has a purpose (ENCRYPT_DECRYPT, ASYMMETRIC_SIGN, ASYMMETRIC_DECRYPT, MAC, and a few more) and a rotation schedule.CryptoKeyVersion. The actual key material. A key has one or more versions, and for symmetric keys one of them is the primary, the version that new encryptions use. A symmetric key with three versions: new data uses version 3, data written under version 2 still decrypts, and version 1 is on its way out. A key is identified only by its full resource name, projects/my-project/locations/us-east1/keyRings/payments/cryptoKeys/card-data, optionally followed by /cryptoKeyVersions/2. There are no aliases. Nothing can move after it's created: the location is fixed when you create the key ring, and key rings themselves can't be deleted. Each key version also has a protection level: SOFTWARE: keys are protected by Google's software crypto (BoringCrypto, FIPS 140-3 Level 1).HSM: keys live in Cloud HSM, which is FIPS 140-2 Level 3 validated. HSM_SINGLE_TENANT gives you a dedicated HSM.EXTERNAL and EXTERNAL_VPC: Cloud EKM keeps the key material in a supported external key manager that you operate, reached over the internet or a VPC. Google never holds the key. Creating a Key That Rotates Itself The Java client is KeyManagementServiceClient. It uses Application Default Credentials, so on GKE, Cloud Run, or GCE, the workload's service account authenticates with no keys in the code. This creates a key ring and an HSM-backed symmetric key that rotates every 90 days: Java try (KeyManagementServiceClient kms = KeyManagementServiceClient.create()) { KeyRing ring = kms.createKeyRing( LocationName.of("my-project", "us-east1"), "payments", KeyRing.getDefaultInstance()); CryptoKey key = CryptoKey.newBuilder() .setPurpose(CryptoKeyPurpose.ENCRYPT_DECRYPT) .setVersionTemplate(CryptoKeyVersionTemplate.newBuilder() .setAlgorithm(CryptoKeyVersionAlgorithm.GOOGLE_SYMMETRIC_ENCRYPTION) .setProtectionLevel(ProtectionLevel.HSM)) .setRotationPeriod(Duration.newBuilder().setSeconds(90L * 24 * 3600)) // every 90 days .setNextRotationTime(Timestamp.newBuilder() .setSeconds(Instant.now().plusSeconds(90L * 24 * 3600).getEpochSecond())) .build(); kms.createCryptoKey(KeyRingName.parse(ring.getName()), "card-data", key); } The rotation period can be anywhere from one day to 100 years. When it fires, Cloud KMS creates a new version and makes it primary. Earlier versions stay enabled, so old ciphertext still decrypts, and nothing is re-encrypted. To rotate by hand, create a version with CreateCryptoKeyVersion and promote it with UpdateCryptoKeyPrimaryVersion. Automatic rotation applies only to symmetric keys; asymmetric keys are rotated manually. The AWS KMS key rotation process differs in how key material is managed, as described below. Envelope Encryption KMS keys aren't meant to encrypt your data directly. A single Encrypt call accepts at most 64 KiB of plaintext (8 KiB of plaintext plus AAD for HSM keys), and every call is a network round trip. The standard pattern is envelope encryption, also used with AWS KMS envelope encryption: encrypt the data locally with a data encryption key (DEK), then have KMS encrypt (wrap) the DEK with a key encryption key (KEK) that never leaves the service. You store the wrapped DEK next to the ciphertext. Envelope encryption: The payload never goes to KMS; only the 32-byte DEK does. Cloud KMS has no data-key API, so you generate the DEK yourself. Google's guidance is to generate a new DEK every time you write data, never share one across users, and prefer Tink, which packages this whole flow. Doing it by hand shows what's happening: Java CryptoKeyName kek = CryptoKeyName.of("my-project", "us-east1", "payments", "card-data"); ByteString aad = ByteString.copyFromUtf8("tenant=acme"); // must be supplied again to decrypt try (KeyManagementServiceClient kms = KeyManagementServiceClient.create()) { // 1. Generate a fresh DEK locally. Cloud KMS has no GenerateDataKey call. KeyGenerator gen = KeyGenerator.getInstance("AES"); gen.init(256); SecretKey dek = gen.generateKey(); // ... encrypt the payload locally with dek (AES-GCM), then discard dek ... // 2. Wrap the DEK with the KEK, with CRC32C checks against corruption in transit. ByteString dekBytes = ByteString.copyFrom(dek.getEncoded()); EncryptResponse wrapped = kms.encrypt(EncryptRequest.newBuilder() .setName(kek.toString()) .setPlaintext(dekBytes) .setPlaintextCrc32C(crc32c(dekBytes)) .setAdditionalAuthenticatedData(aad) .setAdditionalAuthenticatedDataCrc32C(crc32c(aad)) .build()); if (!wrapped.getVerifiedPlaintextCrc32C() || wrapped.getCiphertextCrc32C().getValue() != crc32c(wrapped.getCiphertext()).getValue()) { throw new IllegalStateException("integrity check failed"); } ByteString wrappedDek = wrapped.getCiphertext(); // store next to the encrypted payload String usedVersion = wrapped.getName(); // the primary version that wrapped it // 3. Later: unwrap. Pass the key, not the version; Cloud KMS picks the version. ByteString unwrapped = kms.decrypt(DecryptRequest.newBuilder() .setName(kek.toString()) .setCiphertext(wrappedDek) .setAdditionalAuthenticatedData(aad) .build()) .getPlaintext(); SecretKey restored = new SecretKeySpec(unwrapped.toByteArray(), "AES"); } static Int64Value crc32c(ByteString data) { return Int64Value.of(Hashing.crc32c().hashBytes(data.toByteArray()).padToLong()); } Encrypt uses the key's current primary version. Decrypt also takes the key, not a version: the ciphertext records which version produced it, so rotation is invisible to readers. The optional CRC32C fields let the server confirm it received your plaintext intact, and let you confirm the ciphertext you got back. The AAD (tenant=acme) is bound to the ciphertext, so one tenant's wrapped DEK can't be swapped into another tenant's record. Disabling, destroying, and deleting Lifecycle operations act on versions, not keys. You disable a version (ENABLED to DISABLED) to stop it from being used, and re-enable it later. There's no switch that disables a whole key; you disable each version. Destroying a version is deliberately slow. DestroyCryptoKeyVersion moves it to DESTROY_SCHEDULED for 30 days by default. The duration is set when the key is created, and an organization policy can enforce a minimum. During that window, RestoreCryptoKeyVersion brings it back in the DISABLED state. After the window, the material is gone and data encrypted by that version can't be recovered. Destroyed versions are no longer billed. Cloud KMS has also added permanent deletion: you can delete key versions and then the key itself, though a deleted key's name can't be reused. Access control is just IAM Cloud KMS has no policy language of its own. Access is ordinary IAM, granted at the organization, folder, project, key ring, or key level and inherited downward. The predefined roles split duties cleanly: roles/cloudkms.admin manages keys but can't encrypt or decrypt.roles/cloudkms.cryptoKeyEncrypterDecrypter, cryptoKeyEncrypter, and cryptoKeyDecrypter can use keys but not manage them.roles/cloudkms.signerVerifier and publicKeyViewer cover asymmetric keys. The operator who rotates keys never sees plaintext, and the service that decrypts can't destroy its key. Grant usage roles on the specific key, not the project. Two more controls round this out. Cloud Audit Logs record admin operations (create, rotate, destroy) automatically, but Encrypt and Decrypt are Data Access logs, which are off by default. If you want a record of every key use, enable them for Cloud KMS. Key Access Justifications, used with Cloud EKM keys, attaches a reason code to every Google request for your key, and your external key manager can refuse it. Bringing your own key To import key material, you create an import job with a wrapping method, such as RSA_OAEP_3072_SHA256_AES_256 or one of the newer post-quantum HPKE methods. The job gives you a public key, and it expires after three days. You wrap your key material with that public key and import it as a new version of a key. For a key that must only ever hold imported material, set importOnly, which means rotation is manual: import new material and make it primary. If you destroy an imported version, you can re-import the same material later. AWS KMS for comparison AWS KMS covers the same ground with a flatter, version-less model: Keys, not versions. A KMS key has a key ID (a UUID), an ARN, and optional aliases such as alias/card-data. There's no hierarchy above it beyond the account and Region.Rotation is hidden. Automatic rotation is off by default for customer managed keys. When enabled, it runs every 365 days by default, configurable from 90 to 2,560 days, and RotateKeyOnDemand allows up to 25 extra rotations per key. The key ID never changes; AWS keeps old material internally and picks it on decrypt.All keys in HSMs. Keys in the default key store live in HSMs validated at FIPS 140-3 Level 3. There's no software tier. CloudHSM key stores and external key stores (XKS) cover the dedicated and hold-your-own-key cases.A data-key API. GenerateDataKey returns a fresh DEK in plaintext along with a copy wrapped under your key, so step 1 and step 2 above are one call.Encryption context. The AWS equivalent of AAD is a map of key-value pairs. Unlike Cloud KMS AAD, it's recorded in CloudTrail and can be used in policy conditions such as kms:EncryptionContext:tenant.Three authorization layers. Every key has exactly one key policy, a resource policy that supports Deny and conditions. IAM policies work only if the key policy enables them, and grants add temporary, programmatic permissions on top.Whole-key lifecycle. DisableKey and ScheduleKeyDeletion act on the whole key. The deletion window is 7 to 30 days, and CancelKeyDeletion backs it out.Replication. Multi-Region keys share the same key ID and material across Regions, and ReplicateKey copies one to a new Region. You choose this at creation; a single-Region key can't be converted. Here's the same envelope flow on AWS: Java Map<String, String> context = Map.of("tenant", "acme"); // logged in CloudTrail, usable in policy try (KmsClient kms = KmsClient.create()) { GenerateDataKeyResponse dataKey = kms.generateDataKey(GenerateDataKeyRequest.builder() .keyId("alias/card-data") .keySpec(DataKeySpec.AES_256) .encryptionContext(context) .build()); byte[] dek = dataKey.plaintext().asByteArray(); // encrypt locally, then discard SdkBytes wrappedDek = dataKey.ciphertextBlob(); // store next to the payload byte[] unwrapped = kms.decrypt(DecryptRequest.builder() .keyId("alias/card-data") // optional for symmetric keys .ciphertextBlob(wrappedDek) .encryptionContext(context) .build()) .plaintext().asByteArray(); } KeyId is optional on Decrypt for symmetric keys because the ciphertext blob identifies the key, but AWS recommends passing it so you don't decrypt with a key you didn't expect. Cloud KMS requires the key name on every Decrypt. Side by side ConceptGoogle Cloud KMSAWS KMSKey identifierResource name projects/…/cryptoKeys/k, no aliasesKey ID, ARN, aliasesVersionsFirst-class CryptoKeyVersion with its own stateHidden behind the key IDProtectionSOFTWARE, HSM (FIPS 140-2 L3), single-tenant HSM, EXTERNAL, EXTERNAL_VPCAll HSM-backed (FIPS 140-3 L3), plus CloudHSM and external key storesMax plaintext per call64 KiB (8 KiB with AAD for HSM keys)4,096 bytesData keysGenerate locally (or with Tink)GenerateDataKeyAuthenticated dataAAD bytes, not loggedEncryption context map, logged and usable in policiesAuto-rotation1 day to 100 years90 to 2,560 days, off by default, plus 25 on-demandDisablePer versionWhole keyDestroyPer version, 30 days scheduled by default, restorableWhole key, 7–30 days, cancellableAccess controlIAM roles on the hierarchy, admin and usage splitKey policy (with Deny) + IAM + grantsUsage loggingData Access audit logs, off by defaultCloudTrail management eventsMulti-regionChoose a multi-region location for the key ringMulti-Region keys, ReplicateKeyBYOKImport job, expires in 3 daysGetParametersForImport, token valid 24 hoursThroughput quota (default)Token-based per region; about 60,000 software operations a minute10,000 to 100,000 symmetric operations a second, by Region What the AWS KMS vs. Google Cloud KMS encryption comparison means in practice: Cloud KMS is simpler to reason about for access. It's one IAM model with roles that separate managing keys from using them. AWS is more expressive: a key policy can Deny, and conditions on encryption context can tie a key's use to a tenant. That power comes with the risk of a key policy that locks out its own administrators.Versions are explicit on Google Cloud. You can disable or destroy just the key material that was compromised and leave the rest working. On AWS you disable or delete the whole key, or rotate and keep the old material.AWS logs every use by default. On Google Cloud, enable Data Access logs for Cloud KMS on day one, or you won't have a record of who decrypted what.Plan for quotas. AWS gives each account and Region a large shared pool of cryptographic operations. Cloud KMS's default is far smaller per project and region, so cache unwrapped DEKs for hot reads where your threat model allows, and request a quota increase before launch. Both quotas are adjustable.Both want envelope encryption. Neither service is built to encrypt bulk data directly. On Google Cloud, Tink makes the local half trivial. On AWS, the AWS Encryption SDK plays the same role. Conclusion Google Cloud KMS treats keys as resources in your project hierarchy: versions are visible, protection level is chosen per key, and IAM is the whole authorization story. That's easy to audit and gives fine-grained control over key material, as long as you turn on Data Access logs and size your quotas. AWS KMS hides versions and puts authorization in a per-key policy language that's more expressive, especially with encryption context, and its data-key API saves a step. These differences are worth considering when reviewing AWS KMS best practices alongside Google Cloud KMS key management. On both clouds, the envelope pattern is the common ground: generate a DEK, wrap it, store it next to the data, and let KMS guard only the key that guards everything else.

When I work with Claude or Codex, I often discuss software architecture with them. I’ve noticed a recurring pattern: both are quick to suggest designs that are more complex than the problem currently requires. The proposals are more general, more extensible, and more “future-proof.” There’s something funny about that last word. The models don’t know what the future of my software will look like. They’re applying familiar engineering advice: anticipate change, make components reusable, leave room to extend the design. Sometimes that’s right. Often, it means building things I haven’t asked for and may never need. What Is the YAGNI Principle in Software Engineering? In those moments, my job is to cut off the extra features. Software engineering has a name for that instinct: YAGNI — “You Aren’t Gonna Need It.” Taken literally, YAGNI sounds like a claim about the future: this feature will never be needed. But that’s stronger than what we can usually know. In practice, it means something more modest: we probably don’t need this now. If we discover that we need it later, we’ll deal with it then. We’ll cross that river when we get to it — and build a bridge when there’s a reason to cross. But why not build the bridge ahead of time? Imagine building a road toward a river, then discovering that you need to cross it. If you already built the bridge, the road may connect to it neatly. You avoid stopping to redesign the approach or rebuild part of the road. Building ahead can make later work easier and keep the pieces compatible. The trouble is that the bridge has a cost before it provides any value. Someone has to build it. Someone has to drag it through the desert to the river. In software, that means writing the code, fitting it into the design, testing it, explaining it, and maintaining it as the project changes around it. The bridge might save us work later, but carrying it there is work now. How AI Coding Tools Change the Cost of Building Software For most of software’s history, that calculation was shaped by the cost of human programming. Designing and implementing extra capability took time. So did reviewing and maintaining it. A feature that might be useful someday had to justify the present cost of building it. Often it couldn’t. YAGNI was a practical defense against paying that cost too early. AI coding tools change part of the calculation. They can make it much cheaper to produce code. A generalized interface, an extra implementation, or a future extension point may take far less time to generate than it once did. If building the bridge becomes cheap, perhaps it makes sense to build it early. The Hidden Costs of AI-Generated Code But “cheaper to write” does not mean “free to own.” Generated code still has to fit the actual design. It can introduce more concepts, more paths through the system, more tests, and more behavior that a developer must understand. It can make the current problem harder to see. And if the imagined future never arrives, the project still carries the bridge. That leaves us with a question, not a new rule: when AI reduces the cost of building ahead, how much should that change our understanding of YAGNI? We shouldn’t answer using only the instinct we developed when every line of code was expensive to write. But we also shouldn’t assume that cheap generation makes speculative features harmless. The useful question is not simply, “Could this be useful later?” Almost anything could. It is: “Is the likely future benefit worth the cost of adding and maintaining this now?” That cost has changed. Our judgment should change with it.

By Peter Verhas CORE

RabbitMQ is a message broker that applications use to communicate asynchronously. A publisher sends a message to an exchange with a routing key. The exchange compares the routing key with the binding keys and routes the message to all matching queues, which deliver it to consumers. A successful publish means RabbitMQ accepted the message at the exchange. It does not always mean that a queue received it. If no binding matches, the message is unroutable and may be dropped or returned to the publisher, depending on the publisher settings. RabbitMQ Exchange Types An exchange decides where a message should go. Its type determines how it makes that decision: Direct: Routes when the message routing key exactly matches the binding key.Topic: Uses dot-separated keys and wildcards. * matches one word; # matches zero or more words.Headers: Routes using message headers rather than the routing key.Fanout: Sends a copy to every bound queue and ignores the routing key. The default exchange is a special direct exchange. RabbitMQ automatically binds each queue to it using the queue name as the routing key. Issue We deployed RabbitMQ in a three-node Kubernetes environment. Several applications use it to broadcast messages to multiple consumer applications. Application teams create topic exchanges and bind queues with binding keys. For example, we created a topic exchange as shown below: Publishing m.event.provisioning.started to the exchange should route a copy to every queue whose binding pattern matches it. In some environments, messages published to topic exchanges were not routed to queues even when the expected routing key was used. We inspected the topology using the management console and rabbitmqctl commands. These commands inspect the configured exchanges, queues, and bindings in the selected virtual host: Shell rabbitmqctl list_exchanges -p <vhost> name type durable arguments This command will list all exchanges in rabbitmq rabbitmqctl list_queues -p <vhost> name durable type messages messages_ready consumers This command will list all queues declared in rabbitmq rabbitmqctl list_bindings -p <vhost> source_name source_kind destination_name destination_kind routing_key arguments This command will list exchange, queue and bindings in rabbitmq The configured topology looked correct. However, in one environment, when we tried to publish a message using the RabbitMQ management UI, we received the message: Message published, but not routed. Impact In a field deployment, this will impact multiple customer workflows. Customers can experience the failure even though the publisher reports a successful publish. A provisioning or configuration workflow can remain in a pending state because the service waiting for the event never receives it.Different services can hold different views of the same customer operation. For example, one service may record that a change started while another service never receives the event needed to update inventory, status, or audit information.Customer onboarding, device activation, policy changes, or software upgrades can be delayed or fail when an expected event is lost between services.Operators may retry the operation or manually intervene. That can increase recovery time and, if the original event is later delivered, create duplicate processing or confusing status changes.The failure can look intermittent because only some bindings or queues are affected. The management UI may still show a healthy declarative topology, which makes the issue harder to identify as a broker-routing problem. Root Cause RabbitMQ persists cluster metadata, including topology such as exchanges, queues, and bindings. During startup, the broker loads that metadata into runtime structures used for message routing. We observed this in a Mnesia-based deployment. Mnesia is a distributed internal database used by RabbitMQ to store cluster metadata. And RabbitMQ loads topic exchange configuration into these internal tables: rabbit_topic_trie_edge and rabbit_topic_trie_binding. rabbit_topic_trie_edge This internal runtime table represents topic-binding segments as a trie. For a binding such as m.event.provisioning.#, the dot-separated segments form edges, and each edge refers to its parent. The root edge has no parent. The command to list the contents of the rabbit_topic_trie_edge table is: rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_edge)'. Below is sample output: Shell rabbitm@rabbitmq-0:/$ rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_binding).' [{topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<48,148,14,244,108,180,212,249,183,91,230,141,99,163,177,115>>, {resource,<<"/">>,queue,<<"provisioning-events">>}, []}, const}, {topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<67,242,234,144,205,89,103,148,212,234,75,22,120,194,157,225>>, {resource,<<"/">>,queue,<<"audit-events">>}, []}, const}, {topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<205,242,142,92,26,182,127,217,6,198,131,59,35,208,63,252>>, {resource,<<"/">>,queue,<<"exact-event">>}, []}, const}] root@my-rabbit:/# rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_edge).' [{topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, root,"m"}, <<217,70,178,13,233,108,102,215,235,100,37,39,215,68,241,133>>}, {topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<210,2,101,2,238,34,177,160,57,102,64,37,37,22,188,199>>, "#"}, <<48,148,14,244,108,180,212,249,183,91,230,141,99,163,177,115>>}, {topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<210,2,101,2,238,34,177,160,57,102,64,37,37,22,188,199>>, "completed"}, <<205,242,142,92,26,182,127,217,6,198,131,59,35,208,63,252>>}, {topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<217,70,178,13,233,108,102,215,235,100,37,39,215,68,241,133>>, "event"}, <<245,28,154,53,247,20,217,197,203,52,13,103,28,112,252,226>>}, {topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<245,28,154,53,247,20,217,197,203,52,13,103,28,112,252,226>>, "#"}, <<67,242,234,144,205,89,103,148,212,234,75,22,120,194,157,225>>}, {topic_trie_edge, {trie_edge, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<245,28,154,53,247,20,217,197,203,52,13,103,28,112,252,226>>, "provisioning"}, <<210,2,101,2,238,34,177,160,57,102,64,37,37,22,188,199>>}] rabbit_topic_trie_binding This internal table associates the topic exchange and destination with the final edge in the topic trie. The command to list the contents of the rabbit_topic_trie_binding table is: rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_binding).' Below is sample output: Shell rabbit@rabbitmq-0:/$ rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_binding).' [{topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<48,148,14,244,108,180,212,249,183,91,230,141,99,163,177,115>>, {resource,<<"/">>,queue,<<"provisioning-events">>}, []}, const}, {topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<67,242,234,144,205,89,103,148,212,234,75,22,120,194,157,225>>, {resource,<<"/">>,queue,<<"audit-events">>}, []}, const}, {topic_trie_binding, {trie_binding, {resource,<<"/">>,exchange,<<"provisioning.exchange">>}, <<205,242,142,92,26,182,127,217,6,198,131,59,35,208,63,252>>, {resource,<<"/">>,queue,<<"exact-event">>}, []}, const}] We developed a diagnostic utility that reconstructed binding paths from these internal runtime tables and compared them with the topology returned by rabbitmqctl. The comparison showed that some configured bindings were not correctly represented in the runtime routing state. How We Investigated the Problem We first treated the symptom as a normal binding problem. We checked that the publisher and the consumer were using the same virtual host, confirmed that the exchange was a topic exchange, and compared the binding keys character by character. We also checked the queue consumer count and published a test message directly from the management UI. The message was accepted, but the UI reported Message published, but not routed. Next, we used rabbitmqctl list_exchanges, rabbitmqctl list_queues, and rabbitmqctl list_bindings to capture the topology. Those commands showed the exchange, queues, and bindings that we expected. We repeated the same checks after connecting to different RabbitMQ pods and did not find a spelling, virtual-host, or wildcard mismatch. At that point, we stopped changing application routing keys and compared the configured topology with the broker's runtime topic-trie state. The configured binding was present in the management output, but the corresponding path was incomplete in the runtime tables on the affected deployment. This explained why recreating the binding changed the behavior without changing the application code. The publisher confirmed, and the mandatory publish option helped us narrow the problem further. A publisher confirm told us that RabbitMQ accepted the publish operation. The mandatory option made an unroutable message visible to the publisher, but it did not tell us which intended consumer was missing. We still had to verify delivery at each expected queue. What the Publish Result Actually Tells Us RabbitMQ accepts a publish at the exchange before it evaluates all of the topic bindings. For a topic exchange, the routing key is split into dot-separated words and matched against each binding pattern. A single-word wildcard (*) matches exactly one segment, while the multi-word wildcard (#) can match zero or more segments. A small difference such as m.event.provisioning.started versus m.events.provisioning.started is enough to produce no match. This was the important distinction in our investigation: a successful publish was not proof of delivery. When the publish is not mandatory, a message that matches no queue can be discarded after the exchange accepts it. In our case, the binding appeared in the declarative topology but was not fully available to the runtime routing path, so the result looked like an ordinary unroutable message even though the management view appeared correct. Changes We Made After the Incident We kept the exchange, queue, and binding declarations idempotent, but added an explicit verification step after broker restarts, pod rescheduling, and upgrades. We also added a utility that reads the rabbit_topic_trie_edge and rabbit_topic_trie_binding tables, reconstructs the binding paths, and compares them with the exchange, queue, and binding information returned by rabbitmqctl. This gives us a direct way to identify a binding that exists in the configured topology but is missing or incomplete in the runtime topic trie. In addition, we enabled detailed metrics at the exchange level so that we can see the unrouted message count at each exchange level. We correlate those metrics with the virtual host, routing key, queue, node, and timestamp captured by the utility. The combination of runtime-table comparison and exchange-level metrics makes it easier to determine whether the problem is a routing mismatch, a missing runtime binding, or a consumer that received the message but did not process it. We now use the application logs, exchange-level metrics, and the utility output as the first evidence after restarts, pod rescheduling, and upgrades. We correlate an actual failed event with the expected exchange and binding path instead of injecting another message into a customer environment. The expected topology, comparison utility, and diagnostic commands are kept with the deployment so that an operator can compare the intended state with the live state during an incident. Observed Workaround For each affected binding, we captured the virtual host, exchange, queue, binding key, and arguments, then deleted and recreated the binding through the RabbitMQ API. We compared the runtime topic-trie state with the topology reported by rabbitmqctl, retried the affected workflows, and confirmed delivery at every expected queue. This was the workaround for our deployment, not a universal fix. Do not edit Mnesia or ETS files directly; internal tables are version-specific and direct changes can leave the cluster inconsistent. Lessons Learned Management and CLI topology output are necessary first checks, but they do not replace an end-to-end routing test. After restarts, pod rescheduling, or upgrades, publish representative routing keys and verify that every expected queue receives the message. Publisher confirms, and mandatory publishing can also make unroutable messages visible to applications. We observed this issue only with topic exchanges.

Architecture Review Boards (ARBs) have traditionally evaluated application architectures against key enterprise criteria such as security, scalability, integration, resilience, supportability, and compliance. With the rapid adoption of artificial intelligence (AI), machine learning (ML), and generative AI technologies, organizations are increasingly deploying intelligent solutions that introduce new architectural, operational, ethical, and regulatory challenges. As a result, ARBs must expand beyond traditional review dimensions to assess AI-specific considerations, including data governance, model lifecycle management, explainability, bias mitigation, AI security, operational monitoring, and responsible AI practices. The evolution of ARBs is essential to ensure AI solutions deliver business value while maintaining trust, compliance, accountability, and enterprise risk controls. The objectives of an ARB for AI-based solutions are to: Enable responsible AI innovationManage business, operational, and regulatory risksProtect enterprise and customer dataEstablish governance and accountabilityImprove the success rate of AI initiativesEnsure compliance with organizational and regulatory requirements The table below highlights the key differences between a traditional Architecture Review Board and an AI-focused Architecture Review Board. areatraditional arbAI Solutions ARB Focus Area Application architecture quality, stability, and alignment with enterprise standards. Governance of AI solutions while ensuring business value and risk management. Review Architecture Components Applications, integrations, databases, infrastructure, APIs, and middleware. AI models, LLMs, prompts, vector databases, AI agents, training data, MLOps / LLMOps platforms, and AI services. Security Network security, application security, IAM, encryption, and vulnerability management. Traditional security + prompt injection protection, model security, adversarial attacks, AI supply chain security, and data leakage prevention. Data Governance Data quality, ownership, retention, and regulatory compliance. Data lineage, training data quality, AI data usage rights, consent management, data sovereignty, and model input/output governance. Compliance Industry standards, regulatory requirements, and enterprise policies. AI regulations, ethical AI principles, transparency requirements, and responsible AI frameworks. Risk Operational, technical, and cybersecurity risks. Hallucinations, bias, misinformation, model drift, ethical risks, and reputational risks. Explainability Not a major review criterion. Critical requirement to ensure AI decisions and outputs can be understood, justified, and audited. Bias and Fairness Outside review scope. Bias detection, fairness controls, and mitigation mechanisms. Testing & Validation Functional, performance, integration, security, and UAT testing. Prompt testing, adversarial testing, hallucination assessment, and AI output validation. Operations & Monitoring Application monitoring, logging, incident management, and SLAs. Performance, drift detection, output quality, token consumption, and AI safety monitoring. Lifecycle Management Software development lifecycle (SDLC), release management, and application support. Model lifecycle management, retraining strategy, model versioning, LLM updates, and MLOps/LLMOps governance. Accountability Application owner and business sponsor accountability. Accountability for AI outcomes, model ownership, risk ownership, and human-in-the-loop oversight. Decision Criteria Security, scalability, resilience, integration, supportability, and compliance. Explainability, trustworthiness, fairness, governance, responsible AI, and business risk controls. Success Measures System reliability, performance, availability, and compliance. Business value realization, responsible AI adoption, model accuracy, user trust, regulatory adherence, and risk reduction. Prompt Governance Not Applicable Prompt design, validation, security, and lifecycle management Model Selection Not Applicable Evaluation of model suitability, cost, accuracy, and risk Human Oversight Limited Human-in-the-loop review and approval processes AI Ethics Rarely Reviewed Fairness, transparency, accountability, and societal impact Vendor Lock-In Infrastructure focused AI provider dependency and model portability Cost Governance Infrastructure and licensing Token consumption, inference cost, and model hosting costs Responsible AI Limited Mandatory governance and compliance requirements The shift in ARB assessment perspectives can be summarized through the following key questions: Below is a six-pillar AI ARB framework for solution reviews and assessments. Business value assessment: Organizations deploy AI because it is technologically attractive rather than business-driven. The ARB team evaluates revenue growth, cost optimization, productivity improvement, customer experience enhancement, and decision-making effectiveness. Data governance and privacy: Data is the most critical asset in any GenAI solution. Poor data governance can expose sensitive information and create significant compliance risks. The ARB team evaluates data classification, data protection, and regulatory compliance.Security architecture review: GenAI systems introduce new attack vectors not commonly found in traditional enterprise applications. The ARB team evaluation areas include identity management integration, role-based access control, API gateway protection, encryption at rest and in transit, network isolation, key management, and security monitoring.Responsible AI governance: Responsible AI should be a mandatory review category within every GenAI ARB process. The objective is to ensure AI systems are fair, transparent, accountable, and trustworthy. The ARB team evaluates fairness, transparency, accountability, and explainability. Architecture and integration assessment: The ARB team evaluation areas on solution architecture, integration strategy, RAG architecture, and multi-model strategy.Operations, monitoring, and FinOps: GenAI projects succeed technically but struggle operationally after deployment. The ARB team evaluates business, technical, and financial metrics. Below are the best practices for enterprise ARBs reviewing GenAI solutions: Establish a dedicated AI review track within the ARB.Mandate Responsible AI assessments.Require data classification and privacy reviews.Standardize AI architecture reference patterns.Define AI monitoring and observability standards.Include Legal, Security, Data Governance, and EA stakeholders.Adopt a FinOps approach for AI cost governance.Establish a continuous AI governance process rather than a one-time approval activity. Conclusion Future ARBs are no longer responsible solely for evaluating system architecture. The review board must become a multidisciplinary governance body capable of balancing innovation, risk, compliance, and ethics. Organizations that evolve their ARB practices to address generative AI-specific concerns will be better positioned to scale AI adoption while maintaining trust, transparency, and regulatory compliance.

By Ravi Kiran Mallidi CORE

Embedded and IoT devices operate in environments where every kilobyte of memory and every millisecond of response time matters. Yet most of them still rely on C or C++, languages powerful enough for low-level control but notorious for memory-safety flaws that can crash systems or expose them to security vulnerabilities. The issue is so widespread that around 70% of Microsoft’s and Google’s reported vulnerabilities stem from memory-safety issues. For connected devices running critical operations, such instability isn’t an option. However, a structured C++ to Rust migration addresses the root cause by replacing unsafe memory handling with compile-time guarantees and predictable real-time performance. Embedded environments leave little room for trial and error; therefore, experience and a clear roadmap make a big difference. For this purpose, companies often work with teams skilled in Rust development services and familiar with embedded software development. With the right expertise, you can introduce new Rust components gradually and safely without interruptions to the existing IoT ecosystem. Now, let’s discover how to migrate from C/C++ to Rust, review the migration challenges and best practices to mitigate them, and analyze a success story of practical Rust migration for embedded systems. Why Consider Migrating From C/C++ to Rust? Modern embedded systems, especially IoT-based medical devices, put heavy pressure on safety, energy use, and long-term maintenance. Rust handles these demands better than traditional C or C++ thanks to its built-in memory safety, reliable concurrency model, and predictable real-time behavior. When organizations consider a C++ to Rust migration, these advantages quickly become deciding factors for modernizing connected devices. They also translate into more stable products and lower life cycle costs: Reduced security risks: Rust’s memory safety drastically lowers vulnerabilities, meaning fewer patches and less downtime for IoT systems.Lower long-term maintenance costs: Rust reduces bugs that emerge later in production, leading to fewer expensive fixes after deployment.Improved reliability: With Rust, IoT devices crash less often, which means more uptime and fewer disruptions for end-users.Better performance: Rust’s efficiency can lead to lower energy consumption, extending the life of battery-powered IoT devices and reducing energy costs.Faster time-to-market for updates: Rust’s safe concurrency allows quicker development of new features without compromising stability, which means you can roll out improvements to your IoT solutions faster.Future-proof technology: Rust’s growing popularity means a sustainable ecosystem, which ensures that your IoT product stays modern and well-supported for years to come.Increased development velocity: Rust reduces debugging time and makes the code easier to reason about, speeding up onboarding and increasing overall developer productivity. Three Common Myths About Rust Migration Many organizations hesitate to begin migrating embedded systems from С or C++ to Rust because they expect major risks or disruptions. In reality, most of these concerns stem from common misconceptions rather than technical limitations in embedded software development. Understanding what Rust migration actually involves helps teams make confident, well-informed decisions. Let’s review the three most common misconceptions. Myth 1: “We’ll Have to Rewrite Everything” You don’t need a full rewrite to begin a C++ to Rust migration. Many teams start small, moving one or two isolated modules that bring the most maintenance pain. Using the Foreign Function Interface (FFI), Rust can work alongside C or C++ within the same firmware. This feature makes it possible to modernize gradually while the rest of the system stays live. Myth 2: “Switching to Rust Will Interrupt Delivery” Migration can happen without stopping production. With a structured approach, Rust modules are added one by one while existing CI/CD pipelines continue to run. Each new piece is tested, deployed, and benchmarked before the next one is introduced. The development cycle continues as usual; only the language inside changes. Myth 3: “We’ll Have to Replace the Entire Engineering Team” Most C or C++ developers adapt to Rust faster than expected. The languages share similar low-level logic, which makes the learning curve manageable with proper mentoring and review. Once trained, the same team can handle both Rust and C++, maintaining delivery speed while building long-term expertise. Three Ways of C/C++ to Rust Migration There’s no one-size-fits-all approach to C++ to Rust migration. The right strategy depends on how your system works today, how critical uptime is, and what level of modernization you’re aiming for. When adopting Rust for IoT and embedded systems, companies typically choose one of three paths: gradual integration, full rewrite, or hybrid migration. 1. Gradual Integration This path is ideal for safety-critical or continuously running devices. Instead of rewriting everything, the developers introduce Rust gradually — one module at a time. Typical starting points include telemetry handlers, utility modules, or background services that can be tested in isolation. The existing C/C++ software keeps running while new Rust components are integrated and validated. Over time, as more modules are proven stable, Rust takes over a larger share of functionality. 2. Full Rewrite If you’re launching a new product or phasing out outdated firmware, a full rewrite may be the most efficient route. The development starts with Rust from day one. It covers firmware logic, communication layers, and testing infrastructure. This scenario removes legacy constraints entirely and lets your team build clean architectures without legacy dependencies, design for memory safety from the start, and use Rust’s concurrency model to its full advantage. A full rewrite requires a skilled team, proven toolchains, and time for onboarding, but it also gives you complete control over the system’s design. Yalantis supports such transformations through end-to-end Internet of Things solutions, helping businesses develop secure, scalable, and future-ready connected products. 3. Hybrid Approach Many embedded systems can’t be fully rewritten due to vendor SDKs, licensed components, or hardware dependencies. In those cases, Rust and C/C++ coexist. Engineers connect both through FFI, so each language can do what it does best. C++ handles hardware-level operations, while Rust manages business logic, safety-critical processes, or communication layers. This model works particularly well for medical devices, industrial equipment, and long-lifecycle IoT platforms, where reliability and compliance matter just as much as modernization speed. Challenges and Best Practices by Yalantis for C/C++ to Rust Migration Before starting a full-scale migration, it’s important to understand the technical and architectural risks involved. Each challenge that appears during the embedded C++ to Rust migration can be handled through structured engineering practices. Below is an overview of key challenges in migrating embedded firmware from C++ to Rust and the Rust migration best practices Yalantis uses to keep modernization predictable and low-risk. Challenge 1: Maintaining Compatibility With Existing C/C++ Code Legacy firmware often has C++ modules woven tightly together. You can’t rewrite everything at once without risking device instability or unexpected regressions. Challenge 2: Operating Under Hardware and Microcontroller Limitations Most embedded devices run on minimal RAM, limited storage, and low CPU frequencies. A rushed migration can overload the hardware or increase latency. Challenge 3: Ensuring Deterministic Behavior in Real-Time Systems IoT and embedded applications often have strict timing and reliability requirements. Even slight delays in execution can affect safety, user experience, or device coordination. Challenge 4: Integrating With External Libraries and APIs Embedded firmware relies on vendor drivers, communication stacks, and low-level APIs. Poor integration planning can cause instability or break device compatibility. Conclusion Modernizing embedded and IoT systems requires predictable performance, strong security guarantees, and code that stays maintainable as the product evolves. Traditional C/C++ firmware often struggles with these demands. Rust solves many of these challenges upfront with built-in memory safety, reliable concurrency, deterministic execution, and a modern development ecosystem that reduces defects and speeds up delivery. A well-planned C/C++ to Rust migration strengthens product stability, lowers lifecycle costs, and gives you a foundation that scales with new device architectures and connectivity needs.

Traditionally, developers use different query optimization techniques, such as complex conditional looping and procedure branching, and query hints like OPTION (RECOMPILE). The query hint option sometimes leads to high CPU utilization. SQL Server 2022 (Compatibility Level 160) introduced a new feature called Parameter Sensitive Plan (PSP) Optimization as part of the core capabilities of the Intelligent Query Processing (IQP) feature family. PSP optimization addresses parameter sniffing at the engine level by allowing a single parameterized query to generate and maintain multiple active execution plans in the plan cache simultaneously, tailored to different data volume boundaries. How It Works: Mechanics, Dispatchers, and Variants Instead of forcing a single execution plan into cache for all parameter invocations, the SQL Server 2022 Query Optimizer uses data distribution statistics to recognize parameter sensitivity automatically during initial compilation. How to Detect Parameter Sensitivity During compilation, the optimizer evaluates predicates referencing parameter variables. It checks the associated column statistics histogram. If the statistics indicate significant data skew across histogram steps, the query is flagged as PSP-eligible. When a query qualifies, the engine creates a Dispatcher Plan. The dispatcher plan contains the logic needed to evaluate incoming runtime parameter values against pre-calculated boundary ranges. It acts as an internal router rather than a direct execution plan. The optimizer divides the parameter data distribution spectrum into distinct boundary buckets (typically low, medium, and high cardinality ranges) based on the histogram: Boundary calculation: The engine calculates threshold values using histogram step densitiesVariant generation: As different parameter values are executed, the dispatcher maps the runtime value to its corresponding boundary range and compiles or reuses a specific Variant Plan associated with that range. When reviewing execution plans in SQL Server Management Studio (SSMS), PSP-optimized queries display distinct Query Plan XML attributes under the Dispatcher and Statement Plan nodes, referencing variant identifiers (e.g., PlanVariantID="1" vs. PlanVariantID="2") Demonstrating PSP Optimization in Action The following example uses customers from the WideWorldImporters database. In the sample data, CustomerID = 90 has 150 orders, while CustomerID = 1060 has only 4. To make the execution-plan difference easier to observe, we create a small demonstration table that preserves this skew at a larger scale. SQL USE WideWorldImporters; GO DROP TABLE IF EXISTS dbo.PSP_OrderDemo; GO CREATE TABLE dbo.PSP_OrderDemo ( OrderID BIGINT IDENTITY PRIMARY KEY, CustomerID INT NOT NULL, OrderDate DATE NOT NULL, OrderAmount MONEY NOT NULL, Padding CHAR(200) NOT NULL ); GO CREATE INDEX IX_PSP_OrderDemo_CustomerID ON dbo.PSP_OrderDemo(CustomerID); GO Populate one customer with approximately 500,000 rows and the rare customer with only a few rows: -- Common customer INSERT dbo.PSP_OrderDemo (CustomerID, OrderDate, OrderAmount, Padding) SELECT TOP (500000) 90, DATEADD(DAY, -(ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) % 365), GETDATE()), 100.00, REPLICATE('X',200) FROM sys.all_objects a CROSS JOIN sys.all_objects b; GO -- Rare customer INSERT dbo.PSP_OrderDemo (CustomerID, OrderDate, OrderAmount, Padding) SELECT TOP (4) 1060, GETDATE(), 100.00, REPLICATE('X',200) FROM sys.all_objects; GO UPDATE STATISTICS dbo.PSP_OrderDemo WITH FULLSCAN; GO Create a parameterized stored procedure: CREATE OR ALTER PROCEDURE dbo.GetOrdersByCustomer @CustomerID INT AS BEGIN SELECT OrderID, CustomerID, OrderDate, OrderAmount, Padding FROM dbo.PSP_OrderDemo WHERE CustomerID = @CustomerID; END; GO Enable Actual Execution Plan in SSMS with Ctrl+M, then first test with PSP disabled. Because the rare value is compiled first, SQL Server may cache a plan optimized for four rows and reuse it when the same procedure returns approximately 500,000 rows. This illustrates the traditional parameter-sniffing problem. SQL ALTER DATABASE WideWorldImporters SET COMPATIBILITY_LEVEL = 160; GO ALTER DATABASE SCOPED CONFIGURATION SET PARAMETER_SENSITIVE_PLAN_OPTIMIZATION = OFF; GO ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE; GO SET STATISTICS IO, TIME ON; EXEC dbo.GetOrdersByCustomer @CustomerID = 1060; -- Rare EXEC dbo.GetOrdersByCustomer @CustomerID = 90; -- Common GO Because the rare value is compiled first, SQL Server may cache a plan optimized for four rows and reuse it when the same procedure returns approximately 500,000 rows. This illustrates the traditional parameter-sniffing problem. Now enable PSP and repeat the workload: SQL ALTER DATABASE SCOPED CONFIGURATION SET PARAMETER_SENSITIVE_PLAN_OPTIMIZATION = ON; GO ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE; GO EXEC dbo.GetOrdersByCustomer @CustomerID = 1060; EXEC dbo.GetOrdersByCustomer @CustomerID = 90; GO In the PSP-enabled plans, opening the execution plan XML should also reveal QueryVariantID and PLAN PER VALUE, confirming that SQL Server is maintaining separate plans for different cardinality ranges. The Parameter Sniffing Decision Tree When diagnosing and resolving parameter sniffing issues in production, follow this step-by-step decision framework: Summary of Best Practices Avoid blanket global fixes: Resist the urge to disable parameter sniffing database-wide via ALTER DATABASE SCOPED CONFIGURATION SET PARAMETER_SNIFFING = OFF. Disabling it globally starves steady-state OLTP queries of optimal compilation behavior. Prioritize targeted query hints: Apply OPTION (RECOMPILE) for complex reporting queries run infrequently, and OPTION (OPTIMIZE FOR ...) for high-throughput transactional procedures with well-understood parameters.Establish baseline guardrails in Query Store: Leverage Query Store runtime statistics to track standard deviation spikes in avg_cpu_time and avg_logical_io_reads. Use plan forcing as an immediate temporary fix during active site incidents.Upgrade to compatibility level 160 where feasible: Capitalize on IQP features in SQL Server 2022 to handle non-uniform data distributions natively without littering codebases with query hints. Conclusion Parameter sniffing is not an engine defect. SQL Server’s default mechanism for maximizing plan reuse and minimizing compilation overhead. While skewed data distributions meet volatile execution requirements, captured plans can severely compromise application performance. To conclude, SQL Server 2022’s Parameter Sensitive Plan (PSP) optimization introduces welcome automation; enterprise database systems, at the same time, still require a deliberate, structured approach to query tuning.

Key Takeaways Role-based access control (RBAC) and content-tagging systems check whether a requester's credentials match a label attached to data. That check has nothing to say about a derived, cached result — a plain number with no tag of its own.A cached metric like churn_rate = 0.114 can be computed from a query that touches a restricted column (income) and then be served to a department that was never allowed to see that column, without a single existing access control catching it.The fix is to gate retrieval on the result's derivation graph — the tables, columns, and filters that produced it — rather than on a tag attached after the fact.In benchmark testing, this reduced a structural cross-department leak rate of 18.8%–25.5% to zero, at a real, measurable cost: roughly 13 additional query executions per 100 requests, and a gate-check predicate costing 0.09 microseconds.The mechanism does not stop inference attacks, a compromised memory store, or collusion — it closes one specific, previously unguarded gap in agent-memory retrieval. A Clean Integer That Still Leaks a Restricted Column Take a concrete scenario. A Finance agent runs a query that joins a transactions table against a customer_pii table — which includes an income column — to compute a churn_rate for customers who have gone quiet in the last 90 days. It gets back a single float: 0.114. Nothing about that number looks sensitive. No names, no account numbers, no raw income figures. The agent caches the result in shared memory so the next agent that asks for churn_rate does not have to recompute it. A week later, a Marketing agent asks the same shared memory for churn_rate and gets 0.114 back instantly, with no query executed. Marketing was never granted access to income. Its role-based access-control (RBAC) entry explicitly blocks that column. But nothing in this transaction touched income directly — Marketing did not query the column, did not see a row containing it, and did not request anything with "income" in the name. It asked for a key performance indicator (KPI) by name and got a float back. By any access-control system running in production today, that transaction is clean. And it is a policy violation: the number 0.114 was derived from a computation Marketing was never authorized to run. This is the finding at the center of a paper co-authored with Sudhir Vissa, "Lineage-Aware Memory Governance: A Derivation-Gated Framework for Privacy-Preserving Column-Level Access Control in Enterprise AI Agents," accepted at IEEE Access (DOI: 10.1109/ACCESS.2026.3730363). This article walks through the engineering mechanics of the failure and the fix, using the worked example and architecture diagram from the paper. The Access Check Every System Runs — and the One It Doesn't Strip away the vendor language, and every access-control mechanism guarding a piece of data — RBAC, attribute-based access control (ABAC), content tagging, ownership metadata — runs some version of the same predicate at retrieval time: Shell allow(requester, object) := tag(object) ⊆ permissions(requester) The object carries a label — a sensitivity classification, an owner, a role requirement — and the check asks whether the requester holds a credential matching that label. Apache Ranger and Databricks Unity Catalog enforce this at the column level for raw queries: they intercept a SELECT statement and check whether the requesting principal is allowed to read the specific columns named in it. That works well for exactly the case it was built for: stopping an unauthorized read at the point of query execution. Shared agent-memory systems — MemGPT, Zep, A-MEM, and the newer governed-memory architectures being built for enterprise deployments — extend the same pattern to cached results. A memory entry gets an owner, maybe a sensitivity tag, maybe a role requirement, and retrieval is gated on whether the requester's credentials match. This is a reasonable design, and it is also where every one of these systems stops. The problem is that churn_rate = 0.114 has no tag to check. It is a scalar. Nobody attached a sensitivity classification to it, because nothing about the value is sensitive — a churn rate cannot be reverse-engineered into an income figure. The column-level access-control list (ACL) at the database layer did its job perfectly: it verified that Finance was allowed to touch income when it ran the join. But that check fired once, at write time, for the agent that computed the number. It has nothing left to say about the agent that retrieves the cached result afterward. By the time 0.114 sits in shared memory, the access-control moment has already passed, and the object in front of the second access-control check carries no trace of the columns that produced it. So the predicate every current system checks — does the requester's tag match the object's tag — answers the wrong question for this failure mode. The question that actually determines safety is: Shell safe(requester, result) := columns_touched(result.derivation) ⊆ permissions(requester) That is not a check on the object. It is a check on the object's derivation — the tables it joined, the columns it read from each of them, and the filter predicates that shaped it. No system in the current agent-memory landscape records that, which means none of them can evaluate it. A Worked Example: How Finance's Churn Rate Leaks to Marketing Here is the derivation graph from the paper — Figure 1 — showing exactly what has to be tracked to make that second check possible. Read it left to right. Three source tables feed a join: transactions, customers (with customer_id and signup_date), and customer_pii (with last_txn_date and income). The join step applies a filter — last_txn_date < now()-90d AND income > 20000 — and produces churn_rate = 0.114, tagged with the producing department and a time-to-live (TTL) epoch for eviction. The part that matters is the dashed box on the right. Two values are computed once, at the moment the result is cached, and stored alongside it. The sensitivity tag, S(a), is the union of every column touched across every step of the derivation, intersected with the organization's registry of sensitive columns: Plain Text S(a) = (∪ᵢ Cᵢ) ∩ SensitiveColumnRegistry For this cached entry, S(a) = {income}. It does not matter that income never appears in the output — the tag is computed from what was read, not from what was returned. The definition hash, H(a), is a fingerprint of exactly how the metric was computed: the sorted table names, the sorted column set, and the filter predicate, hashed and truncated to 12 characters: Plain Text H(a) = SHA256(sort(tables) || sort(∪ᵢ Cᵢ) || φ)[0:12] This second value is not for access control. It catches a different, related failure: two departments computing a metric with the same name through different logic — a 90-day churn window versus a 60-day window, say. If two cached entries share a metric name but their hashes differ, that is a silent definition conflict, and it can be surfaced automatically instead of showing up as a mismatch in a board deck. It comes along for free once derivation is already being recorded for the access-control reason. Both values are sub-microsecond to compute. The retrieval-time check is then a single set-difference: is S(a) a subset of what this requester is permitted to see? For Marketing, whose permission set explicitly excludes income, the answer is no, regardless of how inert the value 0.114 looks. The Fix: Gate Retrieval on Derivation, Not on Tags The mechanism described in the paper is called the Analytical Memory Unit (AMU) — a memory schema that carries the derivation graph as a first-class field, plus a retrieval algorithm that gates on it. Figure 2 shows the full architecture. Two paths run through this diagram, and they solve two different problems. The read path (left and center) is the leak-prevention mechanism. A requesting agent asks for a metric by name. The lineage gate pulls every AMU ever stored under that name, resolves the requester's permission set once, and scans candidates most-recent-first — a deliberate policy choice, so that when a safe candidate exists, the department gets the freshest one rather than an arbitrary historical value. For each candidate, the gate checks a single predicate: is the cached S(a) a subset of P(department)? The first candidate to pass gets served, and a memory-reuse counter increments. If a candidate fails — Marketing hitting the Finance-computed churn_rate — it is skipped, and the loop moves to the next-oldest candidate. If nothing in the store passes, the agent falls back to a fresh, in-scope computation of its own, which is then written back into the store through the same path, available to the next requester who is actually permitted to see it. The write path (right) is unrelated to access control — it is the definition-conflict detector. Every new AMU is compared against existing entries under the same metric name; if a cross-department pair has matching metric names but different definition hashes, that is flagged for downstream review. It never blocks the write. It simply surfaces a signal that two teams' business logic has silently diverged. The formal safety guarantee behind the read path is a straightforward case analysis: the retrieval algorithm has exactly two exit points, a memory-served return and a fallback return, and the proof shows neither path can hand a requester a result whose derivation touches a column outside their permission set — provided the lineage record is complete. That qualifier does real work, and it is worth sitting with before treating this as a drop-in fix. What It Costs, Measured None of this is useful if it is a paper mechanism that falls apart under load or eats the redundant computation that shared memory exists to avoid in the first place. Six experiments in the paper attempt to break it. Across a synthetic five-table schema and a denser eight-table TPC-H benchmark schema, naive content-gated memory — the pattern every current system uses — leaked cross-department results at 18.8% and 25.5% of retrievals, respectively (30 seeds each, effect size around d≈7–8, well past the conventional threshold for a large effect). Lineage-gated retrieval brought that to zero in both configurations. To be precise about what that number means: this is the formal safety guarantee holding in the tested setting, not an empirical discovery on its own. The empirical finding is the naive baseline's leak rate; the zero is what the proof predicts. That protection is not free. Gating out unsafe candidates means more fallbacks to fresh computation: the reuse rate dropped from 95.8% under naive caching to 81.5%–82.6% under lineage gating, roughly 13 additional query executions per 100 requests. That cost shows up as extra compute, not as gating latency — the permission-check predicate itself runs at 0.09 microseconds, and even the worst-case retrieval scan, with every candidate blocked across a store of 50, tops out at 13.82 microseconds. In a realistic production store, where a metric accumulates at most a handful of department-variant entries, that is sub-2-microsecond in practice, not a meaningful line item next to a query that takes milliseconds to seconds. Two further results matter for anyone deciding whether to trust this in production. First, a completeness-degradation experiment: the safety proof assumes lineage recording is complete, but real instrumentation is not always. Testing with deliberately truncated lineage — simulating buggy instrumentation or agents that under-report what they touched — showed the protection degrading gracefully rather than falling off a cliff, converging back to zero leakage by 75%–90% completeness, and degrading faster specifically on schemas where sensitive columns sit behind more joins. That is a useful operational signal: instrument the deeply nested joins first. Second, and more important for anyone concerned this only works when the agent cooperates: a real-agent test wired lineage extraction directly into the database driver layer, using a Structured Query Language (SQL) parser to read the actual statements a LangChain agent executed, with no self-reporting from the agent required. Across nine round-trips against a real database, there were zero cross-department leaks, and two genuine definition conflicts were caught automatically. The cleanest integration point turns out to be exactly where you would want it: wrap the cursor, parse every executed statement, and the agent never has to know it is being watched. What This Doesn't Fix The credibility of a mechanism like this depends on being precise about its edges, so three limitations are worth stating plainly rather than leaving implicit. It does not stop inference attacks: a derived feature that encodes sensitive information without ever naming the source column in a way the lineage extractor can see. A hand-crafted risk score computed from income, where that dependency is not captured in the executed query, will slip past the gate, because the gate can only check what the lineage record actually contains. This is the gap the paper's authors flag as the most important one left open. It assumes lineage recording is honest and complete. Self-reported lineage from a cooperative agent is a weaker foundation than lineage extracted automatically from executed SQL, which is exactly why driver-layer interception matters more than it might first appear: it removes the agent's ability to misreport what it touched, whether deliberately or through a bug. It does not defend against a compromised memory store or an insider with direct infrastructure access, and it does not, on its own, stop a requester from triangulating a blocked column by combining several individually permitted results that are jointly informative about it. Those need their own controls: encryption, audit logging, differential privacy, and infrastructure trust boundaries. This is one layer in a defense-in-depth stack, not a replacement for the source-layer controls it sits alongside. The Question Worth Asking About Your Agent-Memory Stack If you are building or evaluating a shared memory layer for agents, there is one question worth running through concretely: when your retrieval path decides whether to hand a cached result to a requesting agent, what predicate is it actually evaluating? A tag someone attached when the entry was written? A role the requester happens to hold? Or a set-membership check against the actual columns that produced the value? Most systems on the market today, including every system benchmarked in the referenced paper, are checking the first two. RBAC and content tagging are not wrong; they answer a real question, just not the one that determines whether a specific cached number is safe for a specific requester. That is a derivation question, and it needs a derivation record to answer it. The mechanism described here costs microseconds and a few hundred bytes per cache entry. What it buys is a governance layer that can prove, mechanically, at retrieval time, that a result never crossed a column-permission boundary — rather than discovering the leak in a post-incident audit. This article is a plain-language, engineering-focused walkthrough of "Lineage-Aware Memory Governance: A Derivation-Gated Framework for Privacy-Preserving Column-Level Access Control in Enterprise AI Agents," co-authored with Sudhir Vissa (SAGE7 AI), accepted at IEEE Access (DOI: 10.1109/ACCESS.2026.3730363). Figures are reproduced from the published paper. The full derivations, threat model, and experimental protocol are available in the paper and its supplementary material.