Deployment

AI-generated pull requests are moving from experimental workflow to routine software delivery. Coding agents can now plan changes, edit multiple files, run tests, and submit pull requests with limited human intervention. A 2025 study of 567 Claude Code pull requests across 157 open-source projects found that 83.8% were eventually merged, and about 55% of merged changes required no further modification. A broader 2026 study analyzed more than 302,000 verified AI-authored commits from 6,299 repositories. The practical question is therefore no longer whether generated code can enter production. The harder question is what happens after the merge, when readability problems, architectural drift, latent defects, and future maintenance costs become part of the codebase. Faster Generation Changes the Review Equation AI assistance does not automatically imply lower quality code. A GitHub randomized controlled study involving 202 experienced developers found that Copilot-assisted implementations performed better on the study's functional tests and received modestly higher ratings for readability, reliability, maintainability, and conciseness. That evidence matters because technical debt should not be treated as an unavoidable property of machine-generated code. Production repositories, however, create conditions that controlled coding tasks cannot fully reproduce. The 2026 large-scale study of AI-authored commits identified 484,366 introduced issues where 89.3% were code smells, and more than 15% of commits attributed to every assistant studied introduced at least one detectable issue. Most importantly, 22.7% of tracked AI-introduced issues were still present at the repository's latest analyzed revision. The same research also found that AI-authored commits can remove existing smells, showing that generated code can both create and repay debt. That distinction changes the engineering problem. The central risk is not generation itself but generation speed combined with weak validation. DORA's 2025 research describes AI as an amplifier that magnifies existing organizational strengths and weaknesses. Earlier DORA analysis also associated increased AI adoption with larger change batches, slower review, and weaker delivery stability. Faster code production therefore raises the value of disciplined review rather than reducing it. Debt Begins at the Merge Boundary Technical debt becomes an organizational liability when a change is accepted into a maintained branch. The model that produced the code cannot own an incident, negotiate a migration, preserve a service level objective, or explain an architectural compromise six months later. Accountability must therefore follow the repository and the component, not the mechanism that generated the diff. This makes the pull request author field an increasingly weak representation of ownership. A generated change may carry an agent identity, a bot account, or a human account that delegated most implementation work. None of those identities answers the operational question of which engineering group is responsible for the behavior after deployment. Repository ownership rules provide a stronger boundary. GitHub CODEOWNERS can automatically request reviewers for designated paths, and protected branch settings can require approval from a code owner before merge. Required status checks can additionally prevent merging until defined CI checks pass. A policy can remain simple: Plain Text /src/main/java/com/acme/payments/ @payments-platform /src/main/java/com/acme/identity/ @identity-platform The important property is not the syntax. The important property is that an AI-generated payment change still reaches the team accountable for payment correctness, while an identity change still reaches the team accountable for authentication and authorization. Provenance may influence review depth, but provenance should never replace domain ownership. Human Review Becomes More Valuable, Not Less Higher generation throughput can create pressure to automate review with another model. AI review is useful for summarization, repetitive checks, suspicious patterns, and localized suggestions, but available evidence does not support treating it as equivalent to accountable human review in every context. A 2026 study covering 278,790 review conversations across 300 mature open source projects found that human reviewers exchanged 11.8% more review rounds when handling AI-generated code than human-written code. Human reviewer suggestions were adopted at a much higher rate than AI-agent suggestions, 56.5% versus 16.6%, and more than half of unadopted AI suggestions were either incorrect or addressed through alternative fixes. Another study analyzing more than 22,000 comments from AI-based GitHub review actions found substantial variation in effectiveness and reported that concise, contextual, hunk-level feedback was more likely to produce code changes. Current GitHub documentation explicitly states that Copilot code review does not replace human code review. GitHub also supports configurations in which Copilot approvals can count toward merge requirements, making repository policy an important governance decision rather than a tooling default. For critical components, AI review is better positioned as an additional signal before accountable approval, not as the entity carrying final ownership. Quality Gates Must Follow Risk A generated pull request should not receive weaker controls because tests pass. Passing tests establish only the behavior covered by those tests. Technical debt often appears elsewhere as duplicated logic, boundary violations, unnecessary abstractions, missing failure handling, insecure defaults, excessive coupling, or code that satisfies the immediate task while conflicting with local design conventions. CI can convert some of those conventions from reviewer memory into executable policy. An AI-assisted label, for example, can trigger additional checks without requiring a separate delivery pipeline: YAML ai-change-gate: if: contains(github.event.pull_request.labels.*.name, 'ai-assisted') steps: - run: ./gradlew test integrationTest - run: ./gradlew check - run: ./scripts/verify-architecture.sh The architecture check can encode repository-specific constraints such as forbidden dependencies, package boundaries, or service-layer access rules. The AI label is only a routing signal where the underlying controls should be based on change risk. A generated documentation correction does not require the same scrutiny as an authorization change, while a human-written authorization change should receive the same high-risk checks. GitHub rulesets support required checks and required reviews as merge conditions, allowing these policies to be enforced rather than documented as optional guidance. The Useful Metric Is Rework Counting generated lines, completed pull requests, or acceptance rate measures production volume, not maintenance quality. A more meaningful evaluation connects AI-assisted changes to downstream engineering cost. Review rounds, escaped defects, rollbacks, post-merge rework, issue survival, architecture violations, and time spent modifying recently merged code reveal whether higher implementation throughput is creating future work. This framing also avoids a false binary between "AI code" and "human code." The large-scale debt study found both issue introduction and issue removal in AI-authored commits. The relevant objective is therefore not minimizing AI involvement, but minimizing unmanaged debt while preserving useful productivity gains. Ownership should remain attached to the team that accepts and operates the change, while automated gates provide evidence that acceptance criteria were actually satisfied. Conclusion AI-generated pull requests change who produces code, but they do not change who must maintain software after deployment. Technical debt cannot be delegated to a model, hidden behind a bot identity, or treated as the future responsibility of an unspecified reviewer. The durable ownership boundary is the engineering team responsible for the affected component, supported by required human approval, executable quality gates, repository ownership rules, and measurements that expose downstream rework. AI can accelerate implementation and can even remove existing debt, but merge authority must remain tied to accountable engineering ownership. When generation becomes nearly instantaneous, disciplined acceptance becomes the scarce resource, and that acceptance is where responsibility for technical debt belongs.

By Uthej Mopathi CORE

When JDK 27 was released on 15 September 2026, most of the attention went to compact object headers and post-quantum TLS. JEP 523 received less coverage, partly because its description sounds like housekeeping: G1 becomes the default garbage collector in all environments. For most applications, that changes nothing. For a specific class of container, it changes the collector silently, with no warning in the logs and no failure at startup. This article covers what the fallback did, which workloads lose it, how to identify affected services, and how to run a comparison that produces usable numbers rather than noise. What the Serial GC Fallback Did G1 has been the nominal default since Java 9. What is less widely known is that HotSpot inspected the machine at startup and overrode that choice in constrained environments. The JVM selected Serial GC when either condition was true: ConditionThresholdAvailable processors1Physical memoryLess than 1,792 MB The reasoning was sound at the time. Serial GC is single-threaded with a very small native memory footprint. On a single-core machine, a concurrent collector spends time competing with application threads for the only available core, and the bookkeeping structures G1 maintains were expensive relative to a small heap. G1 has improved substantially since then. Its native memory footprint has been reduced across several releases, and JDK 26 delivered JEP 522, which removed synchronization overhead from G1's write barriers by introducing a second card table. JEP 523 concludes that work. The fallback is removed, and G1 is now selected in every environment regardless of CPU count or available memory. Serial GC remains fully supported and can still be selected explicitly: Shell java -XX:+UseSerialGC -jar application.jar Applications that already specify a collector, whether G1, ZGC, Parallel, or Shenandoah, are unaffected by this change. Which Workloads Are Affected The change only reaches services that never had an explicit collector flag and that run in a constrained environment. In practice, this describes a predictable set of deployments: Sidecar containers. Log shippers, metrics agents, and service mesh components typically receive a fraction of a CPU and a few hundred megabytes of memory. They are rarely tuned.Batch and queue workers. Short-lived jobs scheduled onto whatever capacity is available.Low-traffic internal services. Admin APIs, configuration services, and internal tooling that are deliberately given small pods.CI runners and development containers. Environments where JVM flags are seldom set because the workload is not production.Pods with fractional CPU limits. This case is easy to miss. A container configured with cpu: 500m may report a single available processor to the JVM, which placed it in Serial GC territory on earlier releases. These are the workloads least likely to appear in a performance test plan, which is precisely why the change deserves deliberate verification. Step 1: Identify Your Current Collector Before changing anything, determine which of your services are actually running Serial GC today. The GarbageCollectorMXBean interface exposes this directly: Java import java.lang.management.GarbageCollectorMXBean; import java.lang.management.ManagementFactory; public final class RuntimeFacts { public static void main(String[] args) { Runtime rt = Runtime.getRuntime(); System.out.printf("java version : %s%n", Runtime.version()); System.out.printf("available cpus : %d%n", rt.availableProcessors()); System.out.printf("max heap MB : %,d%n", rt.maxMemory() / (1024 * 1024)); for (GarbageCollectorMXBean gc : ManagementFactory.getGarbageCollectorMXBeans()) { System.out.printf("collector : %s%n", gc.getName()); } } } The collector names identify which implementation is active: CollectorReported MXBean namesSerial GCCopy, MarkSweepCompactG1G1 Young Generation, G1 Concurrent GCParallelPS Scavenge, PS MarkSweepZGCZGC Cycles, ZGC Pauses For a Spring Boot application, registering this as a startup listener puts the information into your existing log aggregation without any additional infrastructure: Java import org.springframework.boot.context.event.ApplicationReadyEvent; import org.springframework.context.event.EventListener; import org.springframework.stereotype.Component; import java.lang.management.GarbageCollectorMXBean; import java.lang.management.ManagementFactory; import java.util.stream.Collectors; @Component public class GcFactsLogger { @EventListener(ApplicationReadyEvent.class) public void logCollector() { Runtime rt = Runtime.getRuntime(); String collectors = ManagementFactory.getGarbageCollectorMXBeans() .stream() .map(GarbageCollectorMXBean::getName) .collect(Collectors.joining(", ")); System.out.printf( "JVM %s | cpus=%d | heap=%,dMB | gc=%s%n", Runtime.version(), rt.availableProcessors(), rt.maxMemory() / (1024 * 1024), collectors); } } Deploy this across your estate, then filter your logs for entries reporting Copy as a collector name. That result set defines the scope of your testing. Step 2: Understand the Native Memory Difference Discussions about garbage collector selection usually center on pause times. For this particular change, native memory consumption is the more consequential metric. G1 maintains bookkeeping structures that Serial GC does not require, including remembered sets, card tables, and per-region metadata. This memory is allocated outside the Java heap, which means it counts against a container memory limit without appearing in heap metrics. Measurements published after the JDK 27 release place G1's native memory overhead at approximately 43 MB, compared with roughly 1.7 MB for Serial GC. Ankur Mundra measured both collectors on Spring Boot 4.1 within real cgroup limits and reported these figures. The absolute difference is modest. The proportional impact depends entirely on container size: Container memory limitApproximate G1 overhead share1024 MB4%512 MB8%256 MB16% On a tightly sized container, this overhead is consumed before the application allocates its first object. Where memory limits were set with minimal headroom, the change alone can produce OOM kills after an upgrade that altered nothing else. Native Memory Tracking provides the per-category breakdown if you want to confirm the figure in your own environment: Shell java -XX:NativeMemoryTracking=summary \ -XX:+UnlockDiagnosticVMOptions \ -jar application.jar Then, against the running process: Shell jcmd <pid> VM.native_memory summary The GC section of the output reports committed native memory attributable to the collector. Step 3: Run a Controlled Comparison A meaningful comparison requires the smallest container size you actually deploy, representative load, and a run long enough to reach steady state. A thirty-second run measures startup behavior rather than collection behavior. Capture GC logs for both configurations: Shell # Pre-Java 27 behaviour java -XX:+UseSerialGC \ -Xlog:gc*:file=serial.log:time,uptime,level,tags \ -jar application.jar # Java 27 default java -Xlog:gc*:file=g1.log:time,uptime,level,tags \ -jar application.jar Three measurements matter, and the third is the one most often omitted: Total pause time. Sum the pause durations across the run rather than comparing individual collections. Serial GC performs fewer, longer pauses; G1 performs more, shorter ones. Only the aggregate is comparable.Tail latency. G1 runs concurrent phases. On a single-core container, those phases contend with request handling for the only available processor. This effect is invisible in a mean and clearly visible at p99.Resident set size under load. This is where the native memory difference becomes operationally relevant. Sample RSS throughout the run rather than reading it once at startup: Shell while true; do ps -o rss= -p "$PID" | awk '{printf "%s %d MB\n", strftime("%H:%M:%S"), $1/1024}' sleep 5 done If RSS approaches your container limit under G1 where it did not under Serial, you have found the problem before it found you. Step 4: Pin the Collector When G1 Loses If measurement shows that G1 performs worse on a constrained workload, the appropriate response is not to defer the JDK upgrade. It is to state the collector explicitly: Shell java -XX:+UseSerialGC -jar application.jar This is a supported configuration. Serial GC is neither deprecated nor scheduled for removal, and JEP 523 explicitly preserves it. In a Kubernetes deployment, express the choice where it will be reviewed: YAML spec: containers: - name: batch-worker env: - name: JAVA_TOOL_OPTIONS value: "-XX:+UseSerialGC" There is an argument for setting the flag explicitly even where G1 performs well. An explicit collector choice is self-documenting. A future engineer reading the manifest can see that a decision was made and tested, rather than inferring what the JVM selected at runtime. Conclusion JEP 523 is a reasonable change. G1 has genuinely closed the gap that justified the fallback, and removing a hidden conditional from the JVM's startup logic simplifies the mental model for everyone. The operational consideration is narrower: a set of small, rarely tested containers will change collectors on upgrade without any signal that it happened. Identifying those containers takes one log statement. Testing them takes one load run. The cost of skipping both is a latency regression or an OOM kill that appears unrelated to any change your team made. Java 27 is not a long-term support release, and many teams will remain on Java 25. That does not remove the relevance of this work. The same default will arrive with the next LTS, and evaluating it on a short-term release carries considerably less risk than discovering it mid-migration. Further Reading JEP 523: Make G1 the Default Garbage Collector in All EnvironmentsJEP 522: G1 GC: Improve Throughput by Reducing SynchronizationJDK 27 Release Notes

When you move messaging onto a third-party platform, you don't hand off the authorization problem. You split it across two systems. Most incidents live in the gap. A simulation harness, and a documentation audit of QuickBlox as a worked example. The Revoked User Who Kept Reading Say a clinic's app removes a contractor's access to a patient thread at 14:02. The app's database says so, the UI hides the thread, and the audit table has the row. The contractor's phone keeps receiving new messages until 15:00, and then keeps going. Nothing in the application is wrong. The membership removal went to the chat platform's API as a single best-effort call, and the API was having a bad ten minutes. The call failed, and the exception was swallowed. The contractor's session token is still valid, and the contractor's client keeps making requests that keep it valid. This is a hypothetical, but every piece is routine: a removal path with no retry, an admin API that can be unavailable, a session whose lifetime depends on activity. The useful question is how long the window lasts, and it becomes measurable once you stop treating the integration as an API call and start treating it as a consistency boundary. This article builds a small deterministic harness, seamlab.py, to measure that window along with seven other failures at the same boundary, including an AI agent acting after a human has taken over a conversation. It then audits one real platform, QuickBlox, against the same questions using its public documentation. QuickBlox is the worked example because its developer documentation is public and detailed enough to check. The harness measures a class of platforms and nothing about any vendor. The documentation audit reads documentation and tests nothing. I keep those two kinds of evidence apart throughout. The Flawed Assumption The usual framing of "build vs. buy" for chat or calling is about transport: delivery, ordering, presence, media. Delegating those is reasonable. But "who may read this thread, who may post, who may trigger an action" is not a transport concern, and delegating transport forks it: Your application holds the relationships: this clinician treats that patient, this contractor was removed.The platform holds a projection of them: dialog memberships, session tokens, message history. The projection is only as fresh as your last successful write to it, and the platform enforces the projection, not your database. So at least two places answer "is this allowed," and they can disagree. Each failure below is a case of that disagreement, or of one side believing a claim the other never checked. Two quantities make this concrete. Revocation window W: the time between the app committing a revocation and the last successful read by the revoked principal. If the platform checks live membership on every read, W is bounded by the time until the removal call lands. If it authorizes from the session token alone, W is bounded by the token's remaining life, and whether that life is fixed or extended by activity decides everything.Authority provenance: for any action in the system, whose authority is it exercising? For a human action, the answer is trivial. For an AI agent in a multi-party conversation, it usually isn't. Threat Model Six attacker positions, all realistic for a chat integration: #AttackerCapabilityAExternalKnows the webhook URLBAuthenticated low-privilege participantCan post messages into a thread the agent readsCSelf-registered platform userPlatform allows signup, no verified binding to your user modelDRevoked participantHolds a valid token from before revocationEReplayerHas a captured, validly signed event (logs, proxy, compromised sink)FWriter on the audit storeCan modify rows Trust boundaries: client to app, app to platform (admin API), platform to app (event delivery), agent to tools, and the audit store. ClassPreconditionMechanismTestControlForged eventAUnauthenticated event endpoint acceptedforgedSignature verificationIdentity confusionCSender resolved by mutable attribute (email)spoofed identityImmutable-ID binding tableReplayEDedupe TTL shorter than replay horizonreplayTimestamp window plus retention ≥ 2×windowDuplicate side effectNormal at-least-once deliveryNon-atomic check-then-actraceAtomic claim before actingStale accessDToken and membership outlive revocationrevocation sweepShort TTL, outbox retry, token killAgent confused deputyBInjected text drives a privileged toolA1–A4Parameter-bound approvalPost-handoff actionRace with human takeoverAgent call in flight at handoffA5–A6Epoch fencingScope overreachBAgent reads history beyond sessionA7Capability-scoped history Audit tampering (F) is covered separately below. The Harness seamlab.py is about 480 lines of stdlib Python on a logical clock, with no network and no randomness. It contains: A platform simulator: at-least-once event envelopes signed with HMAC, expiring tokens with scopes frozen at mint, optional idle (sliding) expiry, self-service registration, and an admin API that can be switched off. Whether reads check live membership, and whether a server-side token kill exists, are switches.Two app implementations: a naive one and a hardened one.An agent world with a tool proxy configurable per control, so the table shows which control closes which attack.A worst-case agent: it obeys every CALL tool key=value line in its context. That is deliberate. The tests target the authorization layer and say nothing about how robust any model is to injection. If your model resists injection, that's a bonus. These controls assume it doesn't. The duplicate-delivery test enumerates every distinct interleaving of two workers handling the same event, rather than sampling a few. Naive Design, Then the Attacks The naive ingress handler is a shape I'd expect to find in plenty of codebases: no authentication on the endpoint, dedupe with a TTL, check then act then record, sender resolved by email. def handle(self, env): t = self.seen.get(env["id"]); yield # check if t is not None and self.clock.t - t < self.DEDUPE_TTL: # 300s return "dup" uid = self.resolve(env["body"]["sender"]) # email match if uid: self.run_command(uid, env["body"]["text"]) # act yield self.seen[env["id"]] = self.clock.t; yield # record AttackNaiveHardenedForged unsigned eventsucceedsblockedValidly signed event, spoofed identitysucceedsblockedReplay at t+225 sblockedblockedReplay at t+3600 ssucceedsblockedDuplicate delivery: interleavings producing a double effect60 of 660 of 12 The spoofed-identity row is the interesting one. The event carries a correct platform signature. Nothing was forged. The attacker registered a platform account with email [email protected], unverified. The naive app lowercases and matches against its user table, so a perfectly authentic message from the wrong person runs with the clinician's authority. Signature verification can't help, because the signature is telling the truth about who sent it. The control is a binding table keyed on the platform's immutable user ID, written only by your backend when it provisions the platform user for an authenticated app user. The attacker's ID has no binding, so the event is rejected as unbound. The race result is worse than it looks. 60 of 66 distinct schedules double-fire. Only interleavings where the first worker finishes recording before the second starts checking are safe. This is the ordinary case for any retry storm. The fix is one atomic operation (SET NX EX, or an insert against a unique constraint) before any side effect. The hardened handler: def handle(self, env): if not hmac.compare_digest(env.get("sig", ""), sig_of(env)): return "bad_sig" if abs(self.clock.t - env["ts"]) > self.TOL: return "stale" ok = self.claim(env["id"]); yield # atomic set-if-absent, retention 2*TOL if not ok: return "dup" uid = self.bindings.get(env["body"]["sender"]) if uid is None: return "unbound" self.run_command(uid, env["body"]["text"]); yield return "ok" Mutation-Testing the Control The suite should be able to fail the hardened design; otherwise, the green column is decoration. So I mutated one parameter: dedupe retention set to half the timestamp tolerance. The replay at t+225 s then succeeds, because the event is still inside the accepted time window while its dedupe record has expired. The replay at t+3600 s is still blocked by the timestamp check. That yields an invariant worth writing into code review: dedupe retention must be at least twice the accepted timestamp skew (once for events slightly in the future, once for slightly in the past). Timestamp validation and dedupe are two halves of one control, and shortening either silently opens the gap. One caveat: claim-before-act gives at-most-once side effects if the process dies between claim and effect. For effects that must not be lost, the downstream operation needs its own idempotency key derived from the event ID. The harness does not model crash recovery. Revocation Window, Measured The harness drives a revoked user who keeps polling and refreshing, with the token's age at revocation sampled across its lifetime. The scenario that matters: the admin API is down when the revocation is issued and comes back 600 s later. In the first table, the platform checks live membership on every read: ConfigWorst W (s)Mean W (s)Token mints per user per hourNaive, TTL 3600, single attempt360018301Hardened, TTL 303016120Hardened, TTL 60603060Hardened, TTL 30030015212Hardened, TTL 9006014054Hardened, TTL 36006015561 "Hardened" means a transactional outbox retrying the removal with capped backoff, a broker that re-checks the app's authorization on every token mint, and a short TTL. Once TTL exceeds the outage, W is capped by the outage plus up to one backoff interval (601 s in this run). Below the outage length, TTL is the binding constraint. With a healthy admin API, every configuration, naive included, shows W = 0. The naive design only fails when the platform is degraded, which is when nobody is watching. When the Platform Doesn’t Check Membership, and Tokens Slide Two platform properties change the picture, and I only added the second after reading the vendor documentation described below, which is why the harness changed between drafts. If reads are authorized from the token alone, removal calls don't matter, and the TTL is the only built-in lever. If the token's expiry is idle-based, so each request pushes it out, then a revoked client that keeps making requests never lets the token expire, and the TTL stops being a bound at all. The broker's re-check at refresh never runs, because refresh never happens. The last control available is a server-side token kill, if the platform offers one. Results for a token-only platform (hardened TTL 60, naive TTL 3600; "never ends" means access persisted to the end of the simulated horizon): Expiry model, admin APINaiveHardened, no token killHardened + token killFixed TTL, healthy3600 s60 s0 sFixed TTL, 600 s outage3600 s60 s60 sIdle expiry, healthynever endsnever ends0 sIdle expiry, 600 s outagenever endsnever ends601 s Read the third row twice. With idle expiry and no kill, the short TTL and the broker re-check buy nothing against a client that keeps talking. The only controls that end access are live-membership enforcement on the platform or an invalidation you can trigger from the server. So the most valuable hour you can spend before integrating a platform is finding out which of those you have. At TTL 60, each active user generates 60 token mints per hour. Whether that is trivial depends on your user count and the platform's rate limits, which the harness does not model. The AI Participant: Confused Deputy, Not “Authorization for Bots” A multi-party conversation is a prompt authored by several principals with different privileges. The agent reading it holds one set of tool credentials. Any principal who can put text in the thread can attempt to spend authority they don't have. This is the classic confused deputy, with the deputy now parsing natural language. The harness has a dialog with a patient and a clinician, and three tools: get_my_record (low privilege), approve_refill (clinician only), and read_history. The proxy runs in three configurations: naive (the agent's standing credential decides everything), role-check (the call executes with the authority of whoever authored the triggering message), and hardened (role check plus parameter-bound approval, epoch fencing, and history clamping). AttackNaiveRole-check onlyHardenedA1 Patient's message steers agent into approve_refillsucceedsblockedblockedA2 Clinician's benign turn; patient's injected text sits in contextsucceedssucceedsblockedA3 Approval for rx42 reused to approve rx99succeedssucceedsblockedA4 One approval consumed twicesucceedssucceedsblockedA5 In-flight agent call executes after human takeoversucceedssucceedsblockedA6 New agent turn minted after takeoversucceedssucceedsblockedA7 Agent reads history outside session scopesucceedssucceedsblockedB1 Clinician approves refill through explicit approvalworksworksworksB2 Patient reads own record through agentworksworksworks The B rows make sure the hardened column isn't green merely because it denies everything. A2 is the finding that changed my plan. My pre-run design was authority as the intersection of the acting principals' rights. Role-check is that rule, and it stops the obvious attack (A1). It fails A2 completely. The clinician sends "please summarize the thread." The agent's capability is legitimately the clinician's. The patient's earlier message, still in context, asks for a refill approval, and the agent obliges, spending authority that was never granted for that purpose. Intersection scopes who the agent acts for. It says nothing about what that authority was meant to cover. The fix is to stop treating conversation turns as grants for privileged operations. A privileged call needs an approval bound to the exact call, issued out-of-band by an authenticated UI session of a privileged human: def _approval_ok(self, tool, args, ap): b = ap["body"] good = hmac.new(APP_KEY, json.dumps(b, sort_keys=True).encode(), hashlib.sha256).hexdigest() if not hmac.compare_digest(good, ap["mac"]): return False if (b["tool"], b["args"], b["epoch"]) != (tool, args, self.epoch): return False if self.clock.t > b["exp"] or b["nonce"] in self.used: return False if self.users.get(b["principal"]) != "clinician": return False self.used.add(b["nonce"]); return True The approval binds tool, arguments, epoch, expiry, and a single-use nonce. A3 fails because the arguments differ, and A4 because the nonce is spent. Injected text can still request the call. It cannot carry the authority to make it. The system doesn't try to detect injection. It makes sure injected text can't hold a capability. Fenced handoff (A5, A6). When a human takes over, the app increments a per-dialog epoch. Every capability carries the epoch it was minted under, and the tool proxy rejects stale ones. A5 is the race: a turn starts at t=0, the human takes over at t=5, and the agent's call arrives at t=8. Without the fence it executes. This is a fencing token, and it has a fencing token's limit: it only works if the resource performing the effect checks the epoch. If the agent can also post through a path that bypasses your proxy, such as the platform's own API with the agent's own credentials, the fence is decorative. The platform-side complement is removing the agent from the dialog at handoff, which is another admin call, subject to the same outage window measured above, and which not every dialog type allows (see below). History scope (A7). The capability carries the index where the current session begins, and read_history refuses anything earlier. The harness checks the proxy's enforcement. It does not check that the agent's context assembly respects the boundary, which is separate code you would need to test. The audit log, and what a hash chain does not do The hardened path records each agent tool call (allowed or denied) and each handoff in a hash-chained log with actor type, model version, tool, arguments, result, and principal: Clean chain, verified against an externally stored head hash: passes.Row modified without recomputing the chain: detected.Row modified and the chain recomputed from that row forward (an attacker with write access): verifies clean unless you compare against the externally stored head, in which case it is detected. The chain alone defends against accidental corruption and clumsy tampering. It does not defend against someone who can write the store and knows the scheme. What does is anchoring the head hash somewhere that an attacker can't write: a separate account, an append-only object store, a periodic signed checkpoint. If your threat model includes a malicious insider with database access, the anchor is the control, and the chain is the mechanism that makes anchoring cheap. On compliance: in a regulated setting, "who did what, when, under whose authority" is what this log makes answerable. The HIPAA Security Rule's audit-control standard is 45 CFR §164.312(b) (cited from memory; confirm before relying on it). A platform's compliance posture does not discharge your obligations, and a log design is one input to them, not compliance itself. A Documentation Audit: The Seam on Quickblox Everything above is simulation. This section asks the checklist questions of a real platform using its public developer documentation, retrieved on 4 October 2026. Nothing here was tested against a live QuickBlox account. Documentation can be incomplete or out of date, and where I could not find an answer, I say so instead of guessing. Sources are listed at the end. QuestionWhat the documentation saysConsequence for the seam1. Are server-to-app events authenticated?I found no first-party documentation of outbound server-to-server event webhooks. A competitor's comparison page asserts QuickBlox doesn't support webhooks. That is a competitor's claim and I haven't confirmed it.If there is no signed event feed, the forged/replay/race tests apply to whatever you build instead. The rule that survives is to never let your backend act on events reported by clients, because client-reported events are attacker-controlled. If you poll, poll with a server credential and treat the response as the source.2. Does a read check live membership, or only the token?Not stated in the pages I read.Unknown. This decides which row of the revocation tables applies. It is a two-minute sandbox test: remove an occupant, then read with that user's still-valid token.3. Token lifetimeBoth user and application session tokens expire 2 hours after the last REST request, and each REST response carries an expiry header. I found no configuration for it.This is idle expiry, which is the dangerous row of the table above. A client that keeps making REST calls keeps the session alive. I don't know whether live chat-connection activity also extends it, and I haven't verified an absolute cap.4. How is membership removed?Removal goes through the dialog update call. A dialog's owner can remove any occupant (the REST reference says admins can too), and an occupant can remove only themselves. The SDK and dashboard docs show removing other users working for group dialogs, not for private (one-to-one) or public ones. Deleting a user removes them from group dialogs but not from private dialogs' occupant lists.Private dialogs have no documented remove-another-user operation. If revocation matters, a clinician-patient conversation modeled as a private dialog can't be revoked by removal. You'd have to delete the dialog for everyone (which needs the owner and loses history). Model sensitive conversations as group dialogs with an app-controlled owner. This is a design decision to make before launch, not after.5. Immutable IDs, self-registrationUser creation with an application session is documented as acceptable for prototypes. For production the docs recommend creating users from your backend with the API key and disabling application-session user creation. Users carry an external_user_id field intended to link to an outside system, and login, email and external ID must be unique.The identity-confusion attack depends on that production setting. Bind your app user to the platform's own user ID, written only by your backend, and don't resolve senders by login or email. external_user_id is a convenient mapping field, but the binding should still be enforced on your side.6. Can the agent be a distinct principal?Dialogs have owner, admin, and occupant roles.Make the agent a separate user and an occupant, never an admin or owner. To fence it at handoff by removing it, the dialog must be a group dialog, per row 4. I did not verify whether the agent's session can be invalidated server-side.7. Can you reconcile after a missed event?Dialog and message REST endpoints exist, and the update and delete references show them taking an API key. I did not verify listing and filter semantics for drift detection.Plan the reconciler as a poll of dialogs and occupants compared with your database. Confirm pagination and filter support in a sandbox first. Three observations from the table, none of them about vendor quality. Every platform has choices like these, and every one is a place the seam can fail: The documented 2-hour idle expiry changes the revocation answer. In the harness's token-only, idle-expiry rows, short TTLs and broker re-checks buy nothing, and the only thing that ends access is membership enforcement or a server-side kill. Whether QuickBlox enforces membership at read time is the first thing to test, and whether you can invalidate a session from the server is the second.The private-dialog limitation is the most actionable finding. It is a design constraint you can only avoid by choosing group dialogs up front.The absence of a documented signed event feed moves work to you. The ingress hardening in this article becomes a design for whatever sync mechanism replaces it. On the vendor's compliance statements: QuickBlox's own pages state that it offers HIPAA-ready hosting with a signed BAA, and that it is SOC 2 certified. I haven't verified those claims, and a BAA does not make your application compliant. The seam problems in this article are exactly the part a BAA doesn't cover. Cost and trade-offs Ingress overhead. Hardened ingress costs about 8.8 µs per event against 2.0 µs for the naive handler, in-process, on one machine (Python 3.12). Indicative only. In production, the dominant cost is one atomic store write per event.Token churn. Short TTLs convert revocation latency into mint load: 60 mints per user-hour at TTL 60, 120 at TTL 30. And if expiry is idle-based, TTL tuning may not be available to you at all.Dependence on the admin API. Forced removal and token kill are only as available as that API. The outbox hides outages from the revoking caller. It doesn't shorten them.Approval friction. Every privileged action needs a human click bound to specific arguments. The agent can't autonomously perform privileged operations. That is the intended outcome, and it will frustrate some product teams. Mitigate it with a good approval surface: show the exact arguments, make approval one tap, and expire quickly.Complexity. The hardened design adds a broker, an outbox, an approval service, a per-dialog epoch, and an anchored log. For an internal tool with a few trusted users, it is probably more than you need. Limitations of this work The simulator is simple: one dialog, instantaneous admin success when up, no clock skew, no partial failures inside the platform, a single-node claim store.The worst-case agent models the threat, not any LLM.I did not implement or test a membership reconciler, although I recommend one.The QuickBlox audit is documentation-only. Docs may be incomplete, inconsistent (the owner-versus-admin removal rule is stated differently in different pages), or out of date. Every "unknown" in the table is a sandbox test I haven't run.Remaining exposure none of this addresses: injected text can still influence unprivileged tools (what the agent says, what it summarizes), exfiltration through in-scope reads, and approval fatigue.Nothing here shows that any real platform is vulnerable to any attack in the harness. Conclusion The integration boundary is a consistency boundary, and the interesting failures are state disagreements and authority crossing it unchecked. The revocation window gives you a number to argue about, instead of a feeling, and the number depends on three platform properties you can look up before writing code: whether reads check live membership, whether token expiry is fixed or idle-based, and whether you can remove a member or kill a session from the server. For the AI participant, the real question is whose authority a tool call exercises. Intersecting principals' rights is necessary but not sufficient. A privileged call needs an approval bound to its exact arguments, or conversation text will eventually carry authority it shouldn't. Sources (QuickBlox documentation, retrieved 4 October 2026): Key Concepts: https://docs.quickblox.com/docs/key-conceptsAuthentication (JS): https://docs.quickblox.com/sdks/js-authenticationUpdate Dialog: https://docs.quickblox.com/reference/update-dialogiOS Dialogs: https://docs.quickblox.com/sdks/ios-chat-dialogsDashboard, Chat: https://docs.quickblox.com/docs/chat-dashboardDashboard, Users: https://docs.quickblox.com/docs/users-dashboardCreate User: https://docs.quickblox.com/reference/create-userDelete Dialog: https://docs.quickblox.com/reference/delete-dialog

By Igboanugo David Ugochukwu CORE

A security review started after connection strings were visible through a Kubernetes management tool. The first request sounded simple: remove Secrets from the cluster and move the values to Azure Key Vault. That request focused on which system stored the values. The access review found a wider problem. Kubernetes stored a deployment service account's legacy token in a kubernetes.io/service-account-token Secret. The token had no expiry in its normal use, and RoleBindings allowed the same identity to operate in three application namespaces. The Role used resources: ["*"] across several API groups and allowed get, list, watch, create, update, patch and delete. The Role included the core API group, so the token could read every Secret in each of those namespaces. The review found no evidence of token misuse. It found an avoidable access path with a long lifetime and a larger scope than the pipeline needed. Moving application passwords to Key Vault would improve their source of truth and rotation. It would not revoke this token, narrow its RBAC permissions or remove the cross-namespace bindings. Those controls needed their own fix. Why the Kubernetes Secrets Existed The application images carried JSON configuration files for local development. Database, cache and message-broker endpoints pointed to localhost. Those defaults were useful on a developer machine and unusable inside the cluster. The shared Helm chart supplied the deployment values at runtime. It rendered an Opaque Secret from a secrets map, then loaded the Secret into the application container through envFrom. The reduced chart looked like this: YAML {{- if .Values.secrets } apiVersion: v1 kind: Secret metadata: name: {{ include "app.fullname" . }-secrets type: Opaque stringData: {{- range $key, $value := .Values.secrets } {{ $key }: {{ $value | quote } {{- end } {{- end } The Deployment then loaded every entry as an environment variable: YAML envFrom: - secretRef: name: {{ include "app.fullname" . }-secrets The application configuration placed environment variables after the file-based settings, so the runtime values won. That ordering made the Kubernetes Secret part of the configuration path. Removing it without replacing that path would leave the containers pointing at local services. Figure 1. The existing deployment used a Kubernetes Secret as the runtime override for local file defaults. The next question was, "Which components can retrieve the value, and how long does each form of access last?" Key Vault Does Not Always Remove the Kubernetes Secret The Azure Key Vault provider for the Secrets Store CSI Driver supports more than one consumption model. A pod can read mounted files directly from a CSI volume. This path can avoid creating a native Kubernetes Secret. The application still needs a configuration method that reads those files and responds safely when their contents rotate. The driver can also copy mounted values into a native Kubernetes Secret through the secretObjects field of a SecretProviderClass. Microsoft documents this as secret synchronisation. It is useful when an application already consumes secretKeyRef or envFrom, which was the case here. It also means that moving the source value to Key Vault does not remove the Secret object from Kubernetes. YAML apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: app-vault spec: provider: azure secretObjects: - secretName: app-runtime-secrets type: Opaque data: - objectName: database-password key: Database__Password The choice affects rotation too. Microsoft notes that an existing process does not receive updates when a container consumes a Kubernetes Secret through an environment variable. The team must restart the pod before the process receives the new value. Key Vault remains a sound source for secrets. The important point is narrower: the storage migration and the Kubernetes access review solve different problems. The Service Account Crossed Namespace Boundaries The next part of the review followed the identities that could read Secrets. Kubernetes allows a RoleBinding in one namespace to name a ServiceAccount from another namespace. That feature is useful for controlled cross-namespace work. It also means that checking only the ServiceAccount's home namespace can understate its reach. The review found a layout equivalent to this generalized example: YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pipeline-deployer namespace: environment-b rules: - apiGroups: ["", "apps", "batch", "autoscaling", "networking.k8s.io", "policy"] resources: ["*"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pipeline-deployer-from-a namespace: environment-b subjects: - kind: ServiceAccount name: pipeline-deployer namespace: environment-a roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: pipeline-deployer Two bindings of this form extended the identity from its home namespace into two more namespaces. The same bearer token could therefore authenticate once and use the permissions that each Role granted. Figure 2. Two cross-namespace bindings extended one long-lived credential into three namespaces. The identity was not a cluster administrator. The Role listed its API groups, and it did not include rbac.authorization.k8s.io. The token could not grant itself new RoleBindings through this Role. That limit mattered, but it did not make the existing scope acceptable. The core API group still covered Secrets, and the wildcard resource rule covered much more than the deployment needed. The following authorization check avoids printing any secret value: Shell identity="system:serviceaccount:environment-a:pipeline-deployer" for namespace in environment-a environment-b environment-c do printf '%s: ' "$namespace" kubectl auth can-i list secrets \ --namespace "$namespace" \ --as "$identity" done For the reviewed identity, the effective result was yes in all three namespaces. The Token Type Made the Scope Harder to Contain The service account had one manually created Secret of type kubernetes.io/service-account-token. Kubernetes describes this as a legacy mechanism that provides a long-lived credential. The current documentation recommends the TokenRequest API for short-lived tokens instead. The difference changes the failure window. A deployment job can request a token that expires shortly after the job finishes. If someone copies a static token into a CI/CD service connection, that credential can remain useful until somebody deletes or rotates it. An old export, agent workspace or copied configuration can preserve the same access during that period. The cluster version already supported projected service account tokens. Another environment-specific service account had no legacy token Secret, which confirmed that the static pattern was not present in every namespace. The First Fix Reduces the Token's Reach The quickest containment step is to remove the two cross-namespace RoleBindings. That immediately reduces the token's potential reach from three namespaces to one without changing the applications. Each environment should then have its own deployment identity, Role and RoleBinding. A pipeline for one namespace should not borrow the service account from another. This keeps the credential, audit trail and revocation boundary aligned with the environment it deploys. The desired shape is simple: Plain Text pipeline A uses a short-lived token for its Role in namespace A pipeline B uses a short-lived token for its Role in namespace B pipeline C uses a short-lived token for its Role in namespace C These deployments require no cross-namespace subject. Figure 3. The target design aligns one service account and its short-lived token with one namespace. A Short-Lived Token Replaces the Static Credential Kubernetes can issue a bounded token with the TokenRequest API. kubectl create token is a convenient way to exercise that API: Shell kubectl create token pipeline-deployer \ --namespace environment-a \ --duration 15m The command covers only one part of a CI/CD design. The runner still needs a trusted way to request the token. The pipeline should receive the credential at run time, the requester should limit its audience, and the token should expire soon after the deployment window. The service connection must stop storing the old bearer token before that change has value. After the new path succeeds, delete the legacy token Secret and confirm that the API rejects the old credential. Explicit RBAC Replaces Wildcard Access resources: ["*"] made future chart changes part of the role automatically. Kubernetes warns that wildcard rules can grant access to resource types that future API extensions add. A deployment role should name the resource types and verbs the release actually uses. The rendered manifests provide the initial resource list: Shell helm template app ./chart \ --namespace environment-a \ --values values.environment-a.yaml \ > rendered.yaml Inventory every apiVersion and kind in that output. Then compare the result with deployment calls in API audit logs or a dry run. The resulting Role may need access to Deployments, Services, ConfigMaps, Secrets, Jobs, Ingresses and autoscaling resources. It should not receive every resource merely because the chart might use one later. If a separate controller owns runtime Secrets, the deployment identity can lose Secret write access as well. If Helm still renders those Secrets, the role needs the exact verbs required by that release process. The policy must match the delivery design instead of hiding a required permission until the next deployment fails. The Review Found Other Paths to the Same Values The cluster token was the widest durable access path that the review found. Several pipeline variables also stored credentials as ordinary variables. Azure DevOps treats secret variables as protected resources. It encrypts their values at rest and masks exact values in logs. A task can still send a value elsewhere, and masking does not always hide partial strings. Marking the variables as secret still removes routine plaintext visibility from the library interface and logs. The remediation plan combined four related controls: Mark credential variables as secret and restrict which pipelines can use each variable group.Remove routine human permission to read Kubernetes Secrets through cluster tools.Rotate the database, cache and broker credentials that broader access paths exposed.Move non-sensitive settings out of the Helm secrets map and into a ConfigMap. Moving non-sensitive settings to a ConfigMap makes the review surface honest. Feature flags, log levels and endpoint names do not need to sit beside credentials merely because the original release populated only one values map. Validation Uses the Identity's Point of View The validation must test permissions from the identity's point of view. For each deployment identity, build an explicit matrix: Plain Text own namespace other namespaces read required objects yes no apply release objects yes no read application Secrets only if required no modify RBAC no no Then verify the old paths: the deleted legacy token fails authenticationeach identity returns no for Secret access outside its namespacethe pipeline can still deploy and roll backvariable-group values appear masked in the interface and logscluster users without an operational need cannot reveal Secret valuesrotated application credentials work and the previous values fail This sequence keeps the deployment usable while shrinking the access path in measurable steps. Secret Location and Secret Access Need Separate Reviews The visible Kubernetes Secret started the investigation, but deleting that object first would have left the long-lived deployment credential untouched. Using CSI secret synchronization could even have recreated the same type of object from a better source. The useful review order was identity, scope, lifetime, source and rotation. It exposed one token that crossed three namespace boundaries, wildcard permissions that exceeded the release surface, and pipeline values that lacked basic secrecy controls. Key Vault can improve storage and rotation. Short-lived credentials and narrow RBAC control who can reach the Kubernetes API. Environment-specific identities contain the effect of a leaked token. These controls work together, and none of them can substitute for the others. References Kubernetes documentation: SecretsKubernetes documentation: Service AccountsKubernetes documentation: Managing Service AccountsKubernetes documentation: RBAC good practicesMicrosoft Learn: Azure Key Vault provider configuration options for AKSMicrosoft Learn: Access Key Vault through the CSI Driver on AKSMicrosoft Learn: Secrets in Azure PipelinesMicrosoft Learn: Manage Azure Pipelines variable groups

The enterprise paradigm for artificial intelligence has fractured. While massive cloud-hosted large language models continue to dominate consumer applications, highly regulated sectors like defense, aerospace, and high-tier finance are actively blocking external API calls to public AI services due to strict data sovereignty mandates. The solution is migrating toward small language models (SLMs) localized at the tactical edge. Deploying an SLM into an air-gapped, "sovereign cloud in a box" environment requires absolute cryptographic certainty over the supply chain. A compromised base image running deep inside a disconnected network invalidates the entire security perimeter. The recent release of Docker Hardened Images (DHI) and the expansion of Google Distributed Cloud (GDC) air-gapped appliances give backend engineers the exact toolchain required to host models like Google's Gemma with zero-trust enforcement. This technical guide details an end-to-end framework for curating a zero-CVE container, packing a fine-tuned Gemma SLM, and pushing it to an isolated GDC cluster. The Target Infrastructure: GDC Air-Gapped Google Distributed Cloud air-gapped is an integrated hardware and software platform designed to operate physically disconnected from the internet. It allows enterprises to deploy container-based workloads and managed services like Vertex AI in completely isolated environments. The architecture supports rigorous compliance frameworks, including NIST, ISO 27001, and SOC II. Notably, GDC offers packaged generative AI solutions enabled by Gemma, providing local, conversational search capabilities on-premises. However, when deploying custom fine-tuned checkpoints or building the Java microservices that handle backend inference orchestration- a core focus of my daily work as an IEEE Senior Member and backend developer — we cannot rely on standard public container registries. The container must be hardened from the base layer up. Base Image Curation With Docker DHI To build our localized SLM container, we start with Docker Hardened Images. DHI provides minimal, secure, production-ready container images maintained by Docker. Rather than blindly pulling a generic Alpine or Debian base image and running a vulnerability scanner after the fact, DHI guarantees a hardened foundation. Using the docker dhi CLI, available in Docker Desktop 4.65 and later, we can inspect and mirror these hardened artifacts directly to our enterprise registries. Shell # Verify local DHI CLI installation docker dhi --help # Mirror a hardened Python or Java base image to your local org docker dhi mirror start --org enterprise-registry \ dhi/python,enterprise-registry/dhi-python By switching our Dockerfile FROM instruction to enterprise-registry/dhi-python, we structurally eliminate the attack surface before installing our inference engine dependencies. Generating the SLM Artifact and SBOM As I detailed in my Springer book chapter Introduction to Trustworthy AI, accountability in machine learning relies heavily on technical robustness and traceability. We cannot deploy a black-box container into an air-gapped network. We must generate a cryptographic Software Bill of Materials (SBOM) that catalogs the exact SLM weight hashes, the inference engine binaries, and the OS packages. The DHI CLI handles this transparently. Once you package your Gemma model weights into the hardened container, you can retrieve the exact attestations and SPDX SBOMs attached to the base images. Shell # Retrieve the SLSA provenance and SPDX SBOM for audit docker dhi attestation get dhi/python:3.11 sha256:<digest> -o provenance.json This JSON payload is ingested into the local CI/CD pipeline. The pipeline validates that no unauthorized dependencies were injected during the model packaging phase. If the cryptographic signature fails, the build halts immediately. Deploying to the Sovereign Edge With a cryptographically signed, DHI-backed container housing our Gemma SLM, we transition to the deployment phase on GDC air-gapped. Because GDC air-gapped operates a single cluster encompassing bare metal nodes without internet access, deployment relies on a physical handoff. The hardened container image, the validated SBOM, and the deployment manifests are loaded onto a portable storage device. Once transferred to the secure facility, the infrastructure operator pushes the image to the local, disconnected GDC container registry. GKE on GDC orchestrates the container workload. We apply Kubernetes network policies to isolate the SLM namespace, ensuring that the model can only be queried by authorized internal Java backend services. By enforcing kernel isolation at the container level and physical isolation at the hardware level, we construct a true defense-in-depth architecture. The Gemma model processes highly classified proprietary data while satisfying the strictest data sovereignty requirements. The transition from cloud-first to sovereign-edge AI marks a critical maturation in software engineering. Trustworthy intelligence demands predictable execution boundaries. By merging the zero-trust foundations of Docker Hardened Images with the physical isolation of Google Distributed Cloud, engineering teams can deliver state-of-the-art reasoning capabilities directly to the tactical edge without compromising a single byte of sensitive data.

GitHub's contribution calendar gives you an already iconic overview of your activity over the year. Each day has a square, and its color changes with the contribution level. And of course, those days and contributions can also be arranged into buildings, with a week becoming a small part of a city and busier weeks adding height to the skyline. I had some spare minutes and built Quarkus Contribution City around this idea, using Java and Quarkus to generate an SVG from GitHub's contribution data. The visual inspiration came from Pink Pixel's skyline, and you can see a version with my real data on my GitHub profile. A scheduled GitHub Action updates the image at night. For a Java developer, this is a small project you can follow from the API request through to the image. You can use the released action on your own profile, then change the drawing code locally once you have a feel for how the city is put together. For this article, I created a special DZone example that adds a small piece of character art above the buildings. Add the City to Your Profile Your GitHub profile README lives in a public repository with the same name as your account. For my account, that repository is myfear/myfear. Open your own profile repository, or create it with a README if you haven't used this GitHub feature before. Copy the project's complete publishing workflow into .github/workflows/contribution-city.yml on the default branch. This example checks out your profile repository and the action's v1 release line. It then generates the image and publishes it to an output branch. The workflow includes the repository write permission needed for that last step. The generation step uses the repository owner's username, so the same configuration can be copied between personal profile repositories. You can add the layout options to that step: - name: Generate contribution city id: city uses: ./.city-action with: github-token: ${{ secrets.CONTRIBUTION_TOKEN || secrets.GITHUB_TOKEN } username: ${{ github.repository_owner } weeks: '53' height: '14' theme: github-dark The local path refers to the action checkout made earlier in the example workflow. The built-in GITHUB_TOKEN is the starting point for reading your visible contribution data. An optional CONTRIBUTION_TOKEN secret can supply a personal token when additional access is needed. The calendar returned by GitHub depends on the token's permissions and your account's contribution visibility settings. After committing the workflow, open Actions in your profile repository, select publish-contribution-city, and choose Run workflow. A successful run creates contribution-city.svg on the output branch. Add this image reference to your profile README, replacing both occurrences of USERNAME:  The example also has a daily schedule for 03:17 UTC. Subsequent runs create a commit when the generated image changes. For this setup, the workflow and its run history can be found in your profile repository, while the Java implementation is maintained in the action's repository. Give Each Week a Building The city is arranged in time order, with the most recent week on the right. Every building is two character cells wide, so all weeks occupy the same horizontal space. Weekly contribution totals determine the height, and daily contribution levels provide the window colors. Month labels beneath the street help you find your place in the year. Height needs a little extra thought. A very busy week can have many times the contributions of a quiet one. With a linear scale, those quieter weeks would be compressed close to the street. The city uses a logarithmic scale, which gives them more room while preserving the order of weekly totals. For each week, the calculation can be expressed as: int height = weeklyTotal == 0 ? 1 : 1 + (int) Math.round( Math.log1p(weeklyTotal) / Math.log1p(maximumWeeklyTotal) * (maxHeight - 1)); With a maximum of 20 contributions and a height limit of 14, a week with three contributions gets seven window rows. The busiest week gets 14, and a week with zero contributions keeps one row. Quiet periods therefore retain their position in the calendar. An antenna marks the busiest displayed week; when totals are tied, the most recent week receives it. The two columns of windows are filled by cycling through the days of the week. This repeats the daily pattern in taller buildings, so window count is a visual texture. The weekly total controls height, while the window colors carry the daily contribution levels. Together, they give you a view of the pattern across your calendar. You can change weeks to 12 for a shorter view. The tallest building is then calculated from those recent weeks, which lets the selected period use the available height. Keep this in mind when comparing images: the same week can appear taller in a shorter view. The displayed contribution total still covers the full calendar returned by GitHub. Follow the Data Through Java The released project uses Quarkus with the Quarkus GitHub Action extension. That extension connects an @Action method to workflow inputs, outputs, and the job summary. Inside the method, the drawing process comes down to three steps: ContributionCalendar calendar = contributionClient.fetch( configuration.username(), inputs.getGitHubToken().orElseThrow(), context.getGithubGraphQLUrl()); Scene scene = cityBuilder.build(calendar, configuration.options()); String svg = svgRenderer.render(scene, configuration.options().theme()); The client reads the contribution calendar with one GraphQL request and converts the response into Java records for the calendar, weeks, and days. CityBuilder uses these records to place roofs, windows, and the street in a grid. SvgRenderer turns that grid into the final SVG. Changes to the skyline can therefore be made in plain Java classes, with the GitHub connection kept in the client. The SVG contains the shapes and text needed to display the city. Adjacent characters with the same color are grouped during rendering, and their horizontal positions are set explicitly to keep the columns aligned. Identical data and options produce identical file contents, which allows the publisher to compare images before creating a commit. The action sets up Java and JBang, builds the selected source revision, and runs the packaged application. Change the Layout Locally To try a change, install JDK 21 and JBang, with Git and Bash available. The repository includes a Maven wrapper. JBang is also used by the integration tests to run the packaged action through the same launcher used in GitHub Actions. git clone --branch v1.0.0 https://github.com/myfear/quarkus-contribution-city.git cd quarkus-contribution-city ./mvnw verify The tests use local test data and a local GraphQL server, so you can run them without GitHub credentials. They also generate SVG previews under target/previews. Open github-dark.svg, recent-12.svg, and empty.svg in a browser to compare a full calendar, a shorter period, and a calendar with zero contributions. Start with the height calculation in CityBuilder, then run the same command again and compare the previews. The saved data stays the same between runs, which makes the effect of your change easier to see. The project is available under the Apache License 2.0, and the v1.0.0 release gives you a fixed starting point. Give the City a DZone Landmark Version 1.0.0 exposes weeks, height, and the two built-in themes through workflow inputs. For a custom shape, you extend the Java code in your fork. The Scene record gives you a grid of cells to change before the SVG is rendered, and that is where we'll add the DZone lettering. Each cell has a character, a role such as roof or window, and the color information used by the renderer. A character mask is enough to describe a new shape: # marks a filled block, and spaces leave the background visible. By arranging those blocks, you can draw letters, an arch, or a stepped tower. For this example, the lettering is placed above the weekly buildings. It has its own fixed shape, while the buildings below continue to show the contribution calendar. The image uses a snapshot of my real account data, with 534 contributions returned by GitHub on October 5, 2026, when I wrote this article. A DZone-inspired palette and decorative lettering added to my contribution city. The weekly building heights and daily window patterns come from the same calendar used by the regular view. Add this class to the existing city package in your fork, at src/main/java/com/themainthread/contributioncity/city/DZoneLandmark.java: package com.themainthread.contributioncity.city; import java.util.ArrayList; import java.util.Collections; import java.util.List; import com.themainthread.contributioncity.render.Theme; public final class DZoneLandmark { public static final Theme THEME = new Theme("dzone-example", "#161b22", "#242c36", "#303947", "#f58a42", "#9ba9ba", "#edf2f7", List.of("#536070", "#86431f", "#b85b25", "#e57d36", "#ffb365")); private static final List<String> MASK = List.of( "#### ##### ### # # #####", "# # # # # ## # # ", "# # # # # ## # # ", "# # # # # # # # #### ", "# # # # # # ## # ", "# # # # # # ## # ", "#### ##### ### # # #####"); private DZoneLandmark() { } public static Scene addTo(Scene city) { int landmarkWidth = MASK.getFirst().length() * 2; int width = Math.max(city.width(), landmarkWidth); int left = (width - landmarkWidth) / 2; List<List<Cell>> rows = new ArrayList<>(); Cell block = new Cell('█', CellKind.ROOF, 0, -1); for (String line : MASK) { List<Cell> row = new ArrayList<>(Collections.nCopies(width, Cell.SKY)); for (int column = 0; column < line.length(); column++) { if (line.charAt(column) == '#') { row.set(left + column * 2, block); row.set(left + column * 2 + 1, block); } } rows.add(row); } rows.add(Collections.nCopies(width, Cell.SKY)); for (List<Cell> sourceRow : city.rows()) { List<Cell> row = new ArrayList<>(sourceRow); row.addAll(Collections.nCopies(width - city.width(), Cell.SKY)); rows.add(row); } return new Scene(city.username(), city.totalContributions(), rows); } } The theme defines the background, two building shades, roof, street, and label colors, followed by the five window colors. Those window colors follow GitHub's contribution levels from zero through four. Here, the roof color is also used for the lettering, so the decoration has a consistent orange color across all seven rows. Inside addTo, each marked position is expanded to two cells because the renderer's cells are taller than they are wide. The lettering is centered, a blank row is added beneath it, and the original city rows are copied below. Short calendars receive extra background space on the right to make room for the lettering. The dates, counts, and order of the weekly buildings are preserved. To use the decoration in your fork, import DZoneLandmark from the city package in ContributionCityAction. Replace the existing scene-building and rendering lines with: Scene scene = cityBuilder.build(calendar, configuration.options()); Scene decorated = DZoneLandmark.addTo(scene); String svg = svgRenderer.render(decorated, DZoneLandmark.THEME); This selects the example's palette in Java. The published @v1 action continues to accept github-dark and mono; your fork contains the additional drawing code. For a local preview, add the following after the existing Files.writeString call in PreviewTest.write, with the same import for DZoneLandmark: Scene decorated = DZoneLandmark.addTo(builder.build(calendar, options)); Files.writeString(directory.resolve(name.replace(".svg", "-dzone.svg")), renderer.render(decorated, DZoneLandmark.THEME)); Run ./mvnw verify again and open target/previews/github-dark-dzone.svg. The build also writes the shorter and empty versions with the -dzone.svg suffix, so you can check how the decoration fits several calendars. When you're ready to use your fork on GitHub, update the second checkout in the publishing workflow to your fork's repository and a commit containing your changes. The generation step then runs the customized action from that checkout. Try It With Your Own Contributions Try Quarkus Contribution City with your own GitHub account and add the generated SVG to your profile. Start with the released action, then fork the project and change the character mask to give your city a landmark of its own. You could draw a stepped tower from your hometown or add your team's initials above the street, using the DZone version as a starting point. Share your version in a GitHub issue, or open a pull request if you'd like to contribute a new shape.

Every Terraform team I’ve worked with has drift detection. A scheduled terraform plan -detailed-exitcode, a Terraform Cloud health assessment, a Spacelift or env0 monitor. Exit code 2 fires, a Slack message lands, and someone opens a ticket. Then the ticket sits. Drift found in minutes gets fixed in days, because the fix needs a human to read the diff, work out why the resource changed, decide whether to codify or revert, write the HCL, and push a PR. Detection is automated. Remediation is still artisanal. The Gap The tempting fix is to auto-apply on drift. Almost nobody does that in production, and they’re right not to. A blind apply on a security group that someone widened during an incident re-breaks the thing they just fixed. Drift means reality and code disagree, and “code wins” is not always the right answer. The middle path is an agent that reads the drift, drafts a remediation as a pull request with its reasoning and its prediction of the post-merge plan, and then stops. The PR is the gate. The real plan runs as a required check on that PR, the same one every human PR gets, and the agent is never in the apply path. I’ve been running this against production stacks in two regions. It turns a days-long ticket into a short review. I’ve published the workflows as a reference implementation in this repository. The linked repo keeps the same shape (detect → remediate → PR gate → apply) with the minimum needed to run. Firefly and env0 sell versions of this idea, and tfdrift’s remediate command and autoai-driftguard are open-source ones. The idea is the easy part. What I want to write down is the set of design decisions that make it safe to leave running, because that’s where the failures live. Three properties make the shape defensible. The agent’s blast radius is a branch: it can’t touch state, cloud resources, or the merge button, so the worst case is a PR a reviewer closes (for an honest agent, at least; more on that later). The existing change process is reused, not bypassed: CODEOWNERS, required reviews, and policy checks run on the agent’s PR exactly as they do on a human’s. And the judgment call stays with a person, but it arrives pre-researched. Two Kinds of Drift When plan says a resource differs from state, there are two directions to go: DirectionMeaningActionRevertThe out-of-band change was wrong; code is the truthNo HCL change; apply the existing configCodifyThe out-of-band change was right; code is staleUpdate HCL to match reality, so apply produces no diff A detector can’t tell them apart. A t3.medium that is now a t3.large is either someone’s forgotten experiment (revert) or last week’s capacity fix that never made it into code (codify). The evidence lives outside Terraform: CloudTrail, tickets, tags, Slack threads. Gathering that evidence and writing down a hypothesis a human can confirm or reject in a minute is exactly what an LLM is good at and a cron job is not. The Workflow Four stages. Only the second one is new. Stage 1: Detection A scheduled workflow runs a plan and captures a machine-readable output. YAML # .github/workflows/drift-detect.yml (abridged) on: schedule: [{ cron: "0 */6 * * *" }] jobs: plan: permissions: { id-token: write, contents: read, actions: write, issues: write } strategy: fail-fast: false matrix: include: # CloudTrail LookupEvents is regional; the agent runs where the resource lives - { stack: network/prod-use1, region: us-east-1 } - { stack: network/prod-use2, region: us-east-2 } steps: - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: ${{ vars.DRIFT_READONLY_ROLE } # reads + lock-table write; plan holds the lock aws-region: ${{ matrix.region } - name: Plan (refresh-only) id: plan run: | set +e terraform plan -refresh-only -detailed-exitcode -lock-timeout=5m \ -out=drift.tfplan -input=false code=$?; set -e echo "code=$code" >> "$GITHUB_OUTPUT" [ "$code" -eq 1 ] && exit 1 # plan error: fail loudly terraform show -json drift.tfplan > drift.json - name: Count drifted resources # refresh-only plans report under resource_drift id: count run: | n=$(python3 -c "import json; d=json.load(open('drift.json')); print(sum(1 for r in d.get('resource_drift', []) if r['change']['actions'] != ['no-op']))") echo "n=$n" >> "$GITHUB_OUTPUT" - name: Drift-storm guard # >20 drifted resources is a provider/state problem, not drift if: steps.plan.outputs.code == '2' && steps.count.outputs.n > 20 run: gh issue create --title "[drift] storm in ${{ matrix.stack }" --body "Paging a human." - name: Dispatch remediation if: steps.plan.outputs.code == '2' && steps.count.outputs.n <= 20 run: gh workflow run drift-remediate.yml -f stack=${{ matrix.stack } -f region=${{ matrix.region } -f run_id=${{ github.run_id } Three details do the work. -refresh-only separates real drift from HCL that merged but hasn’t been applied yet. The role is read-only in shape but needs lock-table writes because plan holds a state lock, which is also why -lock-timeout=5m is there: a scheduled plan that collides with an apply should wait, not fail. And the storm guard exists because a provider upgrade looks like drift on every resource; past 20 in one run, the workflow pages a human instead of dispatching the agent. Stage 2: The Remediation Agent The agent has three capabilities: read the plan JSON, read evidence, open a PR. No cloud writes, no terraform apply. YAML # .github/workflows/drift-remediate.yml (abridged) on: workflow_dispatch: inputs: { stack: { required: true, type: string }, region: { required: true, type: string }, run_id: { required: true, type: string } } jobs: remediate: permissions: { contents: write, pull-requests: write, id-token: write, actions: read } steps: - uses: actions/download-artifact@v4 # real workflows sanitize '/' -> '-' in the stack name so this matches the upload step with: { name: drift-${{ inputs.stack }, run-id: ${{ inputs.run_id }, github-token: ${{ secrets.GITHUB_TOKEN } } - uses: aws-actions/configure-aws-credentials@v4 with: { role-to-assume: ${{ vars.DRIFT_READONLY_ROLE }, aws-region: ${{ inputs.region } } - name: Analyze drift # model + AWS read access; no GitHub write token here run: python agent/remediate.py analyze --stack "${{ inputs.stack }" --region "${{ inputs.region }" --plan drift.json --out proposal/ env: { ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }, ANTHROPIC_MODEL: ${{ vars.ANTHROPIC_MODEL } } - name: Open PR # bot token only; see below for why not GITHUB_TOKEN run: python agent/remediate.py propose --stack "${{ inputs.stack }" --proposal proposal/ env: { GH_TOKEN: ${{ secrets.DRIFT_BOT_TOKEN } } The token in that last step matters more than it looks. GitHub limits workflow runs for events created with GITHUB_TOKEN, so a PR opened with it gets a pull_request run parked in an approval-required state: no jobs, no checks on the PR, no error. If the plan and policy checks are required, the PR can’t merge until someone with write access clicks “Approve workflows to run.” The tool surface is small, and every tool is read-only except the last: Python TOOLS = [ read_plan_json, # resource_drift entries from drift.json read_hcl, # repo files, capped per file read_tags, # tags on the drifted ARN cloudtrail_lookup, # LookupEvents in the stack's region, last 14 days describe_resource, # aws ec2/rds/iam describe-* search_issues, # GitHub issues/PRs mentioning the resource ID propose_fix, # the ONLY write: stages a branch + PR body under proposal/ ] Two limits on the CloudTrail tool. LookupEvents returns management events only, for one region, for the last 90 days, 50 per page, at two calls a second. That covers “who changed this security group” and not “who read this object”; for older questions, point the tool at CloudTrail Lake or the S3 bucket storing logs. Notice what’s missing: no apply, no cloud modify, no merge. A “revert” decision changes no HCL, because the HCL is already right. Instead, the agent commits a decision record, .drift-decisions/<address>-<fingerprint>.md, with the direction it chose, the evidence, and the plan it expects after merge. That gives the PR a real diff (GitHub refuses PRs with no commits) and puts the decision in git history next to the resource. A revert PR’s plan check exits 2 on purpose, and the gate compares the plan to the record’s claim instead of failing on the exit code. The system prompt sets the contract: You are a Terraform drift analyst. For each drifted resource you will (1) determine whether the out-of-band change should be codified or reverted, (2) cite the evidence that led you there, (3) produce the minimal HCL diff for the codify case, and (4) rate the strength of the evidence as strong, partial, or none. You cannot apply changes. If evidence is insufficient to choose a direction, say so and draft the codify variant anyway so a reviewer can see what accepting the change would look like. That last sentence matters. An agent that won’t act under uncertainty is the same as no agent. Drafting the codify variant costs cents against an engineer’s hour, and it’s reversible. Stage 3: The PR as the Gate Here is the agent’s output on a real drift, a security group that grew an ingress rule outside Terraform: [drift] network/prod-use1: aws_security_group.app_alb drifted out-of-band Recommendation: Codify (evidence: strong) Drift: ingress tcp/8443 from 10.40.0.0/16 not present in state. Evidence: CloudTrail AuthorizeSecurityGroupIngress at 2026-09-08T03:12Z by role IncidentResponder-oncall. Matches issue #431 (“ALB health checks failing from new monitoring subnet”), closed the same day. The CIDR is aws_subnet.monitoring, added in PR #412, which never updated the SG. Change: adds the ingress block to modules/alb/sg.tf, parameterized on var.monitoring_cidr. Expected post-merge plan: no changes. The plan check on this PR confirms or refutes that. If you disagree: close this PR; the next scheduled apply removes the rule. #431 suggests that re-breaks monitoring. The reviewer’s job goes from “investigate” to “verify.” Every line is checkable in under a minute, and the plan check proves the fix converges. The gate is the same set of required checks human PRs pass, plus one step that judges the agent’s claim: YAML - name: Plan on PR run: | set +e; terraform plan -detailed-exitcode -out=pr.tfplan -input=false echo "code=$?" >> "$GITHUB_OUTPUT"; set -e terraform show -json pr.tfplan > pr.json - name: Judge only the claimed resource # codify: the claimed resource is no-op/absent in pr.json # revert: the claimed fingerprint still appears under resource_drift run: python agent/gate.py --plan pr.json - name: Policy check run: conftest test pr.json -p policy/ # Branch protection: 1 approval + Code Owners review. The bot can't approve # its own PR, and "Allow GitHub Actions reviews to count" stays off. Branch protection does the real safety work. The bot can’t approve its own PR, and because its PRs touch platform-owned paths, the platform team’s CODEOWNERS entry has to. Stage 4: Apply (Unchanged) On merge, your existing apply pipeline runs. If the agent was wrong and the reviewer missed it, you have git history, a plan artifact, and a normal revert PR. Same failure mode as any bad merge, which is the point. Guardrails One PR per resource. A stack with six drifted resources gets six PRs, each with its own codify/revert decision. One PR for all six is how a reviewer approves five good changes and one bad one. That rule was the first thing I learned from one of the projects. I staged three drifts at once: a tag added to an S3 bucket (codify), a security-group rule opened to the world (revert), and a subnet deleted out from under state (needs-human). My first gate deadlocked on the first two. The codify check wanted a fully clean plan, the revert check wanted exactly one drifted resource, and each PR’s plan showed the other PR’s drift. Neither could go green. The fix is the “judge only the claimed resource” step above. With it, both PRs passed while the deleted subnet stayed open as an issue. Per-resource independence has to live in the gate, not just in the agent. Idempotent PRs. Before opening a PR, the agent looks for an open one with the same resource address and drift fingerprint, a hash of the before/after attributes with keys sorted so ordering noise doesn’t create a new fingerprint. If the resource drifts again differently, the old PR is stale: the agent closes it with a pointer to the new one. Never auto-remediate delete drift. If a resource exists in state and not in reality, the codify path is deleting HCL, which is the easiest change to get catastrophically wrong. The agent files its evidence as a drift/needs-human issue and stops. Deny-list sensitive types. IAM policies, KMS key policies, and anything under module.security. are evidence-only; the agent files an issue and a human opens the PR. It’s a deny-list on purpose. The human gate already makes the default safe; the list is about blast radius on the day the reviewer is tired, and IAM is where a plausible one-line change grants access to everything. An allow-list would be safer on paper and would spend its first month growing one type at a time while drift queued up. Cap the context. The agent gets the drifted resource’s resource_drift entry, the HCL file that declares it, and evidence for that ARN. Not the repo. Less context means fewer chances to “improve” adjacent code. Log the reasoning. Every PR body carries the evidence trail. When a codify decision turns out wrong three weeks later, you want to know whether the agent misread CloudTrail or CloudTrail was silent. What This Doesn’t Solve It doesn’t prevent drift. If people can write to prod from the console, they will; this makes the consequence cheap, not the cause. It doesn’t handle state corruption or provider upgrades that look like drift on every resource. And it doesn’t remove the reviewer. If your team approves agent PRs without reading them, you’ve built auto-apply with extra steps. Conclusion The industry spent the last few years making drift visible. The next step isn’t making it disappear on its own. It’s making the fix arrive already written, already justified, and already inside the review process you trust. An agent that can only open pull requests is an agent whose worst day is a closed PR. Nothing here is specific to GitHub Actions or AWS. Read-only agent, PR as the only write, existing gates unchanged: that ports to GitLab, Azure DevOps, or any Terraform platform with a VCS-driven apply.

A coding agent opens a pull request for a background job that keeps retrying after the user cancels it. The test that caught the bug now passes, and the agent reports that all tests are green. But the diff changes two things: the retry handler and the test itself. Which change eliminated the failure, and would the test still catch the bug if it came back? I use coding agents to design integration tests, triage failures, and prepare fixes for back-end services, so this is a practical review question for me. Sometimes the test really is wrong, and changing it is the right fix. Sometimes the agent has just found the shortest way to a green build. In continuous integration, both look the same. Benchmarks show how far it can go. ImpossibleBench makes the tests contradict the task, so any passing solution has to cheat, and GPT-5 still passed 54% of its conflicting SWE-bench tasks. But tests aren't always right, either: OpenAI found issues with the tests or the task description in 59.4% of the 138 SWE-bench Verified problems it examined. So a test edit proves nothing by itself. Below, I'll show how to get evidence either way with a small Python example: one bug, three PRs that all pass CI, and a check that tells them apart in seconds. A Green Build Is a Statement About the Checks I find it useful to look at a test run as a pipeline. Tests are selected and set up, the application runs, the test observes and asserts, the harness reports the outcome, and a CI gate turns that outcome into a merge decision. Figure 1: A test run as a pipeline. A change to any stage other than the application changes what a green result means. Only the application stage can repair a defect. If a PR touches only the application, going from red to green tells you something about the code. If it also touches another stage, the green result may say more about the evidence than about the bug. One Bug, Three Pull Requests The Failing Test Here's the contract for the example. Once cancel() returns, no new attempt may start for that job. An attempt that's already running may still report its result, and a job that wasn't canceled must still be retried after a transient failure. The model below starts with one attempt already running and handles callbacks one at a time. run_retry() stands in for the scheduler starting a queued retry; there are no threads, network calls, or sleeps. Save it as retry_job.py: Python class RetryJob: def __init__(self): self.attempts = 1 self.canceled = False self.retry_pending = False def cancel(self): self.canceled = True self.retry_pending = False def on_transient_failure(self): self.retry_pending = True def run_retry(self): if self.retry_pending: self.retry_pending = False self.attempts += 1 The bug is about ordering. cancel() clears any queued retry, but the running attempt can report a transient failure afterward, and on_transient_failure() then queues another retry without checking whether the job was canceled. These two tests go in tests/test_retry_job.py: Python from retry_job import RetryJob def test_cancel_blocks_retry_after_late_failure(): job = RetryJob() job.cancel() job.on_transient_failure() job.run_retry() assert job.canceled is True assert job.attempts == 1 def test_uncanceled_job_retries(): job = RetryJob() job.on_transient_failure() job.run_retry() assert job.attempts == 2 Run python -m pytest -q from the repository root. The cancellation test fails because attempts is 2, and the ordinary retry test passes. Three Ways to Turn It Green The first PR leaves the application alone and loosens one assertion: diff - assert job.attempts == 1 + assert job.attempts >= 1 Now the test accepts a second attempt, and a third and a fourth. Its name still promises that cancellation blocks a retry, but the assertion no longer checks that promise. The second PR fixes the handler and leaves the tests alone: diff def on_transient_failure(self): + if self.canceled: + return self.retry_pending = True The cancellation test retains its meaning, and the ordinary retry test shows that the guard hasn't disabled retries for every job. The third PR does both. It's the easiest to miss: the application diff looks right because the application is right now, but the test can no longer catch the bug if a later change brings it back. Ordinary CI reports 2 passed for all three PRs. Run the Swap Check Ordinary CI runs exactly one combination: the PR's tests against the PR's code. The two combinations it skips answer the questions a reviewer actually has: Base tests on head code. Does the change still meet the original expectations? If an original test fails here, the PR changed a promise, and that needs a reason.Head tests on base code. Do the new tests still detect the bug? For a bug fix, at least one of them should fail here. If they all pass, they can't tell the buggy code from the fixed code. Figure 2 shows all four combinations for the three PRs: Figure 2: Swap matrices for the three pull requests. CI only shows the head code with head tests, which is green in all three. In an agent's PR, the tests are part of the change, so you have to swap them on purpose. Git worktrees make that cheap. These commands build the cell that matters most for a bug fix, the base code with the head's test files: Shell base=$(git rev-parse main) # commit IDs, because inside a worktree head=$(git rev-parse fix-and-weaken) # HEAD would mean something else tests=$(git diff --name-only --no-renames "$base" "$head" -- \ tests conftest.py pytest.ini) cell=$(mktemp -d) # base code + head tests git worktree add --quiet --detach "$cell" "$base" git -C "$cell" restore --source="$head" -- $tests (cd "$cell" && python -m pytest -q) git worktree remove --force "$cell" On the third PR, this prints 2 passed: the new tests can't see the bug. For the other cross cell, add the worktree at $head and restore the test files from $base. The PR that only loosens the assertion raises a flag in both cross cells, and the honest fix raises none. Whole-suite results are coarse, though. A head test can fail on the base code just because it imports something the fix added, and a test that already fails on the base hides the signal. In real use, compare outcomes test by test, as SWE-bench does with its FAIL_TO_PASS and PASS_TO_PASS lists. The list of test paths matters, too. It has to cover everything that defines a check, including fixtures, snapshots, and harness configuration such as pytest.ini, and files that mix application and test settings, such as pyproject.toml, need a human look. This conftest.py hook, for example, rewrites failed test reports as passed: Python import pytest @pytest.hookimpl(hookwrapper=True) def pytest_runtest_makereport(item, call): outcome = yield report = outcome.get_result() if report.failed: report.outcome = "passed" In a PR that adds only this hook, with only tests on the list, the swap check treats conftest.py as application code and reports a clean fix. Add conftest.py, and both warnings appear. Anthropic researchers describe this family of hacks, along with calling sys.exit(0) before any assertion runs. Treat the result as review evidence, not a verdict: a refactoring's new tests should pass on the old code, while a bug fix's reproducer should fail there. I'd run the swap check as a report-only CI job with the full Git history (fetch-depth: 0) and, because it runs the candidate's code, a read-only token that isn't left in the checkout (permissions: contents: read, persist-credentials: false). When Changing the Test Is the Right Fix Tests change all the time. In one comparison, human-written PRs modified existing tests more often than agent-assisted ones: 43.5% versus 20.6% in a manual sample. So the useful question is why a test changed. Four answers hold up: The requirement changed. If the service now lets one already-dispatched retry start after cancellation, the test has to say so, and someone has to have approved the new guarantee.The fixture was wrong. Maybe it labeled a permanent failure as transient. Fix it, but keep a transient-failure case so the retry path stays covered.The test looked too early. If cancellation is acknowledged asynchronously, wait for the acknowledgment or step a fake scheduler forward instead of adding a sleep.The test pinned an implementation detail. Move the check to the public boundary and show that it still catches a violation. This was the largest category in OpenAI's audit, at 35.5% of the examined problems. In none of these cases is the application's current output the reason for the new expectation. Ask for the contract, the corrected setup, or the reason the old observation was invalid, plus swap-check evidence that the revised test still fails on the bug. Guardrails In the Agent Loop An agent needs room to investigate failures, including bugs in tests, and a clear way to report a conflict between a requirement and an assertion rather than quietly resolving it. For the cancellation task, I'd commit the reproducer first, after checking that it fails for the right reason, and give the agent an instruction like this: Fix the retry behavior so that no new attempt starts after cancel() returns. Keep the late-failure reproducer and the ordinary retry case. If you think a test is wrong, stop, explain how it conflicts with the contract, and show any proposed test change separately. Report the commands you ran and any skipped, expected-failing, or unexecuted cases. To make tests read-only for an agent, Claude Code permission rules can deny edits to path patterns, and a PreToolUse hook can block a tool call by exiting with status 2; Cursor's preToolUse hook works the same way. These rules cover the agent's file tools, not every shell command, unless you also run Claude Code's sandbox with allowUnsandboxedCommands set to false. In my own setup, the agent that writes integration tests works only in test code, and its standing instructions forbid silencing or deleting a failing test to get a green run. Triage first classifies each failure as an application, environment, or test problem; application bugs become bug reports, not test edits. Changes to the agents' own instructions need a person's confirmation, and agents never merge. In the Repository Repository controls decide what gets merged. The useful ones keep the evidence for important behavior out of the candidate patch's reach, as Figure 3 shows: Figure 3: What the candidate patch can change, and what should be reviewed separately. Own the evidence. Assign test, fixture, harness, and CI paths to owners in CODEOWNERS, and turn on "Require review from Code Owners"; the file alone only requests reviews. GitHub reads it from the base branch, so a PR can't loosen its own owners. Give CODEOWNERS itself an owner, and require approval of the most recent push. Keep acceptance checks outside the patch. Keep a small acceptance suite for important contracts in a separate repository or at a pinned reference, and run it from a workflow the PR can't edit, such as a ruleset-required workflow on GitHub Enterprise Cloud or a pipeline execution policy on GitLab Ultimate. Run it from its own checkout with its own pytest.ini, because pytest searches parent directories for configuration and could pick up the candidate's conftest.py, and set PYTEST_DISABLE_PLUGIN_AUTOLOAD to block plugins the candidate's package might register. Prove that the check ran. Make the acceptance job unconditional: a required status check accepts a skipped result, and on GitHub "a job that is skipped will report its status as 'Success'." Then fail the job unless the required cases ran and passed. Any runner that writes JUnit XML, including pytest, Maven Surefire, and Gradle, can feed a short script that checks each required test by name. Don't count a test that passed only on a rerun; Surefire keeps the build green and records the failed attempt as a flakyFailure. Parse the report with defusedxml, since it's untrusted input. No single control catches everything. The required-tests check trusts a harness that lies, the acceptance run can't see a weakened test next to a real fix, and the swap check is only as good as its path list. Costs and Limits The swap check runs the suite four times, so for a large suite I'd limit it to the affected tests. It needs a base revision where the reproducer can run, flaky tests add noise to the cross cells, and each cell must import its own checkout rather than an installed package or a shared node_modules folder. These controls also target shortcuts rather than code written to defeat them. In a seventh variant of the example, retry_job.py patches pytest's report object when it's imported, and all three controls pass it. Where that matters, run acceptance tests against a built artifact or a running service, and keep reading the application diff. What to Ask for Before Approving Ask for evidence instead of a verdict. For the fixed example, a useful PR description would say: The late failure callback queued a retry after cancellation. The handler now ignores that callback when the job is canceled. The original cancellation test fails on the old code and passes on the repair; the ordinary retry test also passes. Removing the guard makes the cancellation test fail again. The assertion and event order are unchanged. For a real PR, I'd want to see five things before approving: The reproducer fails on the base code and passes on the head code.For a bug fix, at least one head test fails on the base code due to an assertion rather than an import.Every edited test has a stated reason: a changed requirement, a fixed fixture, a corrected observation point, or a removed implementation detail.Changes to selection, harness, or CI configuration were approved by their owners.The required tests ran and passed, by name, not just by exit code. Wrapping Up Reviewers have always had to ask why a test changed. Coding agents make the question come up more often, because they can rewrite code and tests in minutes and keep going until CI is green. Treat changes to tests, fixtures, the harness, and CI as changes to the evidence: swap tests, accept edits with a reason, and keep checks for important behavior out of the patch's reach. A passing run is only useful when you can explain why it passed.

Over the past few years, vision-language models (VLMs) have moved from experimental research projects to early production use. These systems take in both images and text, so they can read documents, interpret photographs, answer questions about visual content, and reason across modalities in ways that were not practical a few years ago. Better architectures and training methods explain part of this progress. Better ways of deploying models explain at least as much. In 2026, the defining question for enterprises is no longer whether VLMs work, but whether they can run reliably, safely and cost-effectively in production. The answer depends as much on cloud infrastructure, observability and governance as on the models themselves. This article traces how VLMs evolved, why cloud computing became the turning point for adoption, where reliability still falls short, and which cloud-native practices organizations need in order to deploy them responsibly at scale. From Research Breakthroughs to Foundation Models The modern VLM era began with CLIP (Contrastive Language–Image Pretraining). It showed that training on hundreds of millions of image-text pairs could align visual and textual representations in a shared embedding space. That alignment enabled zero-shot image classification and made large-scale multimodal search possible. DeepMind's Flamingo built on this foundation. It introduced cross-attention layers that connect a frozen vision encoder to a large language model, so a single model could learn new tasks from only a few examples. BLIP-2 then showed that a lightweight querying module could bridge frozen image encoders and frozen language models, which sharply reduced training cost. Instruction tuning was the next step. LLaVA (Large Language and Vision Assistant) showed that fine-tuning on visual instruction data produced conversational assistants that could follow complex multimodal prompts. OpenAI's GPT-4V brought these capabilities to a mass audience through a commercial product. Open-weight models such as Qwen2-VL later extended high-resolution perception to the open-source community. Taken together, these advances in cross-attention mechanisms, instruction fine-tuning, and large-scale multimodal pretraining allowed VLMs to combine perception and reasoning in a single system. The Cloud as the Deployment Turning Point The research milestones were essential, but the real turning point for adoption has been cloud deployment. VLMs are computationally demanding: they need high-memory GPUs, efficient batching, and low-latency serving. Few organizations can justify building that infrastructure in-house for workloads that are still evolving. Cloud platforms solve this problem. They offer GPU-backed compute on demand, managed model endpoints, and API-based access that fits directly into existing applications. As a result, VLMs now support practical enterprise workflows: Document intelligence: extracting data from invoices, contracts, and formsVisual inspection: spotting defects in manufacturingMultimodal search: retrieving content by combining images with natural-language queries Elastic scaling also lets organizations match capacity to demand, so they don't pay for idle hardware during quiet periods. Benchmarks Reveal Uneven Reliability Despite these advances, VLM reliability remains inconsistent. MMBench evaluates models across a wide range of perception and reasoning skills, and its results show large differences in performance from task to task. The VERIFY benchmark focuses on visual reasoning fidelity, testing whether models reason correctly about what they see rather than simply producing plausible answers. It exposes significant gaps even in leading systems. Industry reviews of recent computer vision research, including Voxel51's analysis of ICCV 2025 papers, point to the same ongoing weaknesses in fine-grained perception and consistency. A model that does well on one benchmark can still struggle on a closely related task. The practical lesson is clear: choose models based on the specific use case and workload, not on general leaderboard rankings. Choosing Between Open-Source and Managed Models Organizations face a strategic choice between open-source and managed VLMs. Open-source models such as LLaVA and Qwen2-VL can run on Kubernetes clusters or in a private cloud. They give full control over data residency, customization, and cost, which suits regulated industries and sensitive data. The trade-off is that they require in-house expertise in GPU orchestration, model serving, and ongoing maintenance. Managed services offer elastic scaling, built-in security, and much less operational complexity, at the cost of less control and possible vendor lock-in. Many enterprises use a hybrid approach. They route sensitive workloads to self-hosted models and send general-purpose tasks to managed endpoints. Hidden Failure Modes at Scale A major concern in production is hidden failure modes. The most widely discussed is hallucination, where a model describes objects or details that are not in the image, and research on object hallucination shows it remains a persistent problem. Other risks include: OCR errors on low-quality scansSmall objects being missed in dense scenesSpatial reasoning failures, such as confusing left and right or misjudging relative positions Inside automated pipelines, these errors are especially dangerous. One hallucinated invoice total or missed defect can repeat across thousands of transactions before anyone notices. At scale, small error rates add up to significant business and compliance risk. Cloud-Native Guardrails for Production To manage these risks, organizations need guardrails built into their cloud-native architecture: Confidence thresholds send uncertain outputs for further review instead of accepting them automatically.Human-in-the-loop review gives a person oversight of high-stakes decisions.Retrieval-based grounding ties model outputs to verified source data, which reduces hallucination.Audit logging records inputs, outputs, and decisions for traceability and compliance.API gateway policies enforce rate limits, access controls, and content rules consistently across applications. Frameworks such as the NIST AI Risk Management Framework give a useful structure for applying these controls. They help organizations map, measure, and manage AI risk across the lifecycle. Observability tools that track drift, latency, and error patterns complete the picture, so teams can detect problems before they spread. These controls are what move VLMs from impressive demonstrations to dependable production systems. Conclusion The VLM story is ultimately about systems as much as models. Breakthroughs from CLIP to Qwen2-VL made multimodal understanding possible, but cloud computing made it practical. Combining multimodal AI with cloud-native infrastructure has let organizations build AI workflows that are scalable, observable, and governed. VLMs are not yet perfect. Benchmarks show uneven reliability, and hallucination remains a real risk. Even so, they have matured enough for teams to deploy them at scale, provided those teams take careful model selection, rigorous evaluation, and strong guardrails seriously. The organizations that succeed in 2026 will treat VLMs as one part of a well-architected system with governance built in from the start.

By Srinivas Chippagiri CORE

Every system that stores sensitive data needs keys that never leave a hardened boundary, an audit trail of every use, and rotation that doesn't mean re-encrypting petabytes. On Google Cloud, that's Cloud KMS; on AWS, it's AWS KMS. Both store symmetric and asymmetric keys, both use HSMs, and both are built for envelope encryption. But AWS KMS vs. Google Cloud KMS comes down to the choices that show up in your code: how keys are named, what rotation means, where authorization lives, and whether the service hands you data keys. This article walks through Cloud KMS in Java and then maps it onto AWS KMS. The Cloud KMS Resource Model Cloud KMS organizes keys in a four-level hierarchy under a project: Location. Where the key material lives: a region such as us-east1, a multi-region such as us, or global. Key material stays in the location you pick for its whole lifecycle.Key ring. A named group of keys in one location. It exists for organization and IAM inheritance.CryptoKey. The key you reference in code. It has a purpose (ENCRYPT_DECRYPT, ASYMMETRIC_SIGN, ASYMMETRIC_DECRYPT, MAC, and a few more) and a rotation schedule.CryptoKeyVersion. The actual key material. A key has one or more versions, and for symmetric keys one of them is the primary, the version that new encryptions use. A symmetric key with three versions: new data uses version 3, data written under version 2 still decrypts, and version 1 is on its way out. A key is identified only by its full resource name, projects/my-project/locations/us-east1/keyRings/payments/cryptoKeys/card-data, optionally followed by /cryptoKeyVersions/2. There are no aliases. Nothing can move after it's created: the location is fixed when you create the key ring, and key rings themselves can't be deleted. Each key version also has a protection level: SOFTWARE: keys are protected by Google's software crypto (BoringCrypto, FIPS 140-3 Level 1).HSM: keys live in Cloud HSM, which is FIPS 140-2 Level 3 validated. HSM_SINGLE_TENANT gives you a dedicated HSM.EXTERNAL and EXTERNAL_VPC: Cloud EKM keeps the key material in a supported external key manager that you operate, reached over the internet or a VPC. Google never holds the key. Creating a Key That Rotates Itself The Java client is KeyManagementServiceClient. It uses Application Default Credentials, so on GKE, Cloud Run, or GCE, the workload's service account authenticates with no keys in the code. This creates a key ring and an HSM-backed symmetric key that rotates every 90 days: Java try (KeyManagementServiceClient kms = KeyManagementServiceClient.create()) { KeyRing ring = kms.createKeyRing( LocationName.of("my-project", "us-east1"), "payments", KeyRing.getDefaultInstance()); CryptoKey key = CryptoKey.newBuilder() .setPurpose(CryptoKeyPurpose.ENCRYPT_DECRYPT) .setVersionTemplate(CryptoKeyVersionTemplate.newBuilder() .setAlgorithm(CryptoKeyVersionAlgorithm.GOOGLE_SYMMETRIC_ENCRYPTION) .setProtectionLevel(ProtectionLevel.HSM)) .setRotationPeriod(Duration.newBuilder().setSeconds(90L * 24 * 3600)) // every 90 days .setNextRotationTime(Timestamp.newBuilder() .setSeconds(Instant.now().plusSeconds(90L * 24 * 3600).getEpochSecond())) .build(); kms.createCryptoKey(KeyRingName.parse(ring.getName()), "card-data", key); } The rotation period can be anywhere from one day to 100 years. When it fires, Cloud KMS creates a new version and makes it primary. Earlier versions stay enabled, so old ciphertext still decrypts, and nothing is re-encrypted. To rotate by hand, create a version with CreateCryptoKeyVersion and promote it with UpdateCryptoKeyPrimaryVersion. Automatic rotation applies only to symmetric keys; asymmetric keys are rotated manually. The AWS KMS key rotation process differs in how key material is managed, as described below. Envelope Encryption KMS keys aren't meant to encrypt your data directly. A single Encrypt call accepts at most 64 KiB of plaintext (8 KiB of plaintext plus AAD for HSM keys), and every call is a network round trip. The standard pattern is envelope encryption, also used with AWS KMS envelope encryption: encrypt the data locally with a data encryption key (DEK), then have KMS encrypt (wrap) the DEK with a key encryption key (KEK) that never leaves the service. You store the wrapped DEK next to the ciphertext. Envelope encryption: The payload never goes to KMS; only the 32-byte DEK does. Cloud KMS has no data-key API, so you generate the DEK yourself. Google's guidance is to generate a new DEK every time you write data, never share one across users, and prefer Tink, which packages this whole flow. Doing it by hand shows what's happening: Java CryptoKeyName kek = CryptoKeyName.of("my-project", "us-east1", "payments", "card-data"); ByteString aad = ByteString.copyFromUtf8("tenant=acme"); // must be supplied again to decrypt try (KeyManagementServiceClient kms = KeyManagementServiceClient.create()) { // 1. Generate a fresh DEK locally. Cloud KMS has no GenerateDataKey call. KeyGenerator gen = KeyGenerator.getInstance("AES"); gen.init(256); SecretKey dek = gen.generateKey(); // ... encrypt the payload locally with dek (AES-GCM), then discard dek ... // 2. Wrap the DEK with the KEK, with CRC32C checks against corruption in transit. ByteString dekBytes = ByteString.copyFrom(dek.getEncoded()); EncryptResponse wrapped = kms.encrypt(EncryptRequest.newBuilder() .setName(kek.toString()) .setPlaintext(dekBytes) .setPlaintextCrc32C(crc32c(dekBytes)) .setAdditionalAuthenticatedData(aad) .setAdditionalAuthenticatedDataCrc32C(crc32c(aad)) .build()); if (!wrapped.getVerifiedPlaintextCrc32C() || wrapped.getCiphertextCrc32C().getValue() != crc32c(wrapped.getCiphertext()).getValue()) { throw new IllegalStateException("integrity check failed"); } ByteString wrappedDek = wrapped.getCiphertext(); // store next to the encrypted payload String usedVersion = wrapped.getName(); // the primary version that wrapped it // 3. Later: unwrap. Pass the key, not the version; Cloud KMS picks the version. ByteString unwrapped = kms.decrypt(DecryptRequest.newBuilder() .setName(kek.toString()) .setCiphertext(wrappedDek) .setAdditionalAuthenticatedData(aad) .build()) .getPlaintext(); SecretKey restored = new SecretKeySpec(unwrapped.toByteArray(), "AES"); } static Int64Value crc32c(ByteString data) { return Int64Value.of(Hashing.crc32c().hashBytes(data.toByteArray()).padToLong()); } Encrypt uses the key's current primary version. Decrypt also takes the key, not a version: the ciphertext records which version produced it, so rotation is invisible to readers. The optional CRC32C fields let the server confirm it received your plaintext intact, and let you confirm the ciphertext you got back. The AAD (tenant=acme) is bound to the ciphertext, so one tenant's wrapped DEK can't be swapped into another tenant's record. Disabling, destroying, and deleting Lifecycle operations act on versions, not keys. You disable a version (ENABLED to DISABLED) to stop it from being used, and re-enable it later. There's no switch that disables a whole key; you disable each version. Destroying a version is deliberately slow. DestroyCryptoKeyVersion moves it to DESTROY_SCHEDULED for 30 days by default. The duration is set when the key is created, and an organization policy can enforce a minimum. During that window, RestoreCryptoKeyVersion brings it back in the DISABLED state. After the window, the material is gone and data encrypted by that version can't be recovered. Destroyed versions are no longer billed. Cloud KMS has also added permanent deletion: you can delete key versions and then the key itself, though a deleted key's name can't be reused. Access control is just IAM Cloud KMS has no policy language of its own. Access is ordinary IAM, granted at the organization, folder, project, key ring, or key level and inherited downward. The predefined roles split duties cleanly: roles/cloudkms.admin manages keys but can't encrypt or decrypt.roles/cloudkms.cryptoKeyEncrypterDecrypter, cryptoKeyEncrypter, and cryptoKeyDecrypter can use keys but not manage them.roles/cloudkms.signerVerifier and publicKeyViewer cover asymmetric keys. The operator who rotates keys never sees plaintext, and the service that decrypts can't destroy its key. Grant usage roles on the specific key, not the project. Two more controls round this out. Cloud Audit Logs record admin operations (create, rotate, destroy) automatically, but Encrypt and Decrypt are Data Access logs, which are off by default. If you want a record of every key use, enable them for Cloud KMS. Key Access Justifications, used with Cloud EKM keys, attaches a reason code to every Google request for your key, and your external key manager can refuse it. Bringing your own key To import key material, you create an import job with a wrapping method, such as RSA_OAEP_3072_SHA256_AES_256 or one of the newer post-quantum HPKE methods. The job gives you a public key, and it expires after three days. You wrap your key material with that public key and import it as a new version of a key. For a key that must only ever hold imported material, set importOnly, which means rotation is manual: import new material and make it primary. If you destroy an imported version, you can re-import the same material later. AWS KMS for comparison AWS KMS covers the same ground with a flatter, version-less model: Keys, not versions. A KMS key has a key ID (a UUID), an ARN, and optional aliases such as alias/card-data. There's no hierarchy above it beyond the account and Region.Rotation is hidden. Automatic rotation is off by default for customer managed keys. When enabled, it runs every 365 days by default, configurable from 90 to 2,560 days, and RotateKeyOnDemand allows up to 25 extra rotations per key. The key ID never changes; AWS keeps old material internally and picks it on decrypt.All keys in HSMs. Keys in the default key store live in HSMs validated at FIPS 140-3 Level 3. There's no software tier. CloudHSM key stores and external key stores (XKS) cover the dedicated and hold-your-own-key cases.A data-key API. GenerateDataKey returns a fresh DEK in plaintext along with a copy wrapped under your key, so step 1 and step 2 above are one call.Encryption context. The AWS equivalent of AAD is a map of key-value pairs. Unlike Cloud KMS AAD, it's recorded in CloudTrail and can be used in policy conditions such as kms:EncryptionContext:tenant.Three authorization layers. Every key has exactly one key policy, a resource policy that supports Deny and conditions. IAM policies work only if the key policy enables them, and grants add temporary, programmatic permissions on top.Whole-key lifecycle. DisableKey and ScheduleKeyDeletion act on the whole key. The deletion window is 7 to 30 days, and CancelKeyDeletion backs it out.Replication. Multi-Region keys share the same key ID and material across Regions, and ReplicateKey copies one to a new Region. You choose this at creation; a single-Region key can't be converted. Here's the same envelope flow on AWS: Java Map<String, String> context = Map.of("tenant", "acme"); // logged in CloudTrail, usable in policy try (KmsClient kms = KmsClient.create()) { GenerateDataKeyResponse dataKey = kms.generateDataKey(GenerateDataKeyRequest.builder() .keyId("alias/card-data") .keySpec(DataKeySpec.AES_256) .encryptionContext(context) .build()); byte[] dek = dataKey.plaintext().asByteArray(); // encrypt locally, then discard SdkBytes wrappedDek = dataKey.ciphertextBlob(); // store next to the payload byte[] unwrapped = kms.decrypt(DecryptRequest.builder() .keyId("alias/card-data") // optional for symmetric keys .ciphertextBlob(wrappedDek) .encryptionContext(context) .build()) .plaintext().asByteArray(); } KeyId is optional on Decrypt for symmetric keys because the ciphertext blob identifies the key, but AWS recommends passing it so you don't decrypt with a key you didn't expect. Cloud KMS requires the key name on every Decrypt. Side by side ConceptGoogle Cloud KMSAWS KMSKey identifierResource name projects/…/cryptoKeys/k, no aliasesKey ID, ARN, aliasesVersionsFirst-class CryptoKeyVersion with its own stateHidden behind the key IDProtectionSOFTWARE, HSM (FIPS 140-2 L3), single-tenant HSM, EXTERNAL, EXTERNAL_VPCAll HSM-backed (FIPS 140-3 L3), plus CloudHSM and external key storesMax plaintext per call64 KiB (8 KiB with AAD for HSM keys)4,096 bytesData keysGenerate locally (or with Tink)GenerateDataKeyAuthenticated dataAAD bytes, not loggedEncryption context map, logged and usable in policiesAuto-rotation1 day to 100 years90 to 2,560 days, off by default, plus 25 on-demandDisablePer versionWhole keyDestroyPer version, 30 days scheduled by default, restorableWhole key, 7–30 days, cancellableAccess controlIAM roles on the hierarchy, admin and usage splitKey policy (with Deny) + IAM + grantsUsage loggingData Access audit logs, off by defaultCloudTrail management eventsMulti-regionChoose a multi-region location for the key ringMulti-Region keys, ReplicateKeyBYOKImport job, expires in 3 daysGetParametersForImport, token valid 24 hoursThroughput quota (default)Token-based per region; about 60,000 software operations a minute10,000 to 100,000 symmetric operations a second, by Region What the AWS KMS vs. Google Cloud KMS encryption comparison means in practice: Cloud KMS is simpler to reason about for access. It's one IAM model with roles that separate managing keys from using them. AWS is more expressive: a key policy can Deny, and conditions on encryption context can tie a key's use to a tenant. That power comes with the risk of a key policy that locks out its own administrators.Versions are explicit on Google Cloud. You can disable or destroy just the key material that was compromised and leave the rest working. On AWS you disable or delete the whole key, or rotate and keep the old material.AWS logs every use by default. On Google Cloud, enable Data Access logs for Cloud KMS on day one, or you won't have a record of who decrypted what.Plan for quotas. AWS gives each account and Region a large shared pool of cryptographic operations. Cloud KMS's default is far smaller per project and region, so cache unwrapped DEKs for hot reads where your threat model allows, and request a quota increase before launch. Both quotas are adjustable.Both want envelope encryption. Neither service is built to encrypt bulk data directly. On Google Cloud, Tink makes the local half trivial. On AWS, the AWS Encryption SDK plays the same role. Conclusion Google Cloud KMS treats keys as resources in your project hierarchy: versions are visible, protection level is chosen per key, and IAM is the whole authorization story. That's easy to audit and gives fine-grained control over key material, as long as you turn on Data Access logs and size your quotas. AWS KMS hides versions and puts authorization in a per-key policy language that's more expressive, especially with encryption context, and its data-key API saves a step. These differences are worth considering when reviewing AWS KMS best practices alongside Google Cloud KMS key management. On both clouds, the envelope pattern is the common ground: generate a DEK, wrap it, store it next to the data, and let KMS guard only the key that guards everything else.