DevOps and CI/CD

When JDK 27 was released on 15 September 2026, most of the attention went to compact object headers and post-quantum TLS. JEP 523 received less coverage, partly because its description sounds like housekeeping: G1 becomes the default garbage collector in all environments. For most applications, that changes nothing. For a specific class of container, it changes the collector silently, with no warning in the logs and no failure at startup. This article covers what the fallback did, which workloads lose it, how to identify affected services, and how to run a comparison that produces usable numbers rather than noise. What the Serial GC Fallback Did G1 has been the nominal default since Java 9. What is less widely known is that HotSpot inspected the machine at startup and overrode that choice in constrained environments. The JVM selected Serial GC when either condition was true: ConditionThresholdAvailable processors1Physical memoryLess than 1,792 MB The reasoning was sound at the time. Serial GC is single-threaded with a very small native memory footprint. On a single-core machine, a concurrent collector spends time competing with application threads for the only available core, and the bookkeeping structures G1 maintains were expensive relative to a small heap. G1 has improved substantially since then. Its native memory footprint has been reduced across several releases, and JDK 26 delivered JEP 522, which removed synchronization overhead from G1's write barriers by introducing a second card table. JEP 523 concludes that work. The fallback is removed, and G1 is now selected in every environment regardless of CPU count or available memory. Serial GC remains fully supported and can still be selected explicitly: Shell java -XX:+UseSerialGC -jar application.jar Applications that already specify a collector, whether G1, ZGC, Parallel, or Shenandoah, are unaffected by this change. Which Workloads Are Affected The change only reaches services that never had an explicit collector flag and that run in a constrained environment. In practice, this describes a predictable set of deployments: Sidecar containers. Log shippers, metrics agents, and service mesh components typically receive a fraction of a CPU and a few hundred megabytes of memory. They are rarely tuned.Batch and queue workers. Short-lived jobs scheduled onto whatever capacity is available.Low-traffic internal services. Admin APIs, configuration services, and internal tooling that are deliberately given small pods.CI runners and development containers. Environments where JVM flags are seldom set because the workload is not production.Pods with fractional CPU limits. This case is easy to miss. A container configured with cpu: 500m may report a single available processor to the JVM, which placed it in Serial GC territory on earlier releases. These are the workloads least likely to appear in a performance test plan, which is precisely why the change deserves deliberate verification. Step 1: Identify Your Current Collector Before changing anything, determine which of your services are actually running Serial GC today. The GarbageCollectorMXBean interface exposes this directly: Java import java.lang.management.GarbageCollectorMXBean; import java.lang.management.ManagementFactory; public final class RuntimeFacts { public static void main(String[] args) { Runtime rt = Runtime.getRuntime(); System.out.printf("java version : %s%n", Runtime.version()); System.out.printf("available cpus : %d%n", rt.availableProcessors()); System.out.printf("max heap MB : %,d%n", rt.maxMemory() / (1024 * 1024)); for (GarbageCollectorMXBean gc : ManagementFactory.getGarbageCollectorMXBeans()) { System.out.printf("collector : %s%n", gc.getName()); } } } The collector names identify which implementation is active: CollectorReported MXBean namesSerial GCCopy, MarkSweepCompactG1G1 Young Generation, G1 Concurrent GCParallelPS Scavenge, PS MarkSweepZGCZGC Cycles, ZGC Pauses For a Spring Boot application, registering this as a startup listener puts the information into your existing log aggregation without any additional infrastructure: Java import org.springframework.boot.context.event.ApplicationReadyEvent; import org.springframework.context.event.EventListener; import org.springframework.stereotype.Component; import java.lang.management.GarbageCollectorMXBean; import java.lang.management.ManagementFactory; import java.util.stream.Collectors; @Component public class GcFactsLogger { @EventListener(ApplicationReadyEvent.class) public void logCollector() { Runtime rt = Runtime.getRuntime(); String collectors = ManagementFactory.getGarbageCollectorMXBeans() .stream() .map(GarbageCollectorMXBean::getName) .collect(Collectors.joining(", ")); System.out.printf( "JVM %s | cpus=%d | heap=%,dMB | gc=%s%n", Runtime.version(), rt.availableProcessors(), rt.maxMemory() / (1024 * 1024), collectors); } } Deploy this across your estate, then filter your logs for entries reporting Copy as a collector name. That result set defines the scope of your testing. Step 2: Understand the Native Memory Difference Discussions about garbage collector selection usually center on pause times. For this particular change, native memory consumption is the more consequential metric. G1 maintains bookkeeping structures that Serial GC does not require, including remembered sets, card tables, and per-region metadata. This memory is allocated outside the Java heap, which means it counts against a container memory limit without appearing in heap metrics. Measurements published after the JDK 27 release place G1's native memory overhead at approximately 43 MB, compared with roughly 1.7 MB for Serial GC. Ankur Mundra measured both collectors on Spring Boot 4.1 within real cgroup limits and reported these figures. The absolute difference is modest. The proportional impact depends entirely on container size: Container memory limitApproximate G1 overhead share1024 MB4%512 MB8%256 MB16% On a tightly sized container, this overhead is consumed before the application allocates its first object. Where memory limits were set with minimal headroom, the change alone can produce OOM kills after an upgrade that altered nothing else. Native Memory Tracking provides the per-category breakdown if you want to confirm the figure in your own environment: Shell java -XX:NativeMemoryTracking=summary \ -XX:+UnlockDiagnosticVMOptions \ -jar application.jar Then, against the running process: Shell jcmd <pid> VM.native_memory summary The GC section of the output reports committed native memory attributable to the collector. Step 3: Run a Controlled Comparison A meaningful comparison requires the smallest container size you actually deploy, representative load, and a run long enough to reach steady state. A thirty-second run measures startup behavior rather than collection behavior. Capture GC logs for both configurations: Shell # Pre-Java 27 behaviour java -XX:+UseSerialGC \ -Xlog:gc*:file=serial.log:time,uptime,level,tags \ -jar application.jar # Java 27 default java -Xlog:gc*:file=g1.log:time,uptime,level,tags \ -jar application.jar Three measurements matter, and the third is the one most often omitted: Total pause time. Sum the pause durations across the run rather than comparing individual collections. Serial GC performs fewer, longer pauses; G1 performs more, shorter ones. Only the aggregate is comparable.Tail latency. G1 runs concurrent phases. On a single-core container, those phases contend with request handling for the only available processor. This effect is invisible in a mean and clearly visible at p99.Resident set size under load. This is where the native memory difference becomes operationally relevant. Sample RSS throughout the run rather than reading it once at startup: Shell while true; do ps -o rss= -p "$PID" | awk '{printf "%s %d MB\n", strftime("%H:%M:%S"), $1/1024}' sleep 5 done If RSS approaches your container limit under G1 where it did not under Serial, you have found the problem before it found you. Step 4: Pin the Collector When G1 Loses If measurement shows that G1 performs worse on a constrained workload, the appropriate response is not to defer the JDK upgrade. It is to state the collector explicitly: Shell java -XX:+UseSerialGC -jar application.jar This is a supported configuration. Serial GC is neither deprecated nor scheduled for removal, and JEP 523 explicitly preserves it. In a Kubernetes deployment, express the choice where it will be reviewed: YAML spec: containers: - name: batch-worker env: - name: JAVA_TOOL_OPTIONS value: "-XX:+UseSerialGC" There is an argument for setting the flag explicitly even where G1 performs well. An explicit collector choice is self-documenting. A future engineer reading the manifest can see that a decision was made and tested, rather than inferring what the JVM selected at runtime. Conclusion JEP 523 is a reasonable change. G1 has genuinely closed the gap that justified the fallback, and removing a hidden conditional from the JVM's startup logic simplifies the mental model for everyone. The operational consideration is narrower: a set of small, rarely tested containers will change collectors on upgrade without any signal that it happened. Identifying those containers takes one log statement. Testing them takes one load run. The cost of skipping both is a latency regression or an OOM kill that appears unrelated to any change your team made. Java 27 is not a long-term support release, and many teams will remain on Java 25. That does not remove the relevance of this work. The same default will arrive with the next LTS, and evaluating it on a short-term release carries considerably less risk than discovering it mid-migration. Further Reading JEP 523: Make G1 the Default Garbage Collector in All EnvironmentsJEP 522: G1 GC: Improve Throughput by Reducing SynchronizationJDK 27 Release Notes

When you move messaging onto a third-party platform, you don't hand off the authorization problem. You split it across two systems. Most incidents live in the gap. A simulation harness, and a documentation audit of QuickBlox as a worked example. The Revoked User Who Kept Reading Say a clinic's app removes a contractor's access to a patient thread at 14:02. The app's database says so, the UI hides the thread, and the audit table has the row. The contractor's phone keeps receiving new messages until 15:00, and then keeps going. Nothing in the application is wrong. The membership removal went to the chat platform's API as a single best-effort call, and the API was having a bad ten minutes. The call failed, and the exception was swallowed. The contractor's session token is still valid, and the contractor's client keeps making requests that keep it valid. This is a hypothetical, but every piece is routine: a removal path with no retry, an admin API that can be unavailable, a session whose lifetime depends on activity. The useful question is how long the window lasts, and it becomes measurable once you stop treating the integration as an API call and start treating it as a consistency boundary. This article builds a small deterministic harness, seamlab.py, to measure that window along with seven other failures at the same boundary, including an AI agent acting after a human has taken over a conversation. It then audits one real platform, QuickBlox, against the same questions using its public documentation. QuickBlox is the worked example because its developer documentation is public and detailed enough to check. The harness measures a class of platforms and nothing about any vendor. The documentation audit reads documentation and tests nothing. I keep those two kinds of evidence apart throughout. The Flawed Assumption The usual framing of "build vs. buy" for chat or calling is about transport: delivery, ordering, presence, media. Delegating those is reasonable. But "who may read this thread, who may post, who may trigger an action" is not a transport concern, and delegating transport forks it: Your application holds the relationships: this clinician treats that patient, this contractor was removed.The platform holds a projection of them: dialog memberships, session tokens, message history. The projection is only as fresh as your last successful write to it, and the platform enforces the projection, not your database. So at least two places answer "is this allowed," and they can disagree. Each failure below is a case of that disagreement, or of one side believing a claim the other never checked. Two quantities make this concrete. Revocation window W: the time between the app committing a revocation and the last successful read by the revoked principal. If the platform checks live membership on every read, W is bounded by the time until the removal call lands. If it authorizes from the session token alone, W is bounded by the token's remaining life, and whether that life is fixed or extended by activity decides everything.Authority provenance: for any action in the system, whose authority is it exercising? For a human action, the answer is trivial. For an AI agent in a multi-party conversation, it usually isn't. Threat Model Six attacker positions, all realistic for a chat integration: #AttackerCapabilityAExternalKnows the webhook URLBAuthenticated low-privilege participantCan post messages into a thread the agent readsCSelf-registered platform userPlatform allows signup, no verified binding to your user modelDRevoked participantHolds a valid token from before revocationEReplayerHas a captured, validly signed event (logs, proxy, compromised sink)FWriter on the audit storeCan modify rows Trust boundaries: client to app, app to platform (admin API), platform to app (event delivery), agent to tools, and the audit store. ClassPreconditionMechanismTestControlForged eventAUnauthenticated event endpoint acceptedforgedSignature verificationIdentity confusionCSender resolved by mutable attribute (email)spoofed identityImmutable-ID binding tableReplayEDedupe TTL shorter than replay horizonreplayTimestamp window plus retention ≥ 2×windowDuplicate side effectNormal at-least-once deliveryNon-atomic check-then-actraceAtomic claim before actingStale accessDToken and membership outlive revocationrevocation sweepShort TTL, outbox retry, token killAgent confused deputyBInjected text drives a privileged toolA1–A4Parameter-bound approvalPost-handoff actionRace with human takeoverAgent call in flight at handoffA5–A6Epoch fencingScope overreachBAgent reads history beyond sessionA7Capability-scoped history Audit tampering (F) is covered separately below. The Harness seamlab.py is about 480 lines of stdlib Python on a logical clock, with no network and no randomness. It contains: A platform simulator: at-least-once event envelopes signed with HMAC, expiring tokens with scopes frozen at mint, optional idle (sliding) expiry, self-service registration, and an admin API that can be switched off. Whether reads check live membership, and whether a server-side token kill exists, are switches.Two app implementations: a naive one and a hardened one.An agent world with a tool proxy configurable per control, so the table shows which control closes which attack.A worst-case agent: it obeys every CALL tool key=value line in its context. That is deliberate. The tests target the authorization layer and say nothing about how robust any model is to injection. If your model resists injection, that's a bonus. These controls assume it doesn't. The duplicate-delivery test enumerates every distinct interleaving of two workers handling the same event, rather than sampling a few. Naive Design, Then the Attacks The naive ingress handler is a shape I'd expect to find in plenty of codebases: no authentication on the endpoint, dedupe with a TTL, check then act then record, sender resolved by email. def handle(self, env): t = self.seen.get(env["id"]); yield # check if t is not None and self.clock.t - t < self.DEDUPE_TTL: # 300s return "dup" uid = self.resolve(env["body"]["sender"]) # email match if uid: self.run_command(uid, env["body"]["text"]) # act yield self.seen[env["id"]] = self.clock.t; yield # record AttackNaiveHardenedForged unsigned eventsucceedsblockedValidly signed event, spoofed identitysucceedsblockedReplay at t+225 sblockedblockedReplay at t+3600 ssucceedsblockedDuplicate delivery: interleavings producing a double effect60 of 660 of 12 The spoofed-identity row is the interesting one. The event carries a correct platform signature. Nothing was forged. The attacker registered a platform account with email [email protected], unverified. The naive app lowercases and matches against its user table, so a perfectly authentic message from the wrong person runs with the clinician's authority. Signature verification can't help, because the signature is telling the truth about who sent it. The control is a binding table keyed on the platform's immutable user ID, written only by your backend when it provisions the platform user for an authenticated app user. The attacker's ID has no binding, so the event is rejected as unbound. The race result is worse than it looks. 60 of 66 distinct schedules double-fire. Only interleavings where the first worker finishes recording before the second starts checking are safe. This is the ordinary case for any retry storm. The fix is one atomic operation (SET NX EX, or an insert against a unique constraint) before any side effect. The hardened handler: def handle(self, env): if not hmac.compare_digest(env.get("sig", ""), sig_of(env)): return "bad_sig" if abs(self.clock.t - env["ts"]) > self.TOL: return "stale" ok = self.claim(env["id"]); yield # atomic set-if-absent, retention 2*TOL if not ok: return "dup" uid = self.bindings.get(env["body"]["sender"]) if uid is None: return "unbound" self.run_command(uid, env["body"]["text"]); yield return "ok" Mutation-Testing the Control The suite should be able to fail the hardened design; otherwise, the green column is decoration. So I mutated one parameter: dedupe retention set to half the timestamp tolerance. The replay at t+225 s then succeeds, because the event is still inside the accepted time window while its dedupe record has expired. The replay at t+3600 s is still blocked by the timestamp check. That yields an invariant worth writing into code review: dedupe retention must be at least twice the accepted timestamp skew (once for events slightly in the future, once for slightly in the past). Timestamp validation and dedupe are two halves of one control, and shortening either silently opens the gap. One caveat: claim-before-act gives at-most-once side effects if the process dies between claim and effect. For effects that must not be lost, the downstream operation needs its own idempotency key derived from the event ID. The harness does not model crash recovery. Revocation Window, Measured The harness drives a revoked user who keeps polling and refreshing, with the token's age at revocation sampled across its lifetime. The scenario that matters: the admin API is down when the revocation is issued and comes back 600 s later. In the first table, the platform checks live membership on every read: ConfigWorst W (s)Mean W (s)Token mints per user per hourNaive, TTL 3600, single attempt360018301Hardened, TTL 303016120Hardened, TTL 60603060Hardened, TTL 30030015212Hardened, TTL 9006014054Hardened, TTL 36006015561 "Hardened" means a transactional outbox retrying the removal with capped backoff, a broker that re-checks the app's authorization on every token mint, and a short TTL. Once TTL exceeds the outage, W is capped by the outage plus up to one backoff interval (601 s in this run). Below the outage length, TTL is the binding constraint. With a healthy admin API, every configuration, naive included, shows W = 0. The naive design only fails when the platform is degraded, which is when nobody is watching. When the Platform Doesn’t Check Membership, and Tokens Slide Two platform properties change the picture, and I only added the second after reading the vendor documentation described below, which is why the harness changed between drafts. If reads are authorized from the token alone, removal calls don't matter, and the TTL is the only built-in lever. If the token's expiry is idle-based, so each request pushes it out, then a revoked client that keeps making requests never lets the token expire, and the TTL stops being a bound at all. The broker's re-check at refresh never runs, because refresh never happens. The last control available is a server-side token kill, if the platform offers one. Results for a token-only platform (hardened TTL 60, naive TTL 3600; "never ends" means access persisted to the end of the simulated horizon): Expiry model, admin APINaiveHardened, no token killHardened + token killFixed TTL, healthy3600 s60 s0 sFixed TTL, 600 s outage3600 s60 s60 sIdle expiry, healthynever endsnever ends0 sIdle expiry, 600 s outagenever endsnever ends601 s Read the third row twice. With idle expiry and no kill, the short TTL and the broker re-check buy nothing against a client that keeps talking. The only controls that end access are live-membership enforcement on the platform or an invalidation you can trigger from the server. So the most valuable hour you can spend before integrating a platform is finding out which of those you have. At TTL 60, each active user generates 60 token mints per hour. Whether that is trivial depends on your user count and the platform's rate limits, which the harness does not model. The AI Participant: Confused Deputy, Not “Authorization for Bots” A multi-party conversation is a prompt authored by several principals with different privileges. The agent reading it holds one set of tool credentials. Any principal who can put text in the thread can attempt to spend authority they don't have. This is the classic confused deputy, with the deputy now parsing natural language. The harness has a dialog with a patient and a clinician, and three tools: get_my_record (low privilege), approve_refill (clinician only), and read_history. The proxy runs in three configurations: naive (the agent's standing credential decides everything), role-check (the call executes with the authority of whoever authored the triggering message), and hardened (role check plus parameter-bound approval, epoch fencing, and history clamping). AttackNaiveRole-check onlyHardenedA1 Patient's message steers agent into approve_refillsucceedsblockedblockedA2 Clinician's benign turn; patient's injected text sits in contextsucceedssucceedsblockedA3 Approval for rx42 reused to approve rx99succeedssucceedsblockedA4 One approval consumed twicesucceedssucceedsblockedA5 In-flight agent call executes after human takeoversucceedssucceedsblockedA6 New agent turn minted after takeoversucceedssucceedsblockedA7 Agent reads history outside session scopesucceedssucceedsblockedB1 Clinician approves refill through explicit approvalworksworksworksB2 Patient reads own record through agentworksworksworks The B rows make sure the hardened column isn't green merely because it denies everything. A2 is the finding that changed my plan. My pre-run design was authority as the intersection of the acting principals' rights. Role-check is that rule, and it stops the obvious attack (A1). It fails A2 completely. The clinician sends "please summarize the thread." The agent's capability is legitimately the clinician's. The patient's earlier message, still in context, asks for a refill approval, and the agent obliges, spending authority that was never granted for that purpose. Intersection scopes who the agent acts for. It says nothing about what that authority was meant to cover. The fix is to stop treating conversation turns as grants for privileged operations. A privileged call needs an approval bound to the exact call, issued out-of-band by an authenticated UI session of a privileged human: def _approval_ok(self, tool, args, ap): b = ap["body"] good = hmac.new(APP_KEY, json.dumps(b, sort_keys=True).encode(), hashlib.sha256).hexdigest() if not hmac.compare_digest(good, ap["mac"]): return False if (b["tool"], b["args"], b["epoch"]) != (tool, args, self.epoch): return False if self.clock.t > b["exp"] or b["nonce"] in self.used: return False if self.users.get(b["principal"]) != "clinician": return False self.used.add(b["nonce"]); return True The approval binds tool, arguments, epoch, expiry, and a single-use nonce. A3 fails because the arguments differ, and A4 because the nonce is spent. Injected text can still request the call. It cannot carry the authority to make it. The system doesn't try to detect injection. It makes sure injected text can't hold a capability. Fenced handoff (A5, A6). When a human takes over, the app increments a per-dialog epoch. Every capability carries the epoch it was minted under, and the tool proxy rejects stale ones. A5 is the race: a turn starts at t=0, the human takes over at t=5, and the agent's call arrives at t=8. Without the fence it executes. This is a fencing token, and it has a fencing token's limit: it only works if the resource performing the effect checks the epoch. If the agent can also post through a path that bypasses your proxy, such as the platform's own API with the agent's own credentials, the fence is decorative. The platform-side complement is removing the agent from the dialog at handoff, which is another admin call, subject to the same outage window measured above, and which not every dialog type allows (see below). History scope (A7). The capability carries the index where the current session begins, and read_history refuses anything earlier. The harness checks the proxy's enforcement. It does not check that the agent's context assembly respects the boundary, which is separate code you would need to test. The audit log, and what a hash chain does not do The hardened path records each agent tool call (allowed or denied) and each handoff in a hash-chained log with actor type, model version, tool, arguments, result, and principal: Clean chain, verified against an externally stored head hash: passes.Row modified without recomputing the chain: detected.Row modified and the chain recomputed from that row forward (an attacker with write access): verifies clean unless you compare against the externally stored head, in which case it is detected. The chain alone defends against accidental corruption and clumsy tampering. It does not defend against someone who can write the store and knows the scheme. What does is anchoring the head hash somewhere that an attacker can't write: a separate account, an append-only object store, a periodic signed checkpoint. If your threat model includes a malicious insider with database access, the anchor is the control, and the chain is the mechanism that makes anchoring cheap. On compliance: in a regulated setting, "who did what, when, under whose authority" is what this log makes answerable. The HIPAA Security Rule's audit-control standard is 45 CFR §164.312(b) (cited from memory; confirm before relying on it). A platform's compliance posture does not discharge your obligations, and a log design is one input to them, not compliance itself. A Documentation Audit: The Seam on Quickblox Everything above is simulation. This section asks the checklist questions of a real platform using its public developer documentation, retrieved on 4 October 2026. Nothing here was tested against a live QuickBlox account. Documentation can be incomplete or out of date, and where I could not find an answer, I say so instead of guessing. Sources are listed at the end. QuestionWhat the documentation saysConsequence for the seam1. Are server-to-app events authenticated?I found no first-party documentation of outbound server-to-server event webhooks. A competitor's comparison page asserts QuickBlox doesn't support webhooks. That is a competitor's claim and I haven't confirmed it.If there is no signed event feed, the forged/replay/race tests apply to whatever you build instead. The rule that survives is to never let your backend act on events reported by clients, because client-reported events are attacker-controlled. If you poll, poll with a server credential and treat the response as the source.2. Does a read check live membership, or only the token?Not stated in the pages I read.Unknown. This decides which row of the revocation tables applies. It is a two-minute sandbox test: remove an occupant, then read with that user's still-valid token.3. Token lifetimeBoth user and application session tokens expire 2 hours after the last REST request, and each REST response carries an expiry header. I found no configuration for it.This is idle expiry, which is the dangerous row of the table above. A client that keeps making REST calls keeps the session alive. I don't know whether live chat-connection activity also extends it, and I haven't verified an absolute cap.4. How is membership removed?Removal goes through the dialog update call. A dialog's owner can remove any occupant (the REST reference says admins can too), and an occupant can remove only themselves. The SDK and dashboard docs show removing other users working for group dialogs, not for private (one-to-one) or public ones. Deleting a user removes them from group dialogs but not from private dialogs' occupant lists.Private dialogs have no documented remove-another-user operation. If revocation matters, a clinician-patient conversation modeled as a private dialog can't be revoked by removal. You'd have to delete the dialog for everyone (which needs the owner and loses history). Model sensitive conversations as group dialogs with an app-controlled owner. This is a design decision to make before launch, not after.5. Immutable IDs, self-registrationUser creation with an application session is documented as acceptable for prototypes. For production the docs recommend creating users from your backend with the API key and disabling application-session user creation. Users carry an external_user_id field intended to link to an outside system, and login, email and external ID must be unique.The identity-confusion attack depends on that production setting. Bind your app user to the platform's own user ID, written only by your backend, and don't resolve senders by login or email. external_user_id is a convenient mapping field, but the binding should still be enforced on your side.6. Can the agent be a distinct principal?Dialogs have owner, admin, and occupant roles.Make the agent a separate user and an occupant, never an admin or owner. To fence it at handoff by removing it, the dialog must be a group dialog, per row 4. I did not verify whether the agent's session can be invalidated server-side.7. Can you reconcile after a missed event?Dialog and message REST endpoints exist, and the update and delete references show them taking an API key. I did not verify listing and filter semantics for drift detection.Plan the reconciler as a poll of dialogs and occupants compared with your database. Confirm pagination and filter support in a sandbox first. Three observations from the table, none of them about vendor quality. Every platform has choices like these, and every one is a place the seam can fail: The documented 2-hour idle expiry changes the revocation answer. In the harness's token-only, idle-expiry rows, short TTLs and broker re-checks buy nothing, and the only thing that ends access is membership enforcement or a server-side kill. Whether QuickBlox enforces membership at read time is the first thing to test, and whether you can invalidate a session from the server is the second.The private-dialog limitation is the most actionable finding. It is a design constraint you can only avoid by choosing group dialogs up front.The absence of a documented signed event feed moves work to you. The ingress hardening in this article becomes a design for whatever sync mechanism replaces it. On the vendor's compliance statements: QuickBlox's own pages state that it offers HIPAA-ready hosting with a signed BAA, and that it is SOC 2 certified. I haven't verified those claims, and a BAA does not make your application compliant. The seam problems in this article are exactly the part a BAA doesn't cover. Cost and trade-offs Ingress overhead. Hardened ingress costs about 8.8 µs per event against 2.0 µs for the naive handler, in-process, on one machine (Python 3.12). Indicative only. In production, the dominant cost is one atomic store write per event.Token churn. Short TTLs convert revocation latency into mint load: 60 mints per user-hour at TTL 60, 120 at TTL 30. And if expiry is idle-based, TTL tuning may not be available to you at all.Dependence on the admin API. Forced removal and token kill are only as available as that API. The outbox hides outages from the revoking caller. It doesn't shorten them.Approval friction. Every privileged action needs a human click bound to specific arguments. The agent can't autonomously perform privileged operations. That is the intended outcome, and it will frustrate some product teams. Mitigate it with a good approval surface: show the exact arguments, make approval one tap, and expire quickly.Complexity. The hardened design adds a broker, an outbox, an approval service, a per-dialog epoch, and an anchored log. For an internal tool with a few trusted users, it is probably more than you need. Limitations of this work The simulator is simple: one dialog, instantaneous admin success when up, no clock skew, no partial failures inside the platform, a single-node claim store.The worst-case agent models the threat, not any LLM.I did not implement or test a membership reconciler, although I recommend one.The QuickBlox audit is documentation-only. Docs may be incomplete, inconsistent (the owner-versus-admin removal rule is stated differently in different pages), or out of date. Every "unknown" in the table is a sandbox test I haven't run.Remaining exposure none of this addresses: injected text can still influence unprivileged tools (what the agent says, what it summarizes), exfiltration through in-scope reads, and approval fatigue.Nothing here shows that any real platform is vulnerable to any attack in the harness. Conclusion The integration boundary is a consistency boundary, and the interesting failures are state disagreements and authority crossing it unchecked. The revocation window gives you a number to argue about, instead of a feeling, and the number depends on three platform properties you can look up before writing code: whether reads check live membership, whether token expiry is fixed or idle-based, and whether you can remove a member or kill a session from the server. For the AI participant, the real question is whose authority a tool call exercises. Intersecting principals' rights is necessary but not sufficient. A privileged call needs an approval bound to its exact arguments, or conversation text will eventually carry authority it shouldn't. Sources (QuickBlox documentation, retrieved 4 October 2026): Key Concepts: https://docs.quickblox.com/docs/key-conceptsAuthentication (JS): https://docs.quickblox.com/sdks/js-authenticationUpdate Dialog: https://docs.quickblox.com/reference/update-dialogiOS Dialogs: https://docs.quickblox.com/sdks/ios-chat-dialogsDashboard, Chat: https://docs.quickblox.com/docs/chat-dashboardDashboard, Users: https://docs.quickblox.com/docs/users-dashboardCreate User: https://docs.quickblox.com/reference/create-userDelete Dialog: https://docs.quickblox.com/reference/delete-dialog

By Igboanugo David Ugochukwu CORE

A security review started after connection strings were visible through a Kubernetes management tool. The first request sounded simple: remove Secrets from the cluster and move the values to Azure Key Vault. That request focused on which system stored the values. The access review found a wider problem. Kubernetes stored a deployment service account's legacy token in a kubernetes.io/service-account-token Secret. The token had no expiry in its normal use, and RoleBindings allowed the same identity to operate in three application namespaces. The Role used resources: ["*"] across several API groups and allowed get, list, watch, create, update, patch and delete. The Role included the core API group, so the token could read every Secret in each of those namespaces. The review found no evidence of token misuse. It found an avoidable access path with a long lifetime and a larger scope than the pipeline needed. Moving application passwords to Key Vault would improve their source of truth and rotation. It would not revoke this token, narrow its RBAC permissions or remove the cross-namespace bindings. Those controls needed their own fix. Why the Kubernetes Secrets Existed The application images carried JSON configuration files for local development. Database, cache and message-broker endpoints pointed to localhost. Those defaults were useful on a developer machine and unusable inside the cluster. The shared Helm chart supplied the deployment values at runtime. It rendered an Opaque Secret from a secrets map, then loaded the Secret into the application container through envFrom. The reduced chart looked like this: YAML {{- if .Values.secrets } apiVersion: v1 kind: Secret metadata: name: {{ include "app.fullname" . }-secrets type: Opaque stringData: {{- range $key, $value := .Values.secrets } {{ $key }: {{ $value | quote } {{- end } {{- end } The Deployment then loaded every entry as an environment variable: YAML envFrom: - secretRef: name: {{ include "app.fullname" . }-secrets The application configuration placed environment variables after the file-based settings, so the runtime values won. That ordering made the Kubernetes Secret part of the configuration path. Removing it without replacing that path would leave the containers pointing at local services. Figure 1. The existing deployment used a Kubernetes Secret as the runtime override for local file defaults. The next question was, "Which components can retrieve the value, and how long does each form of access last?" Key Vault Does Not Always Remove the Kubernetes Secret The Azure Key Vault provider for the Secrets Store CSI Driver supports more than one consumption model. A pod can read mounted files directly from a CSI volume. This path can avoid creating a native Kubernetes Secret. The application still needs a configuration method that reads those files and responds safely when their contents rotate. The driver can also copy mounted values into a native Kubernetes Secret through the secretObjects field of a SecretProviderClass. Microsoft documents this as secret synchronisation. It is useful when an application already consumes secretKeyRef or envFrom, which was the case here. It also means that moving the source value to Key Vault does not remove the Secret object from Kubernetes. YAML apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: app-vault spec: provider: azure secretObjects: - secretName: app-runtime-secrets type: Opaque data: - objectName: database-password key: Database__Password The choice affects rotation too. Microsoft notes that an existing process does not receive updates when a container consumes a Kubernetes Secret through an environment variable. The team must restart the pod before the process receives the new value. Key Vault remains a sound source for secrets. The important point is narrower: the storage migration and the Kubernetes access review solve different problems. The Service Account Crossed Namespace Boundaries The next part of the review followed the identities that could read Secrets. Kubernetes allows a RoleBinding in one namespace to name a ServiceAccount from another namespace. That feature is useful for controlled cross-namespace work. It also means that checking only the ServiceAccount's home namespace can understate its reach. The review found a layout equivalent to this generalized example: YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pipeline-deployer namespace: environment-b rules: - apiGroups: ["", "apps", "batch", "autoscaling", "networking.k8s.io", "policy"] resources: ["*"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pipeline-deployer-from-a namespace: environment-b subjects: - kind: ServiceAccount name: pipeline-deployer namespace: environment-a roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: pipeline-deployer Two bindings of this form extended the identity from its home namespace into two more namespaces. The same bearer token could therefore authenticate once and use the permissions that each Role granted. Figure 2. Two cross-namespace bindings extended one long-lived credential into three namespaces. The identity was not a cluster administrator. The Role listed its API groups, and it did not include rbac.authorization.k8s.io. The token could not grant itself new RoleBindings through this Role. That limit mattered, but it did not make the existing scope acceptable. The core API group still covered Secrets, and the wildcard resource rule covered much more than the deployment needed. The following authorization check avoids printing any secret value: Shell identity="system:serviceaccount:environment-a:pipeline-deployer" for namespace in environment-a environment-b environment-c do printf '%s: ' "$namespace" kubectl auth can-i list secrets \ --namespace "$namespace" \ --as "$identity" done For the reviewed identity, the effective result was yes in all three namespaces. The Token Type Made the Scope Harder to Contain The service account had one manually created Secret of type kubernetes.io/service-account-token. Kubernetes describes this as a legacy mechanism that provides a long-lived credential. The current documentation recommends the TokenRequest API for short-lived tokens instead. The difference changes the failure window. A deployment job can request a token that expires shortly after the job finishes. If someone copies a static token into a CI/CD service connection, that credential can remain useful until somebody deletes or rotates it. An old export, agent workspace or copied configuration can preserve the same access during that period. The cluster version already supported projected service account tokens. Another environment-specific service account had no legacy token Secret, which confirmed that the static pattern was not present in every namespace. The First Fix Reduces the Token's Reach The quickest containment step is to remove the two cross-namespace RoleBindings. That immediately reduces the token's potential reach from three namespaces to one without changing the applications. Each environment should then have its own deployment identity, Role and RoleBinding. A pipeline for one namespace should not borrow the service account from another. This keeps the credential, audit trail and revocation boundary aligned with the environment it deploys. The desired shape is simple: Plain Text pipeline A uses a short-lived token for its Role in namespace A pipeline B uses a short-lived token for its Role in namespace B pipeline C uses a short-lived token for its Role in namespace C These deployments require no cross-namespace subject. Figure 3. The target design aligns one service account and its short-lived token with one namespace. A Short-Lived Token Replaces the Static Credential Kubernetes can issue a bounded token with the TokenRequest API. kubectl create token is a convenient way to exercise that API: Shell kubectl create token pipeline-deployer \ --namespace environment-a \ --duration 15m The command covers only one part of a CI/CD design. The runner still needs a trusted way to request the token. The pipeline should receive the credential at run time, the requester should limit its audience, and the token should expire soon after the deployment window. The service connection must stop storing the old bearer token before that change has value. After the new path succeeds, delete the legacy token Secret and confirm that the API rejects the old credential. Explicit RBAC Replaces Wildcard Access resources: ["*"] made future chart changes part of the role automatically. Kubernetes warns that wildcard rules can grant access to resource types that future API extensions add. A deployment role should name the resource types and verbs the release actually uses. The rendered manifests provide the initial resource list: Shell helm template app ./chart \ --namespace environment-a \ --values values.environment-a.yaml \ > rendered.yaml Inventory every apiVersion and kind in that output. Then compare the result with deployment calls in API audit logs or a dry run. The resulting Role may need access to Deployments, Services, ConfigMaps, Secrets, Jobs, Ingresses and autoscaling resources. It should not receive every resource merely because the chart might use one later. If a separate controller owns runtime Secrets, the deployment identity can lose Secret write access as well. If Helm still renders those Secrets, the role needs the exact verbs required by that release process. The policy must match the delivery design instead of hiding a required permission until the next deployment fails. The Review Found Other Paths to the Same Values The cluster token was the widest durable access path that the review found. Several pipeline variables also stored credentials as ordinary variables. Azure DevOps treats secret variables as protected resources. It encrypts their values at rest and masks exact values in logs. A task can still send a value elsewhere, and masking does not always hide partial strings. Marking the variables as secret still removes routine plaintext visibility from the library interface and logs. The remediation plan combined four related controls: Mark credential variables as secret and restrict which pipelines can use each variable group.Remove routine human permission to read Kubernetes Secrets through cluster tools.Rotate the database, cache and broker credentials that broader access paths exposed.Move non-sensitive settings out of the Helm secrets map and into a ConfigMap. Moving non-sensitive settings to a ConfigMap makes the review surface honest. Feature flags, log levels and endpoint names do not need to sit beside credentials merely because the original release populated only one values map. Validation Uses the Identity's Point of View The validation must test permissions from the identity's point of view. For each deployment identity, build an explicit matrix: Plain Text own namespace other namespaces read required objects yes no apply release objects yes no read application Secrets only if required no modify RBAC no no Then verify the old paths: the deleted legacy token fails authenticationeach identity returns no for Secret access outside its namespacethe pipeline can still deploy and roll backvariable-group values appear masked in the interface and logscluster users without an operational need cannot reveal Secret valuesrotated application credentials work and the previous values fail This sequence keeps the deployment usable while shrinking the access path in measurable steps. Secret Location and Secret Access Need Separate Reviews The visible Kubernetes Secret started the investigation, but deleting that object first would have left the long-lived deployment credential untouched. Using CSI secret synchronization could even have recreated the same type of object from a better source. The useful review order was identity, scope, lifetime, source and rotation. It exposed one token that crossed three namespace boundaries, wildcard permissions that exceeded the release surface, and pipeline values that lacked basic secrecy controls. Key Vault can improve storage and rotation. Short-lived credentials and narrow RBAC control who can reach the Kubernetes API. Environment-specific identities contain the effect of a leaked token. These controls work together, and none of them can substitute for the others. References Kubernetes documentation: SecretsKubernetes documentation: Service AccountsKubernetes documentation: Managing Service AccountsKubernetes documentation: RBAC good practicesMicrosoft Learn: Azure Key Vault provider configuration options for AKSMicrosoft Learn: Access Key Vault through the CSI Driver on AKSMicrosoft Learn: Secrets in Azure PipelinesMicrosoft Learn: Manage Azure Pipelines variable groups

The enterprise paradigm for artificial intelligence has fractured. While massive cloud-hosted large language models continue to dominate consumer applications, highly regulated sectors like defense, aerospace, and high-tier finance are actively blocking external API calls to public AI services due to strict data sovereignty mandates. The solution is migrating toward small language models (SLMs) localized at the tactical edge. Deploying an SLM into an air-gapped, "sovereign cloud in a box" environment requires absolute cryptographic certainty over the supply chain. A compromised base image running deep inside a disconnected network invalidates the entire security perimeter. The recent release of Docker Hardened Images (DHI) and the expansion of Google Distributed Cloud (GDC) air-gapped appliances give backend engineers the exact toolchain required to host models like Google's Gemma with zero-trust enforcement. This technical guide details an end-to-end framework for curating a zero-CVE container, packing a fine-tuned Gemma SLM, and pushing it to an isolated GDC cluster. The Target Infrastructure: GDC Air-Gapped Google Distributed Cloud air-gapped is an integrated hardware and software platform designed to operate physically disconnected from the internet. It allows enterprises to deploy container-based workloads and managed services like Vertex AI in completely isolated environments. The architecture supports rigorous compliance frameworks, including NIST, ISO 27001, and SOC II. Notably, GDC offers packaged generative AI solutions enabled by Gemma, providing local, conversational search capabilities on-premises. However, when deploying custom fine-tuned checkpoints or building the Java microservices that handle backend inference orchestration- a core focus of my daily work as an IEEE Senior Member and backend developer — we cannot rely on standard public container registries. The container must be hardened from the base layer up. Base Image Curation With Docker DHI To build our localized SLM container, we start with Docker Hardened Images. DHI provides minimal, secure, production-ready container images maintained by Docker. Rather than blindly pulling a generic Alpine or Debian base image and running a vulnerability scanner after the fact, DHI guarantees a hardened foundation. Using the docker dhi CLI, available in Docker Desktop 4.65 and later, we can inspect and mirror these hardened artifacts directly to our enterprise registries. Shell # Verify local DHI CLI installation docker dhi --help # Mirror a hardened Python or Java base image to your local org docker dhi mirror start --org enterprise-registry \ dhi/python,enterprise-registry/dhi-python By switching our Dockerfile FROM instruction to enterprise-registry/dhi-python, we structurally eliminate the attack surface before installing our inference engine dependencies. Generating the SLM Artifact and SBOM As I detailed in my Springer book chapter Introduction to Trustworthy AI, accountability in machine learning relies heavily on technical robustness and traceability. We cannot deploy a black-box container into an air-gapped network. We must generate a cryptographic Software Bill of Materials (SBOM) that catalogs the exact SLM weight hashes, the inference engine binaries, and the OS packages. The DHI CLI handles this transparently. Once you package your Gemma model weights into the hardened container, you can retrieve the exact attestations and SPDX SBOMs attached to the base images. Shell # Retrieve the SLSA provenance and SPDX SBOM for audit docker dhi attestation get dhi/python:3.11 sha256:<digest> -o provenance.json This JSON payload is ingested into the local CI/CD pipeline. The pipeline validates that no unauthorized dependencies were injected during the model packaging phase. If the cryptographic signature fails, the build halts immediately. Deploying to the Sovereign Edge With a cryptographically signed, DHI-backed container housing our Gemma SLM, we transition to the deployment phase on GDC air-gapped. Because GDC air-gapped operates a single cluster encompassing bare metal nodes without internet access, deployment relies on a physical handoff. The hardened container image, the validated SBOM, and the deployment manifests are loaded onto a portable storage device. Once transferred to the secure facility, the infrastructure operator pushes the image to the local, disconnected GDC container registry. GKE on GDC orchestrates the container workload. We apply Kubernetes network policies to isolate the SLM namespace, ensuring that the model can only be queried by authorized internal Java backend services. By enforcing kernel isolation at the container level and physical isolation at the hardware level, we construct a true defense-in-depth architecture. The Gemma model processes highly classified proprietary data while satisfying the strictest data sovereignty requirements. The transition from cloud-first to sovereign-edge AI marks a critical maturation in software engineering. Trustworthy intelligence demands predictable execution boundaries. By merging the zero-trust foundations of Docker Hardened Images with the physical isolation of Google Distributed Cloud, engineering teams can deliver state-of-the-art reasoning capabilities directly to the tactical edge without compromising a single byte of sensitive data.

GitHub's contribution calendar gives you an already iconic overview of your activity over the year. Each day has a square, and its color changes with the contribution level. And of course, those days and contributions can also be arranged into buildings, with a week becoming a small part of a city and busier weeks adding height to the skyline. I had some spare minutes and built Quarkus Contribution City around this idea, using Java and Quarkus to generate an SVG from GitHub's contribution data. The visual inspiration came from Pink Pixel's skyline, and you can see a version with my real data on my GitHub profile. A scheduled GitHub Action updates the image at night. For a Java developer, this is a small project you can follow from the API request through to the image. You can use the released action on your own profile, then change the drawing code locally once you have a feel for how the city is put together. For this article, I created a special DZone example that adds a small piece of character art above the buildings. Add the City to Your Profile Your GitHub profile README lives in a public repository with the same name as your account. For my account, that repository is myfear/myfear. Open your own profile repository, or create it with a README if you haven't used this GitHub feature before. Copy the project's complete publishing workflow into .github/workflows/contribution-city.yml on the default branch. This example checks out your profile repository and the action's v1 release line. It then generates the image and publishes it to an output branch. The workflow includes the repository write permission needed for that last step. The generation step uses the repository owner's username, so the same configuration can be copied between personal profile repositories. You can add the layout options to that step: - name: Generate contribution city id: city uses: ./.city-action with: github-token: ${{ secrets.CONTRIBUTION_TOKEN || secrets.GITHUB_TOKEN } username: ${{ github.repository_owner } weeks: '53' height: '14' theme: github-dark The local path refers to the action checkout made earlier in the example workflow. The built-in GITHUB_TOKEN is the starting point for reading your visible contribution data. An optional CONTRIBUTION_TOKEN secret can supply a personal token when additional access is needed. The calendar returned by GitHub depends on the token's permissions and your account's contribution visibility settings. After committing the workflow, open Actions in your profile repository, select publish-contribution-city, and choose Run workflow. A successful run creates contribution-city.svg on the output branch. Add this image reference to your profile README, replacing both occurrences of USERNAME:  The example also has a daily schedule for 03:17 UTC. Subsequent runs create a commit when the generated image changes. For this setup, the workflow and its run history can be found in your profile repository, while the Java implementation is maintained in the action's repository. Give Each Week a Building The city is arranged in time order, with the most recent week on the right. Every building is two character cells wide, so all weeks occupy the same horizontal space. Weekly contribution totals determine the height, and daily contribution levels provide the window colors. Month labels beneath the street help you find your place in the year. Height needs a little extra thought. A very busy week can have many times the contributions of a quiet one. With a linear scale, those quieter weeks would be compressed close to the street. The city uses a logarithmic scale, which gives them more room while preserving the order of weekly totals. For each week, the calculation can be expressed as: int height = weeklyTotal == 0 ? 1 : 1 + (int) Math.round( Math.log1p(weeklyTotal) / Math.log1p(maximumWeeklyTotal) * (maxHeight - 1)); With a maximum of 20 contributions and a height limit of 14, a week with three contributions gets seven window rows. The busiest week gets 14, and a week with zero contributions keeps one row. Quiet periods therefore retain their position in the calendar. An antenna marks the busiest displayed week; when totals are tied, the most recent week receives it. The two columns of windows are filled by cycling through the days of the week. This repeats the daily pattern in taller buildings, so window count is a visual texture. The weekly total controls height, while the window colors carry the daily contribution levels. Together, they give you a view of the pattern across your calendar. You can change weeks to 12 for a shorter view. The tallest building is then calculated from those recent weeks, which lets the selected period use the available height. Keep this in mind when comparing images: the same week can appear taller in a shorter view. The displayed contribution total still covers the full calendar returned by GitHub. Follow the Data Through Java The released project uses Quarkus with the Quarkus GitHub Action extension. That extension connects an @Action method to workflow inputs, outputs, and the job summary. Inside the method, the drawing process comes down to three steps: ContributionCalendar calendar = contributionClient.fetch( configuration.username(), inputs.getGitHubToken().orElseThrow(), context.getGithubGraphQLUrl()); Scene scene = cityBuilder.build(calendar, configuration.options()); String svg = svgRenderer.render(scene, configuration.options().theme()); The client reads the contribution calendar with one GraphQL request and converts the response into Java records for the calendar, weeks, and days. CityBuilder uses these records to place roofs, windows, and the street in a grid. SvgRenderer turns that grid into the final SVG. Changes to the skyline can therefore be made in plain Java classes, with the GitHub connection kept in the client. The SVG contains the shapes and text needed to display the city. Adjacent characters with the same color are grouped during rendering, and their horizontal positions are set explicitly to keep the columns aligned. Identical data and options produce identical file contents, which allows the publisher to compare images before creating a commit. The action sets up Java and JBang, builds the selected source revision, and runs the packaged application. Change the Layout Locally To try a change, install JDK 21 and JBang, with Git and Bash available. The repository includes a Maven wrapper. JBang is also used by the integration tests to run the packaged action through the same launcher used in GitHub Actions. git clone --branch v1.0.0 https://github.com/myfear/quarkus-contribution-city.git cd quarkus-contribution-city ./mvnw verify The tests use local test data and a local GraphQL server, so you can run them without GitHub credentials. They also generate SVG previews under target/previews. Open github-dark.svg, recent-12.svg, and empty.svg in a browser to compare a full calendar, a shorter period, and a calendar with zero contributions. Start with the height calculation in CityBuilder, then run the same command again and compare the previews. The saved data stays the same between runs, which makes the effect of your change easier to see. The project is available under the Apache License 2.0, and the v1.0.0 release gives you a fixed starting point. Give the City a DZone Landmark Version 1.0.0 exposes weeks, height, and the two built-in themes through workflow inputs. For a custom shape, you extend the Java code in your fork. The Scene record gives you a grid of cells to change before the SVG is rendered, and that is where we'll add the DZone lettering. Each cell has a character, a role such as roof or window, and the color information used by the renderer. A character mask is enough to describe a new shape: # marks a filled block, and spaces leave the background visible. By arranging those blocks, you can draw letters, an arch, or a stepped tower. For this example, the lettering is placed above the weekly buildings. It has its own fixed shape, while the buildings below continue to show the contribution calendar. The image uses a snapshot of my real account data, with 534 contributions returned by GitHub on October 5, 2026, when I wrote this article. A DZone-inspired palette and decorative lettering added to my contribution city. The weekly building heights and daily window patterns come from the same calendar used by the regular view. Add this class to the existing city package in your fork, at src/main/java/com/themainthread/contributioncity/city/DZoneLandmark.java: package com.themainthread.contributioncity.city; import java.util.ArrayList; import java.util.Collections; import java.util.List; import com.themainthread.contributioncity.render.Theme; public final class DZoneLandmark { public static final Theme THEME = new Theme("dzone-example", "#161b22", "#242c36", "#303947", "#f58a42", "#9ba9ba", "#edf2f7", List.of("#536070", "#86431f", "#b85b25", "#e57d36", "#ffb365")); private static final List<String> MASK = List.of( "#### ##### ### # # #####", "# # # # # ## # # ", "# # # # # ## # # ", "# # # # # # # # #### ", "# # # # # # ## # ", "# # # # # # ## # ", "#### ##### ### # # #####"); private DZoneLandmark() { } public static Scene addTo(Scene city) { int landmarkWidth = MASK.getFirst().length() * 2; int width = Math.max(city.width(), landmarkWidth); int left = (width - landmarkWidth) / 2; List<List<Cell>> rows = new ArrayList<>(); Cell block = new Cell('█', CellKind.ROOF, 0, -1); for (String line : MASK) { List<Cell> row = new ArrayList<>(Collections.nCopies(width, Cell.SKY)); for (int column = 0; column < line.length(); column++) { if (line.charAt(column) == '#') { row.set(left + column * 2, block); row.set(left + column * 2 + 1, block); } } rows.add(row); } rows.add(Collections.nCopies(width, Cell.SKY)); for (List<Cell> sourceRow : city.rows()) { List<Cell> row = new ArrayList<>(sourceRow); row.addAll(Collections.nCopies(width - city.width(), Cell.SKY)); rows.add(row); } return new Scene(city.username(), city.totalContributions(), rows); } } The theme defines the background, two building shades, roof, street, and label colors, followed by the five window colors. Those window colors follow GitHub's contribution levels from zero through four. Here, the roof color is also used for the lettering, so the decoration has a consistent orange color across all seven rows. Inside addTo, each marked position is expanded to two cells because the renderer's cells are taller than they are wide. The lettering is centered, a blank row is added beneath it, and the original city rows are copied below. Short calendars receive extra background space on the right to make room for the lettering. The dates, counts, and order of the weekly buildings are preserved. To use the decoration in your fork, import DZoneLandmark from the city package in ContributionCityAction. Replace the existing scene-building and rendering lines with: Scene scene = cityBuilder.build(calendar, configuration.options()); Scene decorated = DZoneLandmark.addTo(scene); String svg = svgRenderer.render(decorated, DZoneLandmark.THEME); This selects the example's palette in Java. The published @v1 action continues to accept github-dark and mono; your fork contains the additional drawing code. For a local preview, add the following after the existing Files.writeString call in PreviewTest.write, with the same import for DZoneLandmark: Scene decorated = DZoneLandmark.addTo(builder.build(calendar, options)); Files.writeString(directory.resolve(name.replace(".svg", "-dzone.svg")), renderer.render(decorated, DZoneLandmark.THEME)); Run ./mvnw verify again and open target/previews/github-dark-dzone.svg. The build also writes the shorter and empty versions with the -dzone.svg suffix, so you can check how the decoration fits several calendars. When you're ready to use your fork on GitHub, update the second checkout in the publishing workflow to your fork's repository and a commit containing your changes. The generation step then runs the customized action from that checkout. Try It With Your Own Contributions Try Quarkus Contribution City with your own GitHub account and add the generated SVG to your profile. Start with the released action, then fork the project and change the character mask to give your city a landmark of its own. You could draw a stepped tower from your hometown or add your team's initials above the street, using the DZone version as a starting point. Share your version in a GitHub issue, or open a pull request if you'd like to contribute a new shape.

Every Terraform team I’ve worked with has drift detection. A scheduled terraform plan -detailed-exitcode, a Terraform Cloud health assessment, a Spacelift or env0 monitor. Exit code 2 fires, a Slack message lands, and someone opens a ticket. Then the ticket sits. Drift found in minutes gets fixed in days, because the fix needs a human to read the diff, work out why the resource changed, decide whether to codify or revert, write the HCL, and push a PR. Detection is automated. Remediation is still artisanal. The Gap The tempting fix is to auto-apply on drift. Almost nobody does that in production, and they’re right not to. A blind apply on a security group that someone widened during an incident re-breaks the thing they just fixed. Drift means reality and code disagree, and “code wins” is not always the right answer. The middle path is an agent that reads the drift, drafts a remediation as a pull request with its reasoning and its prediction of the post-merge plan, and then stops. The PR is the gate. The real plan runs as a required check on that PR, the same one every human PR gets, and the agent is never in the apply path. I’ve been running this against production stacks in two regions. It turns a days-long ticket into a short review. I’ve published the workflows as a reference implementation in this repository. The linked repo keeps the same shape (detect → remediate → PR gate → apply) with the minimum needed to run. Firefly and env0 sell versions of this idea, and tfdrift’s remediate command and autoai-driftguard are open-source ones. The idea is the easy part. What I want to write down is the set of design decisions that make it safe to leave running, because that’s where the failures live. Three properties make the shape defensible. The agent’s blast radius is a branch: it can’t touch state, cloud resources, or the merge button, so the worst case is a PR a reviewer closes (for an honest agent, at least; more on that later). The existing change process is reused, not bypassed: CODEOWNERS, required reviews, and policy checks run on the agent’s PR exactly as they do on a human’s. And the judgment call stays with a person, but it arrives pre-researched. Two Kinds of Drift When plan says a resource differs from state, there are two directions to go: DirectionMeaningActionRevertThe out-of-band change was wrong; code is the truthNo HCL change; apply the existing configCodifyThe out-of-band change was right; code is staleUpdate HCL to match reality, so apply produces no diff A detector can’t tell them apart. A t3.medium that is now a t3.large is either someone’s forgotten experiment (revert) or last week’s capacity fix that never made it into code (codify). The evidence lives outside Terraform: CloudTrail, tickets, tags, Slack threads. Gathering that evidence and writing down a hypothesis a human can confirm or reject in a minute is exactly what an LLM is good at and a cron job is not. The Workflow Four stages. Only the second one is new. Stage 1: Detection A scheduled workflow runs a plan and captures a machine-readable output. YAML # .github/workflows/drift-detect.yml (abridged) on: schedule: [{ cron: "0 */6 * * *" }] jobs: plan: permissions: { id-token: write, contents: read, actions: write, issues: write } strategy: fail-fast: false matrix: include: # CloudTrail LookupEvents is regional; the agent runs where the resource lives - { stack: network/prod-use1, region: us-east-1 } - { stack: network/prod-use2, region: us-east-2 } steps: - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: ${{ vars.DRIFT_READONLY_ROLE } # reads + lock-table write; plan holds the lock aws-region: ${{ matrix.region } - name: Plan (refresh-only) id: plan run: | set +e terraform plan -refresh-only -detailed-exitcode -lock-timeout=5m \ -out=drift.tfplan -input=false code=$?; set -e echo "code=$code" >> "$GITHUB_OUTPUT" [ "$code" -eq 1 ] && exit 1 # plan error: fail loudly terraform show -json drift.tfplan > drift.json - name: Count drifted resources # refresh-only plans report under resource_drift id: count run: | n=$(python3 -c "import json; d=json.load(open('drift.json')); print(sum(1 for r in d.get('resource_drift', []) if r['change']['actions'] != ['no-op']))") echo "n=$n" >> "$GITHUB_OUTPUT" - name: Drift-storm guard # >20 drifted resources is a provider/state problem, not drift if: steps.plan.outputs.code == '2' && steps.count.outputs.n > 20 run: gh issue create --title "[drift] storm in ${{ matrix.stack }" --body "Paging a human." - name: Dispatch remediation if: steps.plan.outputs.code == '2' && steps.count.outputs.n <= 20 run: gh workflow run drift-remediate.yml -f stack=${{ matrix.stack } -f region=${{ matrix.region } -f run_id=${{ github.run_id } Three details do the work. -refresh-only separates real drift from HCL that merged but hasn’t been applied yet. The role is read-only in shape but needs lock-table writes because plan holds a state lock, which is also why -lock-timeout=5m is there: a scheduled plan that collides with an apply should wait, not fail. And the storm guard exists because a provider upgrade looks like drift on every resource; past 20 in one run, the workflow pages a human instead of dispatching the agent. Stage 2: The Remediation Agent The agent has three capabilities: read the plan JSON, read evidence, open a PR. No cloud writes, no terraform apply. YAML # .github/workflows/drift-remediate.yml (abridged) on: workflow_dispatch: inputs: { stack: { required: true, type: string }, region: { required: true, type: string }, run_id: { required: true, type: string } } jobs: remediate: permissions: { contents: write, pull-requests: write, id-token: write, actions: read } steps: - uses: actions/download-artifact@v4 # real workflows sanitize '/' -> '-' in the stack name so this matches the upload step with: { name: drift-${{ inputs.stack }, run-id: ${{ inputs.run_id }, github-token: ${{ secrets.GITHUB_TOKEN } } - uses: aws-actions/configure-aws-credentials@v4 with: { role-to-assume: ${{ vars.DRIFT_READONLY_ROLE }, aws-region: ${{ inputs.region } } - name: Analyze drift # model + AWS read access; no GitHub write token here run: python agent/remediate.py analyze --stack "${{ inputs.stack }" --region "${{ inputs.region }" --plan drift.json --out proposal/ env: { ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }, ANTHROPIC_MODEL: ${{ vars.ANTHROPIC_MODEL } } - name: Open PR # bot token only; see below for why not GITHUB_TOKEN run: python agent/remediate.py propose --stack "${{ inputs.stack }" --proposal proposal/ env: { GH_TOKEN: ${{ secrets.DRIFT_BOT_TOKEN } } The token in that last step matters more than it looks. GitHub limits workflow runs for events created with GITHUB_TOKEN, so a PR opened with it gets a pull_request run parked in an approval-required state: no jobs, no checks on the PR, no error. If the plan and policy checks are required, the PR can’t merge until someone with write access clicks “Approve workflows to run.” The tool surface is small, and every tool is read-only except the last: Python TOOLS = [ read_plan_json, # resource_drift entries from drift.json read_hcl, # repo files, capped per file read_tags, # tags on the drifted ARN cloudtrail_lookup, # LookupEvents in the stack's region, last 14 days describe_resource, # aws ec2/rds/iam describe-* search_issues, # GitHub issues/PRs mentioning the resource ID propose_fix, # the ONLY write: stages a branch + PR body under proposal/ ] Two limits on the CloudTrail tool. LookupEvents returns management events only, for one region, for the last 90 days, 50 per page, at two calls a second. That covers “who changed this security group” and not “who read this object”; for older questions, point the tool at CloudTrail Lake or the S3 bucket storing logs. Notice what’s missing: no apply, no cloud modify, no merge. A “revert” decision changes no HCL, because the HCL is already right. Instead, the agent commits a decision record, .drift-decisions/<address>-<fingerprint>.md, with the direction it chose, the evidence, and the plan it expects after merge. That gives the PR a real diff (GitHub refuses PRs with no commits) and puts the decision in git history next to the resource. A revert PR’s plan check exits 2 on purpose, and the gate compares the plan to the record’s claim instead of failing on the exit code. The system prompt sets the contract: You are a Terraform drift analyst. For each drifted resource you will (1) determine whether the out-of-band change should be codified or reverted, (2) cite the evidence that led you there, (3) produce the minimal HCL diff for the codify case, and (4) rate the strength of the evidence as strong, partial, or none. You cannot apply changes. If evidence is insufficient to choose a direction, say so and draft the codify variant anyway so a reviewer can see what accepting the change would look like. That last sentence matters. An agent that won’t act under uncertainty is the same as no agent. Drafting the codify variant costs cents against an engineer’s hour, and it’s reversible. Stage 3: The PR as the Gate Here is the agent’s output on a real drift, a security group that grew an ingress rule outside Terraform: [drift] network/prod-use1: aws_security_group.app_alb drifted out-of-band Recommendation: Codify (evidence: strong) Drift: ingress tcp/8443 from 10.40.0.0/16 not present in state. Evidence: CloudTrail AuthorizeSecurityGroupIngress at 2026-09-08T03:12Z by role IncidentResponder-oncall. Matches issue #431 (“ALB health checks failing from new monitoring subnet”), closed the same day. The CIDR is aws_subnet.monitoring, added in PR #412, which never updated the SG. Change: adds the ingress block to modules/alb/sg.tf, parameterized on var.monitoring_cidr. Expected post-merge plan: no changes. The plan check on this PR confirms or refutes that. If you disagree: close this PR; the next scheduled apply removes the rule. #431 suggests that re-breaks monitoring. The reviewer’s job goes from “investigate” to “verify.” Every line is checkable in under a minute, and the plan check proves the fix converges. The gate is the same set of required checks human PRs pass, plus one step that judges the agent’s claim: YAML - name: Plan on PR run: | set +e; terraform plan -detailed-exitcode -out=pr.tfplan -input=false echo "code=$?" >> "$GITHUB_OUTPUT"; set -e terraform show -json pr.tfplan > pr.json - name: Judge only the claimed resource # codify: the claimed resource is no-op/absent in pr.json # revert: the claimed fingerprint still appears under resource_drift run: python agent/gate.py --plan pr.json - name: Policy check run: conftest test pr.json -p policy/ # Branch protection: 1 approval + Code Owners review. The bot can't approve # its own PR, and "Allow GitHub Actions reviews to count" stays off. Branch protection does the real safety work. The bot can’t approve its own PR, and because its PRs touch platform-owned paths, the platform team’s CODEOWNERS entry has to. Stage 4: Apply (Unchanged) On merge, your existing apply pipeline runs. If the agent was wrong and the reviewer missed it, you have git history, a plan artifact, and a normal revert PR. Same failure mode as any bad merge, which is the point. Guardrails One PR per resource. A stack with six drifted resources gets six PRs, each with its own codify/revert decision. One PR for all six is how a reviewer approves five good changes and one bad one. That rule was the first thing I learned from one of the projects. I staged three drifts at once: a tag added to an S3 bucket (codify), a security-group rule opened to the world (revert), and a subnet deleted out from under state (needs-human). My first gate deadlocked on the first two. The codify check wanted a fully clean plan, the revert check wanted exactly one drifted resource, and each PR’s plan showed the other PR’s drift. Neither could go green. The fix is the “judge only the claimed resource” step above. With it, both PRs passed while the deleted subnet stayed open as an issue. Per-resource independence has to live in the gate, not just in the agent. Idempotent PRs. Before opening a PR, the agent looks for an open one with the same resource address and drift fingerprint, a hash of the before/after attributes with keys sorted so ordering noise doesn’t create a new fingerprint. If the resource drifts again differently, the old PR is stale: the agent closes it with a pointer to the new one. Never auto-remediate delete drift. If a resource exists in state and not in reality, the codify path is deleting HCL, which is the easiest change to get catastrophically wrong. The agent files its evidence as a drift/needs-human issue and stops. Deny-list sensitive types. IAM policies, KMS key policies, and anything under module.security. are evidence-only; the agent files an issue and a human opens the PR. It’s a deny-list on purpose. The human gate already makes the default safe; the list is about blast radius on the day the reviewer is tired, and IAM is where a plausible one-line change grants access to everything. An allow-list would be safer on paper and would spend its first month growing one type at a time while drift queued up. Cap the context. The agent gets the drifted resource’s resource_drift entry, the HCL file that declares it, and evidence for that ARN. Not the repo. Less context means fewer chances to “improve” adjacent code. Log the reasoning. Every PR body carries the evidence trail. When a codify decision turns out wrong three weeks later, you want to know whether the agent misread CloudTrail or CloudTrail was silent. What This Doesn’t Solve It doesn’t prevent drift. If people can write to prod from the console, they will; this makes the consequence cheap, not the cause. It doesn’t handle state corruption or provider upgrades that look like drift on every resource. And it doesn’t remove the reviewer. If your team approves agent PRs without reading them, you’ve built auto-apply with extra steps. Conclusion The industry spent the last few years making drift visible. The next step isn’t making it disappear on its own. It’s making the fix arrive already written, already justified, and already inside the review process you trust. An agent that can only open pull requests is an agent whose worst day is a closed PR. Nothing here is specific to GitHub Actions or AWS. Read-only agent, PR as the only write, existing gates unchanged: that ports to GitLab, Azure DevOps, or any Terraform platform with a VCS-driven apply.

AI has made producing code dramatically cheaper. Understanding that code has not become cheaper. A developer can generate a service wrapper, test suite, or 500-line pull request in minutes. A reviewer still has to understand what every important line does, what assumptions it makes, and how it behaves when the system changes. That gap creates a new form of technical liability: comprehension debt. It accumulates when teams can produce code faster than they can explain, operate, and safely change it. Comprehension debt shows up when a team accepts code before it has really taken ownership of the reasoning behind it. The code is not always messy. Sometimes it is clean. Sometimes it passes every test. That is what makes the risk easy to miss. The team ships the change, but it does not understand why the implementation works, where it is fragile, or what might happen when the surrounding system changes. That is why this moment matters. Traditional technical debt often accumulated because software was difficult and expensive to produce, so teams cut corners to keep delivery moving. AI changes that economic model. Code production can now outrun the team’s ability to inspect, explain, and safely evolve what was produced. The limiting factor is no longer only implementation capacity; it is human comprehension. If engineering organizations want to benefit from AI without weakening their codebases over time, they need to update their definition of done. A pull request that passes CI/CD is no longer enough. Production-ready AI-assisted code also needs traceability, risk classification, reviewer accountability, and evidence that the team understands the logic it is accepting. When AI-Generated Code Gets the Details Wrong Sometimes, we have specific assumptions or constraints in mind that we may not explicitly encode in the prompt because, as humans, we naturally consider them while writing the code. An AI, however, would not necessarily make the same assumptions. For instance, assume we are building a new API that needs to perform request validation, request authorization, and then execute business logic. If you tell an AI agent to build this, it might implement the API with request validation first, followed by request authorization, and then the rest of the business logic in the service. It might also write many unit and integration tests to ensure the request validation, request authorization, and business logic work correctly. However, from a security standpoint, you might want the request authorization to happen before the rest of the request validation, followed by the business logic. If this instruction is not explicitly given to the AI, it may implement the code in the wrong order, potentially performing request validation first and exposing information to a caller who is not authorized to view it. With the volume of code AI generates, it can be hard to find this kind of bug. However, when you implement code line by line without AI and dig into the implementation details, it may be easier to think through these scenarios. This is one problem with AI-generated code: it may be functionally correct and well tested, yet still miss assumptions a human developer would naturally consider during implementation. Product requirement documents generally describe high-level business requirements and may not specify every minor corner case. As a developer implements the logic, their understanding of the requirements evolves, and writing the code forces them to think through corner cases and decide how the system should behave. With AI-generated code, this iterative process can be missed if the requirements are simply handed to an AI agent as a prompt. The Hidden Cost of Synthetic Velocity Generative AI can produce a lot of plausible code almost instantly. The output often has clean formatting, sensible names, and just enough polish to feel safe. But polish is not understanding. A generated function can look obvious on the page while carrying assumptions no one on the team has checked. The bottleneck then moves from creation to comprehension. A developer can submit a 500-line AI-assisted pull request in minutes, but the reviewer still has to understand the full behavior, trace the edge cases, and decide whether the implementation fits the architecture. Reviewing is harder than generating because it requires reconstructing intent after the fact. When this happens repeatedly, reviewers adapt. With the massive amount of code being generated, they skim more. They lean harder on green tests. They assume the author has already done the deeper reasoning. The team may still be moving quickly, but the review process has quietly shifted from “Do we understand this?” to “Does anything obviously look broken?” Months later, the same code could cause problems in production, but no one can explain why the original implementation was structured that way. What looked like a fast delivery win becomes a forensic debugging exercise because the team accepted code faster than it absorbed the reasoning behind it. That is the real cost of synthetic velocity. The time saved during generation often comes back later in outages, migrations, and security reviews. Sprint metrics can make the codebase look healthy while the people responsible for operating it are quietly losing their map. Introducing the TTT Framework I have started thinking about this less as a tooling problem and more as an ownership problem. Banning AI coding tools is not the answer; developers will keep using tools that help them move faster. The harder question is whether the team still owns the reasoning behind what it ships. The Trace, Tag, and Time-box framework is a simple way to make that ownership visible. Trace what part of the code was AI-generated. Tag the change with the semantic risk. Time-box the work required to understand, test, and refactor it before it turns into institutional mystery. Reverse Walkthroughs are the mechanism for Yellow and Red changes, not a fourth pillar. The framework is intentionally boring. That is the point: it depends less on a specific tool than on habits a team can actually repeat. 1. The Trace Mandate The first rule is simple: teams should be able to identify AI-assisted code later. If a function, migration, test suite, or configuration block was generated or heavily shaped by an AI tool, that fact should not disappear once the pull request is merged. A lightweight trace can live in the pull request description, a commit message, or an internal annotation standard. This trace is not about blaming a tool or treating generated code as automatically suspicious. It is about preserving context. When a strange regular expression, unusual data transformation, or non-obvious retry strategy causes trouble months later, the team should be able to see where the logic came from and what the original intent was. That context can dramatically shorten future debugging and refactoring work. 2. Semantic Risk Tagging Not all AI-generated code carries the same level of risk. A generated CSS utility class and a generated authentication flow should not go through the same review process. Teams need a simple risk language that helps reviewers decide where to spend their attention. A useful tagging model divides AI-assisted changes into three categories: Green, Yellow, and Red. The point is not to create bureaucracy. The point is to make risk visible before reviewers are forced to infer it from the code alone. Green (Commodity): Low-risk generated code such as boilerplate, basic configuration, repetitive type definitions, simple test scaffolding, documentation drafts, or straightforward UI styling. These changes still need review, but they usually do not require deep architectural debate.Yellow (Algorithmic): Medium-risk generated code that changes business logic, data transformations, validation rules, error handling, concurrency behavior, or mathematical operations. These changes require a human reviewer to trace the logic carefully and confirm that the implementation matches the intended behavior.Red (Architectural): High-risk generated code that affects service boundaries, state management, database schemas, security-sensitive logic, permission models, API contracts, or large-scale system behavior. These changes should require senior review and, in some cases, architectural approval before generation is even used. 3. Time-Boxed Comprehension Work The third rule is to give comprehension real space on the engineering calendar. AI-assisted development creates an awkward asymmetry: the author may have produced the code quickly, but the team still has to understand it well enough to operate it later. If that work is not scheduled, it usually gets squeezed into a rushed review or postponed until an incident forces the issue. For Green changes, the time commitment might be minimal: a normal review, a quick note in the pull request, and standard tests. For Yellow and Red changes, teams should allocate defined review and refactoring time before the merge. That time can be used to simplify confusing logic, add integration tests, clarify naming, write a short design note, or pair through the generated implementation with another engineer. One practical way to spend that time is a “Reverse Walkthrough.” Instead of saying, “The tests pass,” the author must explain how the generated code works, why the chosen structure is appropriate, what edge cases were considered, and what failure modes remain. If the author cannot explain the implementation in plain language, the team should not merge it. This rule keeps ownership attached to the human operator, not the model that produced the first draft. A useful Reverse Walkthrough asks the author to answer practical questions before Yellow or Red code merges: What problem does this code solve? What assumptions does it make? What happens when those assumptions fail? Why was this implementation chosen? What alternatives were rejected? What happens under concurrency, load, or failure? Which parts were generated, and which parts were manually modified? Using AI to Make Code Reviews More Effective AI can also be a powerful tool for helping developers review AI-generated code. Instead of relying on the AI to simply perform an automated code review, reviewers can have a conversation with the AI about the changes in a pull request. They can ask it to explain how the code handles specific corner cases, identify scenarios that may not be covered, or verify whether the implementation actually satisfies the requirements of the change. An AI agent can also provide a high-level overview of the pull request, explain how the different files and classes fit together, and describe the responsibilities of the major components. This can significantly reduce the time a reviewer spends trying to understand a large code change. Managing the Velocity Shift The biggest mistake leaders can make is treating AI-assisted development as a pure throughput problem. If success is measured only by tickets closed, lines generated, or pull requests merged, teams will optimize for short-term output while quietly weakening long-term maintainability. A healthier metric combines delivery speed with operational confidence. Can the team explain the code it ships? Can new engineers understand the generated modules without relying on one person? Can the system be debugged under pressure? Can reviewers identify which parts of a pull request need deeper inspection? These questions matter as much as raw velocity. Comprehension Debt Indicators I would not treat these as established industry metrics. They are better understood as proposed operational signals: lightweight measures that help a team notice when AI-assisted development is increasing output faster than shared understanding. Reverse Walkthrough failure rate: The percentage of Yellow or Red AI-assisted pull requests where the author cannot adequately explain the implementation, assumptions, or failure modes.AI-code review time: The median reviewer time for AI-assisted pull requests compared with conventional pull requests of similar size and risk.Comprehension recovery time: The time spent after merge simplifying, documenting, testing, or restructuring AI-generated code so the team can maintain it confidently. None of these metrics should be used to shame individual developers. Used well, they give engineering leaders an early warning system. If the AI-assisted PR ratio rises while walkthrough failures, review time, and recovery time also rise, the team is not just moving faster. It is accumulating knowledge gaps that will show up later in incidents, migrations, or onboarding. Start small. Pick one active repository and apply the TTT framework to the next AI-assisted pull request. Ask the author to trace the generated code, tag each generated section by risk, and reserve a defined comprehension window for anything Yellow or Red. Require a Reverse Walkthrough before merge, or soon after merge if the work is intentionally staged behind a feature flag. The exercise will show quickly where the team’s review process is solid and where comprehension debt is already starting to collect. AI can absolutely make engineering teams faster. But speed that destroys shared understanding is borrowed time. The code is not necessarily bad. The problem is that six weeks later, the person who approved it may have no idea why it works. Comprehension debt is manageable, but only if teams name it, measure it with practical operational signals, and build habits that keep humans in control of the systems they are responsible for maintaining.

Without writing code, Salesforce makes it simple to modify a live system. It only takes a few minutes to create a new field, a validation rule, an updated flow, or a modified permission set. Many teams use the platform because of its speed, which also makes it challenging to securely modify many Salesforce organizations after a few years. On a previous project, I oversaw Copado deployment pipelines, including version control, conflict resolution, and automated regression testing, for a Salesforce environment. This article explains the methods that make declarative modifications as traceable as code changes, along with why they matter more as automation increases. Declarative Changes Are Still Changes The cultural transformation is the most significant. Logic is a flow. An access decision is a sharing regulation. A business rule is a validation rule. Users perceive any errors in production in the same manner as they would a defect in handwritten code. Every modification has a ticket, a branch, a reviewer, and a log of its progress when configuration is treated as code. In order for everybody to know what changed, why, and who approved it, we actually tracked work items in JIRA and connected each one to a user story in Copado. Version Control for Metadata Salesforce metadata, including objects, fields, flows, layouts, profiles, and permission sets, can be retrieved as files and stored in a Git repository. Copado manages that relationship. Each user story captures the metadata components that changed, commits them to a feature branch, and promotes them through environments such as development, integration testing, user acceptance testing, and production. Two habits made version control useful rather than ceremonial: Commit only what changed. Large commits that include whole profiles or layouts make reviews hard and conflicts likely. Selecting the specific components for a story keeps the history readable.Write the reason in the story. A commit that says "updated flow" helps nobody six months later. A story that says which business rule changed, and for whom, makes later investigations far shorter. Resolving Metadata Conflicts Conflicts are common when several people work in the same org. Two administrators may edit the same page layout, or two stories may change the same flow. In a Salesforce context, conflicts often hide inside large XML files such as profiles, where a small change in one place can overwrite a different change elsewhere. My approach was to look for conflicts early and resolve them deliberately: Promote often. Small, frequent promotions surface overlaps while both authors still remember their changes.Review the actual difference. Before accepting either version, compare the components line by line and confirm with both authors what the final behavior should be.Prefer permission sets to profile edits. Keeping profiles minimal and adding access through permission sets reduces the number of changes that collide in the same file. Each resolved conflict was recorded in the story, so the reasoning was available if the behavior was questioned later. Regression Testing for Automation Declarative automation can make mistakes that are easy to overlook in a cursory manual inspection. For a single record, a Record-Triggered Flow might function well, but for a bulk update, it might not. A current flow or integration could be blocked by a new validation rule. For one position, an approval procedure may route correctly, but for another, it may route wrongly. These problems were identified by the pipeline's automated regression tests before they affected customers. We concentrated our testing on business-critical activities, such as generating and updating important records, completing approval procedures from start to finish, and ensuring that automation operated appropriately when records were modified in large quantities. Custom code was covered by Apex tests, and if a test failed, the pipeline would not encourage a modification. Test results were just as important as the tests themselves. In order to ensure that a failed test indicated a true change in behavior rather than missing data, we maintained a small, known set of records in each testing environment that covered the roles, regions, and record kinds that the business utilized most frequently. Test records and anticipated outcomes were added to that set as part of the same tale when a new procedure went live. Not every click was intended to be covered. It was assurance that the operations on which the company relied continued to function with each release. Why This Matters More Now Salesforce orgs keep accumulating automation. Teams moving from Process Builder to Flow, as I have done in my current role, are consolidating logic that was previously scattered. AI-based features add new behavior that depends on the same data, permissions, and automation. The more an org does automatically, the more important it is to know exactly what was released and when. Release discipline is what keeps that growth manageable. It does not slow teams down for long. After the first few cycles, small stories, early conflict resolution, and automated tests make releases routine rather than stressful. A Short Checklist Link every change to a story and a reason. Configuration deserves the same traceability as code.Keep commits small and specific. Select only the components that changed.Promote often and resolve conflicts with both authors. Review the real differences, not just the file names.Test business-critical paths automatically. Include bulk updates, approvals, and integrations.Write release notes and root cause records. They turn production investigations into short tasks. Low-code platforms remove much of the effort of building software. They do not remove the need to manage change carefully. A pipeline that treats Salesforce configuration like code gives teams the speed of the platform without losing track of what they have built.

A coding agent opens a pull request for a background job that keeps retrying after the user cancels it. The test that caught the bug now passes, and the agent reports that all tests are green. But the diff changes two things: the retry handler and the test itself. Which change eliminated the failure, and would the test still catch the bug if it came back? I use coding agents to design integration tests, triage failures, and prepare fixes for back-end services, so this is a practical review question for me. Sometimes the test really is wrong, and changing it is the right fix. Sometimes the agent has just found the shortest way to a green build. In continuous integration, both look the same. Benchmarks show how far it can go. ImpossibleBench makes the tests contradict the task, so any passing solution has to cheat, and GPT-5 still passed 54% of its conflicting SWE-bench tasks. But tests aren't always right, either: OpenAI found issues with the tests or the task description in 59.4% of the 138 SWE-bench Verified problems it examined. So a test edit proves nothing by itself. Below, I'll show how to get evidence either way with a small Python example: one bug, three PRs that all pass CI, and a check that tells them apart in seconds. A Green Build Is a Statement About the Checks I find it useful to look at a test run as a pipeline. Tests are selected and set up, the application runs, the test observes and asserts, the harness reports the outcome, and a CI gate turns that outcome into a merge decision. Figure 1: A test run as a pipeline. A change to any stage other than the application changes what a green result means. Only the application stage can repair a defect. If a PR touches only the application, going from red to green tells you something about the code. If it also touches another stage, the green result may say more about the evidence than about the bug. One Bug, Three Pull Requests The Failing Test Here's the contract for the example. Once cancel() returns, no new attempt may start for that job. An attempt that's already running may still report its result, and a job that wasn't canceled must still be retried after a transient failure. The model below starts with one attempt already running and handles callbacks one at a time. run_retry() stands in for the scheduler starting a queued retry; there are no threads, network calls, or sleeps. Save it as retry_job.py: Python class RetryJob: def __init__(self): self.attempts = 1 self.canceled = False self.retry_pending = False def cancel(self): self.canceled = True self.retry_pending = False def on_transient_failure(self): self.retry_pending = True def run_retry(self): if self.retry_pending: self.retry_pending = False self.attempts += 1 The bug is about ordering. cancel() clears any queued retry, but the running attempt can report a transient failure afterward, and on_transient_failure() then queues another retry without checking whether the job was canceled. These two tests go in tests/test_retry_job.py: Python from retry_job import RetryJob def test_cancel_blocks_retry_after_late_failure(): job = RetryJob() job.cancel() job.on_transient_failure() job.run_retry() assert job.canceled is True assert job.attempts == 1 def test_uncanceled_job_retries(): job = RetryJob() job.on_transient_failure() job.run_retry() assert job.attempts == 2 Run python -m pytest -q from the repository root. The cancellation test fails because attempts is 2, and the ordinary retry test passes. Three Ways to Turn It Green The first PR leaves the application alone and loosens one assertion: diff - assert job.attempts == 1 + assert job.attempts >= 1 Now the test accepts a second attempt, and a third and a fourth. Its name still promises that cancellation blocks a retry, but the assertion no longer checks that promise. The second PR fixes the handler and leaves the tests alone: diff def on_transient_failure(self): + if self.canceled: + return self.retry_pending = True The cancellation test retains its meaning, and the ordinary retry test shows that the guard hasn't disabled retries for every job. The third PR does both. It's the easiest to miss: the application diff looks right because the application is right now, but the test can no longer catch the bug if a later change brings it back. Ordinary CI reports 2 passed for all three PRs. Run the Swap Check Ordinary CI runs exactly one combination: the PR's tests against the PR's code. The two combinations it skips answer the questions a reviewer actually has: Base tests on head code. Does the change still meet the original expectations? If an original test fails here, the PR changed a promise, and that needs a reason.Head tests on base code. Do the new tests still detect the bug? For a bug fix, at least one of them should fail here. If they all pass, they can't tell the buggy code from the fixed code. Figure 2 shows all four combinations for the three PRs: Figure 2: Swap matrices for the three pull requests. CI only shows the head code with head tests, which is green in all three. In an agent's PR, the tests are part of the change, so you have to swap them on purpose. Git worktrees make that cheap. These commands build the cell that matters most for a bug fix, the base code with the head's test files: Shell base=$(git rev-parse main) # commit IDs, because inside a worktree head=$(git rev-parse fix-and-weaken) # HEAD would mean something else tests=$(git diff --name-only --no-renames "$base" "$head" -- \ tests conftest.py pytest.ini) cell=$(mktemp -d) # base code + head tests git worktree add --quiet --detach "$cell" "$base" git -C "$cell" restore --source="$head" -- $tests (cd "$cell" && python -m pytest -q) git worktree remove --force "$cell" On the third PR, this prints 2 passed: the new tests can't see the bug. For the other cross cell, add the worktree at $head and restore the test files from $base. The PR that only loosens the assertion raises a flag in both cross cells, and the honest fix raises none. Whole-suite results are coarse, though. A head test can fail on the base code just because it imports something the fix added, and a test that already fails on the base hides the signal. In real use, compare outcomes test by test, as SWE-bench does with its FAIL_TO_PASS and PASS_TO_PASS lists. The list of test paths matters, too. It has to cover everything that defines a check, including fixtures, snapshots, and harness configuration such as pytest.ini, and files that mix application and test settings, such as pyproject.toml, need a human look. This conftest.py hook, for example, rewrites failed test reports as passed: Python import pytest @pytest.hookimpl(hookwrapper=True) def pytest_runtest_makereport(item, call): outcome = yield report = outcome.get_result() if report.failed: report.outcome = "passed" In a PR that adds only this hook, with only tests on the list, the swap check treats conftest.py as application code and reports a clean fix. Add conftest.py, and both warnings appear. Anthropic researchers describe this family of hacks, along with calling sys.exit(0) before any assertion runs. Treat the result as review evidence, not a verdict: a refactoring's new tests should pass on the old code, while a bug fix's reproducer should fail there. I'd run the swap check as a report-only CI job with the full Git history (fetch-depth: 0) and, because it runs the candidate's code, a read-only token that isn't left in the checkout (permissions: contents: read, persist-credentials: false). When Changing the Test Is the Right Fix Tests change all the time. In one comparison, human-written PRs modified existing tests more often than agent-assisted ones: 43.5% versus 20.6% in a manual sample. So the useful question is why a test changed. Four answers hold up: The requirement changed. If the service now lets one already-dispatched retry start after cancellation, the test has to say so, and someone has to have approved the new guarantee.The fixture was wrong. Maybe it labeled a permanent failure as transient. Fix it, but keep a transient-failure case so the retry path stays covered.The test looked too early. If cancellation is acknowledged asynchronously, wait for the acknowledgment or step a fake scheduler forward instead of adding a sleep.The test pinned an implementation detail. Move the check to the public boundary and show that it still catches a violation. This was the largest category in OpenAI's audit, at 35.5% of the examined problems. In none of these cases is the application's current output the reason for the new expectation. Ask for the contract, the corrected setup, or the reason the old observation was invalid, plus swap-check evidence that the revised test still fails on the bug. Guardrails In the Agent Loop An agent needs room to investigate failures, including bugs in tests, and a clear way to report a conflict between a requirement and an assertion rather than quietly resolving it. For the cancellation task, I'd commit the reproducer first, after checking that it fails for the right reason, and give the agent an instruction like this: Fix the retry behavior so that no new attempt starts after cancel() returns. Keep the late-failure reproducer and the ordinary retry case. If you think a test is wrong, stop, explain how it conflicts with the contract, and show any proposed test change separately. Report the commands you ran and any skipped, expected-failing, or unexecuted cases. To make tests read-only for an agent, Claude Code permission rules can deny edits to path patterns, and a PreToolUse hook can block a tool call by exiting with status 2; Cursor's preToolUse hook works the same way. These rules cover the agent's file tools, not every shell command, unless you also run Claude Code's sandbox with allowUnsandboxedCommands set to false. In my own setup, the agent that writes integration tests works only in test code, and its standing instructions forbid silencing or deleting a failing test to get a green run. Triage first classifies each failure as an application, environment, or test problem; application bugs become bug reports, not test edits. Changes to the agents' own instructions need a person's confirmation, and agents never merge. In the Repository Repository controls decide what gets merged. The useful ones keep the evidence for important behavior out of the candidate patch's reach, as Figure 3 shows: Figure 3: What the candidate patch can change, and what should be reviewed separately. Own the evidence. Assign test, fixture, harness, and CI paths to owners in CODEOWNERS, and turn on "Require review from Code Owners"; the file alone only requests reviews. GitHub reads it from the base branch, so a PR can't loosen its own owners. Give CODEOWNERS itself an owner, and require approval of the most recent push. Keep acceptance checks outside the patch. Keep a small acceptance suite for important contracts in a separate repository or at a pinned reference, and run it from a workflow the PR can't edit, such as a ruleset-required workflow on GitHub Enterprise Cloud or a pipeline execution policy on GitLab Ultimate. Run it from its own checkout with its own pytest.ini, because pytest searches parent directories for configuration and could pick up the candidate's conftest.py, and set PYTEST_DISABLE_PLUGIN_AUTOLOAD to block plugins the candidate's package might register. Prove that the check ran. Make the acceptance job unconditional: a required status check accepts a skipped result, and on GitHub "a job that is skipped will report its status as 'Success'." Then fail the job unless the required cases ran and passed. Any runner that writes JUnit XML, including pytest, Maven Surefire, and Gradle, can feed a short script that checks each required test by name. Don't count a test that passed only on a rerun; Surefire keeps the build green and records the failed attempt as a flakyFailure. Parse the report with defusedxml, since it's untrusted input. No single control catches everything. The required-tests check trusts a harness that lies, the acceptance run can't see a weakened test next to a real fix, and the swap check is only as good as its path list. Costs and Limits The swap check runs the suite four times, so for a large suite I'd limit it to the affected tests. It needs a base revision where the reproducer can run, flaky tests add noise to the cross cells, and each cell must import its own checkout rather than an installed package or a shared node_modules folder. These controls also target shortcuts rather than code written to defeat them. In a seventh variant of the example, retry_job.py patches pytest's report object when it's imported, and all three controls pass it. Where that matters, run acceptance tests against a built artifact or a running service, and keep reading the application diff. What to Ask for Before Approving Ask for evidence instead of a verdict. For the fixed example, a useful PR description would say: The late failure callback queued a retry after cancellation. The handler now ignores that callback when the job is canceled. The original cancellation test fails on the old code and passes on the repair; the ordinary retry test also passes. Removing the guard makes the cancellation test fail again. The assertion and event order are unchanged. For a real PR, I'd want to see five things before approving: The reproducer fails on the base code and passes on the head code.For a bug fix, at least one head test fails on the base code due to an assertion rather than an import.Every edited test has a stated reason: a changed requirement, a fixed fixture, a corrected observation point, or a removed implementation detail.Changes to selection, harness, or CI configuration were approved by their owners.The required tests ran and passed, by name, not just by exit code. Wrapping Up Reviewers have always had to ask why a test changed. Coding agents make the question come up more often, because they can rewrite code and tests in minutes and keep going until CI is green. Treat changes to tests, fixtures, the harness, and CI as changes to the evidence: swap tests, accept edits with a reason, and keep checks for important behavior out of the patch's reach. A passing run is only useful when you can explain why it passed.

A Genkit flow is an excellent unit of AI work. It is typed, traced, testable, and genkit.DefineFlow gives you something you can call from anywhere. What a flow is not is a unit of orchestration. The moment you have twelve documents to summarise, a flow whose output feeds two other flows, or a model call that fails on the third of forty items, you end up writing the same scaffolding again: a worker pool, a retry loop, a way to avoid re-running the eleven items that already succeeded, logs you can actually read, and some way to find out what happened at three in the morning. senro is a pipeline engine where a step already has all of that: retries that distinguish infrastructure from workload, an action cache that skips work it has already done, an append-only event stream, and a live attach protocol. I covered it in the introduction and the monorepo article. This article shows how to combine them. Put a Genkit flow inside a senro step, and every model call becomes a graph node with its own log, state, retry policy, and cache entry. Every example below ran against the real Gemini API with googleai/gemini-2.5-flash, on Genkit Go v1.13.1 and senro v1.4.0. The summaries are what the model actually returned. What You Need Shell go get github.com/firebase/genkit/go go get github.com/xavidop/senro The example is a small corpus of release notes in docs/*.md that gets summarised one document at a time, then merged into a digest. Step 1: The Flow Nothing senro-specific here. This is an ordinary Genkit flow: Go // SummarizeInput is what the summarize flow takes. type SummarizeInput struct { Path string `json:"path"` Text string `json:"text"` } // Summary is what it returns. type Summary struct { Path string `json:"path"` Summary string `json:"summary"` } func defineFlows(g *genkit.Genkit) *core.Flow[SummarizeInput, Summary, struct{}] { return genkit.DefineFlow(g, "summarize", func(ctx context.Context, in SummarizeInput) (Summary, error) { text, err := genkit.GenerateText(ctx, g, ai.WithSystem("You summarize release notes for an operator. "+ "Answer in one sentence, no preamble."), ai.WithPrompt(fmt.Sprintf("Summarize these release notes:\n\n%s", in.Text))) if err != nil { return Summary{}, err } return Summary{Path: in.Path, Summary: text}, nil }) } Called on its own, it does what you expect: Plain Text The billing reconciliation job was moved to a queue worker, accelerating invoice settlement from up to 24 hours to within two minutes. Step 2: Wrap the Flow in a Step senro has two kinds of steps: a command and a registered Go function. A Genkit flow is a Go function, so it is the second kind: Go // summarize is the registered flow. A func step runs in the pipeline's own // process, so a package-level handle is all the wiring the two need. var summarize *core.Flow[SummarizeInput, Summary, struct{}] type SummarizeParams struct { Path string `json:"path"` Name string `json:"name"` } func init() { senro.RegisterFunc("ai/summarize", RunSummarize) } // RunSummarize runs the flow and writes the summary into the one workspace // this step mounts. func RunSummarize(ctx senro.Ctx, p SummarizeParams) error { dir, ok := ctx.Workspace("doc-" + p.Name) if !ok { return fmt.Errorf("step %s mounts no doc-%s workspace", ctx.StepID(), p.Name) } text, err := os.ReadFile(filepath.Join(string(dir), "doc.md")) if err != nil { return err } res, err := summarize.Run(ctx, SummarizeInput{Path: p.Path, Text: string(text)}) if err != nil { return err } fmt.Fprintf(ctx.Stdout(), "%s\n", res.Summary) return os.WriteFile(filepath.Join(string(dir), "summary.txt"), []byte(res.Summary+"\n"), 0o644) } Three details make this work smoothly: senro.Ctx embeds context.Context, so it passes straight into summarize.Run with no adapter. Genkit's tracing and cancellation behave exactly as they always did.ctx.Stdout() is the step's real log stream, recorded and redacted like a command's output. Writing to os.Stdout instead reaches your terminal and no log file.ctx.Workspace(name) hands the function the same path a mount gives a command, on every executor. The function body does not change when the step later runs in a container or a pod. RegisterFunc registers the function once, from an init. The name is the function's identity: it is what the plan records and what feeds the step's cache key, so renaming it invalidates the cache exactly as renaming a command would. Parameters must be JSON-serializable, and decoding is strict, so a renamed field fails loudly instead of running with a zero value. Step 3: Build the Graph Because the pipeline is a Go program, fanning out over a corpus is a for loop: Go func main() { ctx := context.Background() g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}), genkit.WithDefaultModel("googleai/gemini-2.5-flash")) summarize = defineFlows(g) docs, err := filepath.Glob("docs/*.md") if err != nil { log.Fatal(err) } sort.Strings(docs) p := senro.New("ai") w := p.Workflow("summarize") var names, ids []string var wss []*senro.WorkspaceRef corpus := map[string]string{} for _, doc := range docs { text, err := os.ReadFile(doc) if err != nil { log.Fatal(err) } name := strings.TrimSuffix(filepath.Base(doc), ".md") names = append(names, name) ids = append(ids, "summarize/"+name) corpus[name] = string(text) // One workspace per document, so one document changing does not // invalidate the other summaries' cache keys. wss = append(wss, senro.Workspace("doc-"+name, senro.Scope(senro.ScopeRun))) } seed := w.Step("seed", senro.Func("ai/seed", SeedParams{Docs: corpus})) for i, ws := range wss { seed.Mount(ws.At("/doc-"+names[i], senro.RW)) } for i, name := range names { // One AI call, one node: its own log, state, retry and cache entry. w.Step(ids[i], senro.Func("ai/summarize", SummarizeParams{ Path: docs[i], Name: name, })). Needs("seed"). Mount(wss[i].At("/doc", senro.RW)). Retry(3, retry.OnInfra()). Pure(). Inputs(artifact.File("doc.md")). Outputs(artifact.File("summary.txt")) } digestStep := w.Step("digest", senro.Func("ai/digest", DigestParams{Names: names})). Needs(ids...) for i, ws := range wss { digestStep.Mount(ws.At("/doc-"+names[i], senro.RO)) } if err := senro.Run(ctx, p); err != nil { log.Print(err) os.Exit(1) } } seed writes each document into a workspace of its own. Then one summarize step per document, running in parallel. Then digest, which waits for all of them and merges the results. That loop produces this graph, rendered straight from the run's own plan.json: Plain Text pipeline: ai │ ├─ wave 1 seed func │ ├─ wave 2 (3 parallel) summarize/alpha func · retry 3/infra │ summarize/beta func · retry 3/infra │ summarize/gamma func · retry 3/infra │ └─ wave 3 digest needs summarize/alpha, summarize/beta, summarize/gamma · func Three documents give three parallel nodes. Thirty would give thirty, from the same for loop, with no change to the code. Step 4: Run It Run the pipeline binary directly, the way any Go program runs: Shell go run . Or hand the package to the senro CLI, which builds it, execs it, and attaches automatically. You get a terminal UI on a TTY and plain streaming lines anywhere else: Shell senro run . Plain Text seed -> succeeded summarize/beta -> succeeded summarize/alpha -> succeeded summarize/gamma -> succeeded digest -> succeeded Plain Text - alpha: The billing reconciliation job now uses a queue worker for invoice settlement within two minutes, replacing the nightly cron. - beta: The search index now rebuilds incrementally in under a second without blocking writes, replacing the old full rebuild and requiring operators to delete the `rebuild-search` cron job. - gamma: Service gamma's public API now features per-tenant rate limiting set at 100 requests per second, which may cause HTTP 429 errors for existing bursting integrations. Three real model calls in parallel, each with its own log file, and a merge step that ran once they were all done. While that is running, a second terminal can watch it, or steer it: Shell senro attach # terminal UI senro ui # browser view, prints a one-time link senro attach --run <id> # reopen a finished run from disk Every run also lands in runs/<id>/, so senro runs lists what happened and each step's stdout is a file you can read afterward. One thing to know when you switch between go run and senro run. A func step's cache key includes the binary digest, so the first run after changing how the binary is built re-runs every func step once, then caches normally from there. That is correct behaviour rather than a bug: the function's body lives in the binary, and senro cannot tell whether it changed without hashing it. Step 5: Stop Paying for the Same Call Twice Each summarize step declares Pure(). Run the pipeline again with nothing changed: Plain Text seed -> succeeded summarize/beta -> cached summarize/alpha -> cached summarize/gamma -> cached digest -> succeeded cached is a real state, not a label. Those three steps did not execute: no model call, no tokens, no latency. Their recorded outputs were restored from the content-addressed store, so digest still read three summaries and produced the same result. Now change one document out of three: Plain Text seed -> succeeded summarize/alpha -> cached summarize/beta -> cached summarize/gamma -> succeeded digest -> succeeded One document changed, one model call made. And senro will tell you which component of the key moved: Plain Text $ senro cache explain --run <id> summarize/gamma MISS summarize/gamma key d3eb7450 (previous f68652ad) ✗ input_digests: docs/gamma.md 6750cb09 → ef0fdb64 ✗ workspace_digests: doc-gamma 16774a4e → d43e13c8 ✓ command, env, secrets, executor_class, platform, mount_shape, step_shape, func_identity, tool_versions, version unchanged For a few dozen documents this is a convenience. For a nightly re-index over ten thousand where one file changed, it is the difference between a bill you notice and one you do not. Note func_identity in that list. For a func step, the cache key covers the binary digest, the registered name, and the parameter digest. So changing your prompt, your model, or the flow's code all correctly invalidate the cache. It is not keyed on the input file alone, which is the mistake a hand-rolled cache usually makes. Give Each Call Its Own Workspace This is the one structural rule to follow when caching model calls. Inputs feeds the input_digests component of the key, but Mount feeds workspace_digests, which is the digest of the entire mounted workspace. Narrow Inputs do not narrow it. Mount one shared tree across a fan-out and every step's key depends on every document, so changing one document re-runs all of them and the cache buys you nothing. One workspace per unit of work, as in the loop above, keeps each key dependent on its own input. Step 6: Let a Model Explain a Failure senro has a second place a model belongs: explaining a failed step. contrib/genkitanalyzer is a ready-made failure analyzer built on Genkit: Shell go get github.com/xavidop/senro/contrib/genkitanalyzer It is a nested module with its own go.mod, so senro itself never pulls in an AI SDK. If you never install it, your dependency graph never hears about Genkit. Go g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{})) p := senro.New("analyze") w := p.Workflow("build") w.Step("migrate", exec.Command("sh", "-c", `echo "dial tcp 10.0.3.7:5432: connect: connection refused" >&2; exit 1`)) err := senro.Run(ctx, p, senro.WithAnalyzer( genkitanalyzer.New(g, genkitanalyzer.Model("googleai/gemini-2.5-flash")), senro.AnalyzerName("genkit"), senro.AnalyzeTimeout(20*time.Second))) That is the whole integration. When the step fails, an analysis.proposed event lands in the run's ledger: JSON { "id": "migrate@1", "analyzer": "genkit", "duration_ns": 8344562458, "summary": "The 'migrate' step failed because its command explicitly reported a database connection refusal error and exited.", "detail": "... The error message \"dial tcp 10.0.3.7:5432: connect: connection refused\" typically indicates that a client attempted to establish a TCP connection to a server at 10.0.3.7 on port 5432, but the server actively rejected the connection. This could be due to the database server not running, not listening on the specified address/port, or being blocked by a firewall ..." } Because it is an event, the explanation shows up in the TUI, in events.jsonl, and in anything else folding the run. What the Combination Gives You Genkit gives you the model call: typed, traced, provider-agnostic, testable. senro gives you everything around it. ConcernWhat senro already does for a stepOrderingA dependency edge, not a place in a loopOutcomeTen states, so a call that passed on the second attempt is recovered, not silently a successRetriesretry.OnInfra() retries a dropped connection, never a model answer you did not likeLogsOne log file per step per attempt, redactedCostAn action cache keyed on the prompt, parameters, model, and binaryObservabilityOne append-only event stream you can attach to live or replay laterFailure analysisAn analyzer gated behind a human or an explicit policy You can build every one of those yourself. It is never quite the same twice, and it is rarely tested well. Conclusion The pattern is small enough to summarise in four lines: Define the flow with genkit.DefineFlow as you always would.Register a func step with senro.RegisterFunc that calls flow.Run(ctx, ...).Give each call its own workspace and declare Pure() with Inputs and Outputs.Add senro.WithAnalyzer if you want failures explained. From there the graph is a Go program, so the loop over your corpus is a for loop and the logic that decides what runs is testable with go test. senro on GitHubGenkit, and the Go documentationThe last article in this series: Debugging senro pipelines

By Xavier Portilla Edo CORE