DZone Spotlight

Wednesday, November 4 View All Articles »

When 12 Hours of Offline Changes Collide: Designing Conflict-Safe Mobile Sync

By Uthej Mopathi

CORE

Executive Summary Offline-first mobile systems treat local storage as an active source of application state rather than a disposable cache. That design improves responsiveness and preserves core behavior during long network outages, but reconnection creates a distributed-systems problem as local mutations may now compete with server updates or changes from other devices. Android's official offline-first guidance recommends a local data source for repositories that use the network and describes queued synchronization with persistent work and retry behavior. SQLite write-ahead logging can provide efficient local transactional durability, but WAL itself does not resolve cross-device conflicts. A reliable synchronization design therefore requires explicit versioning, deterministic merge rules, idempotent delivery, and observable reconciliation rather than a simple "upload the latest row" strategy. Reconnection Is a Versioning Problem Consider a record edited locally while the device remains offline for 12 hours. During that interval, the server may accept another update to the same record. Both versions can be valid descendants of an earlier state, so network recovery does not establish which value is semantically correct. The central question is not which copy arrived last, but whether one update causally follows the other or whether both happened concurrently. A local mutation log makes that distinction manageable. Each mutation can carry an operation identifier, entity identifier, base version, logical metadata, and payload. The local state change and the queued operation should be committed atomically so a crash cannot preserve one without the other. A compact operation can resemble: Plain Text op = { id: "op-8f31", entityId: "task-42", baseVersion: 17, type: "SET_STATUS", value: "DONE", occurredAt: 1792449120000 } On synchronization, the server checks the base version before applying the operation. A mismatch indicates that another accepted change advanced the entity and that normal application should stop until a merge rule runs. Plain Text apply(op, current) { if (op.baseVersion != current.version) return conflict(op, current) current.status = op.value current.version++ return current } This resembles optimistic concurrency control, but the important mobile property is durability across process death, retries, and extended disconnection. Android guidance explicitly describes queued writes for offline-first applications, while SQLite WAL records committed transactions in an append-oriented log and permits readers to continue while writes are committed. WAL improves local persistence characteristics where the mutation log supplies synchronization semantics. Conflict Resolution Must Match Data Semantics Last-writer-wins is attractive because it is cheap and deterministic. Cloud Firestore documents use last-write-wins behavior for multiple offline changes to the same document. A comparable custom merge can select the value with the greater timestamp: Plain Text merge(local, remote) { if (local.updatedAt != remote.updatedAt) return local.updatedAt > remote.updatedAt ? local : remote return local.replicaId > remote.replicaId ? local : remote } The simplicity hides a correctness risk. Device clocks can drift, timestamps can tie, and a later timestamp does not necessarily represent a more important business action. LWW works best for fields where overwriting is acceptable, such as a display preference, and poorly for additive or irreversible business events. A production rule also needs a deterministic tie-breaker, such as a stable replica identifier, so every participant reaches the same result. Vector clocks solve a different problem by representing causality rather than physical time. Amazon's Dynamo paper associates versions with node-counter pairs and uses clock comparison to determine whether one version descends from another or whether branches are concurrent. A simplified comparison can flag concurrency when neither clock dominates: Plain Text dominates(a, b) { return everyKey(k -> a[k] >= b[k]) && anyKey(k -> a[k] > b[k]) } conflicts(a, b) { return !dominates(a.clock, b.clock) && !dominates(b.clock, a.clock) } This preserves information that LWW discards, but metadata grows with participating replicas unless compaction or bounded identity schemes are introduced. Dynamo explicitly treats concurrent branches as versions requiring reconciliation, illustrating an important principle stating causality detection and semantic merging are separate responsibilities. CRDTs move more reconciliation into the data type. Shapiro and coauthors formalized state-based and operation-based CRDTs that converge when their required algebraic and delivery conditions hold. A grow-only set is the simplest example because additions commute and state merge is set union: Plain Text applyAdd(state, op) { return union(state, setOf(op.value)) } mergeGrowOnlySet(a, b) { return union(a, b) } No accepted addition is lost, and merge is associative, commutative, and idempotent. Operation-based CRDTs can transmit smaller mutations instead of full state, but they require stronger delivery properties. Automerge demonstrates the broader local-first model by recording changes and merging concurrent edits without a central coordinator. CRDTs are powerful for collaborative or additive data, but metadata, tombstones, history retention, and domain constraints can increase storage and implementation cost. Server-authoritative reconciliation remains appropriate when the backend owns invariants that cannot be decided safely on a disconnected client. Inventory allocation, authorization state, settlement, and policy enforcement commonly require server validation. In that model, offline mutations express intent, while reconnection submits intent against the current authoritative state: Plain Text reconcile(op, current) { if (!policy.allows(op, current)) return rejected(current) return applyValidated(op, current) } The server may accept, transform, reject, or return a conflict representation that preserves enough context for deterministic recovery. Couchbase Lite exposes conflict-resolution mechanisms for replicated documents, reinforcing that replicated state needs an explicit policy rather than an accidental overwrite. Delivery Must Be Idempotent and Observable Retries are inevitable after ambiguous failures. A request may reach the server even when the response never reaches the device, so replaying the same mutation must not duplicate its effect. Stripe's API documentation describes idempotency keys specifically for safely retrying requests. A client-generated operation identifier can become that key: Plain Text op.idempotencyKey = uuid() save(op) sync(op) { existing = dedup.find(op.idempotencyKey) if (existing != null) return existing.response result = apply(op) dedup.save(op.idempotencyKey, result) return result } The deduplication record and business mutation should participate in the same transactional boundary where possible. On server systems that also publish downstream events, a transactional outbox can persist the state change and event record together, where Debezium documents the outbox pattern as a way to avoid inconsistencies between stored state and emitted events. Correctness still competes with product and operational costs. Automatic LWW minimizes interaction and metadata but may silently discard meaningful work. Vector clocks preserve concurrency information but add version metadata and reconciliation paths. CRDTs can improve convergence and offline collaboration but may increase stored history, bandwidth, and implementation complexity. Server-authoritative reconciliation protects invariants but can surface rejected offline work after a long delay, creating a difficult user experience. No single policy fits every field, so conflict behavior should be defined at the domain level rather than globally. Production controls should make synchronization behavior inspectable. Deterministic merge functions should produce the same output for the same inputs. Reconciliation endpoints should return authoritative state, accepted operation identifiers, and explicit conflict metadata. Unresolved conflicts should be observable through metrics and diagnostics rather than disappearing into retry loops. Automated tests should exercise duplicated operations, reordered delivery, clock skew, stale base versions, concurrent edits, deletion-versus-update races, process restarts, and repeated reconnects. Background synchronization should also use bounded retries and connectivity constraints where Android WorkManager supports network constraints and retry with backoff for persistent background work. Conclusion An application returning after 12 hours offline does not merely need to "sync the newest data." It needs to establish causality, preserve durable intent, prevent duplicate effects, and resolve concurrent state according to domain semantics. LWW is useful when overwrite is acceptable, vector clocks expose concurrency, CRDTs encode convergence into suitable data structures, and server-authoritative reconciliation protects centralized invariants. The strongest offline-first systems combine those techniques selectively, persist operations transactionally, attach stable idempotency keys, expose reconciliation outcomes, and test failure ordering as aggressively as ordinary business logic. Network recovery should trigger a deterministic protocol, not a race between copies. That distinction turns offline synchronization from a fragile retry mechanism into a controlled consistency model. More

Designing the Runtime Layer for Stateful AI Agents: Where Agent Substrate Fits

By Josephine Eskaline Joyce, Ph.D

CORE

Introduction: The Layer Nobody Designs AI agents are evolving from simple prompt-response systems into stateful, long-running applications that can reason, execute tasks, and interact with external systems. Yet, most agent architectures are driven by models, tools, memory, and orchestration — not runtime infrastructure that can run agents reliably. Unlike traditional applications, stateful AI agents need execution context to run (and recover from failures), concurrent tasks, and the ability to resume workflows without losing progress if they fail, etc. These capabilities cannot be provided by prompts or orchestration frameworks alone — there needs to be a runtime layer. This is where Agent Substrate becomes relevant. It provides a runtime foundation for stateful agent execution so that developers can separate the intelligence of the agent from the infrastructure that runs it (aka runtime). A closer look at stateful AI agents reveals why a dedicated runtime layer is essential, the architectural challenges it must address, and how Agent Substrate fits into the broader agentic AI technology stack. What Makes Agent Workloads Different AI agents might look like traditional workloads, but their execution patterns challenge the fundamental assumptions of application infrastructure. Request-driven services or batch jobs cannot run forever; agents can make random decisions, interact with other systems, and change their own state continuously. Five characteristics that make their runtime requirements fundamentally different. Long-lived, mostly waiting: An agent task can take minutes, hours, or days, and most of that time is spent waiting on a model, a tool, or a human. Holding a process open for the whole duration is wasteful and fragile. Requirement: Execution must be able to pause, release resources, and resume.Non-deterministic: The same input can take a different path each run. You can’t recover by re-running the code and expecting the same result, so what the agent did and decided has to be recorded as it happens. Requirement: Durable records of steps and outcomes, not just final state.Side effects on the real world: Agents send emails, move money, and modify systems. A naive retry after a crash can repeat an action that already happened. Requirement: Know exactly which steps completed, and make retries safe.State is the product: Memory, plans, and open tasks accumulate and are read and rewritten constantly. This isn’t a cache you can throw away; losing it means losing the agent. Requirement: State that is durable, versioned, and structured enough to tell current from stale.Many actors, shared state: Agents act for users, hand work to other agents, and collaborate with humans. Without boundaries, they overwrite each other, and nobody can say who decided what. Requirement: Isolation, access control, and an audit trail. Stateful AI agents need the durability of a database, the recoverability of a workflow engine, and the isolation of a multi-tenant platform, all integrated into a coherent execution environment. While traditional infrastructure offers these capabilities independently, agent workloads demand them together. This is the architectural gap that an agent runtime layer must address. The Lifecycle of a Stateful AI Agent Traditional applications run in predictable request-response cycles, but stateful AI agents run through dynamic long-running workflows. Their execution involves multiple reasoning steps, tool calls from the outside world, human approvals, and periods of inactivity. A typical stateful agent lifecycle involves six stages: Initialize: Receive a task, establish execution context, and create an initial state.Execute: Perform reasoning, make decisions, invoke tools, and update execution state.Persist: Save progress, intermediate results, and relevant context at durable checkpoints.Suspend: Pause execution while waiting for an external event, tool response, or human approval.Resume or Recover: Continue from the last valid checkpoint when the required event arrives or recoverable failures occur, avoiding duplicate side effects.Complete: Finalize the task, record the outcome, and retain relevant execution history for auditability. Importantly, this lifecycle is not strictly linear. An agent may repeatedly execute, persist, suspend, and resume before reaching completion. Failures may also trigger retries, recovery, or escalation. The runtime layer is responsible for managing these transitions reliably, ensuring that agent execution remains durable, observable, and recoverable without requiring the application developer to implement the lifecycle infrastructure from scratch. Why Traditional Infrastructure Is Not Enough Traditional infrastructure provides many of the building blocks needed to run AI agents, but no single piece of it covers the entire lifecycle of a stateful agent. The table below shows where traditional infrastructure falls short. Need Typical tool Why it is not enough for agents Preserve execution progress Stateless service Handles requests, but does not inherently retain agent state or progress across steps Isolated execution Containers Provide process isolation, but not durable execution or recovery Asynchronous processing Queue Moves work reliably, but does not manage full agent context, memory, or state transitions Durable persistence Database Stores data, but does not coordinate execution, retries, or recovery Orchestration and retries Workflow engine Provides strong primitives, but still needs integration with agent memory, policies, and model-driven behavior The real challenge is not the absence of infrastructure primitives. It is the need to combine them into a coherent runtime layer that can preserve state, resume execution, recover from failure, and enforce policy across long-running agent workflows. Designing the Runtime Layer for Stateful AI Agents A stateful AI agent needs more than compute resources and an execution environment. It also needs a runtime that preserves execution state, handles long-running processes, recovers from interruptions, and efficiently allocates resources. Unlike traditional application runtimes, which handle processes and requests only, an agent runtime has workloads that are frequently switching between active execution and waiting for external events. A well-designed agent runtime should address five core capabilities. These capabilities lay the foundation for running stateful agents reliably without requiring every application to build its own execution infrastructure. Runtime capability Responsibility State persistence Preserve execution context, memory, and intermediate progress across interruptions Lifecycle management Manage agent creation, execution, suspension, resumption, and termination Execution isolation Run agents in secure, isolated environments Resource efficiency Release compute resources when agents are idle and restore them when needed Recovery and continuity Restore saved execution state after interruptions and support reliable continuation Agent Substrate is an open-source runtime for running stateful workloads (e.g., AI agents) on Kubernetes. Instead of treating each agent as a container permanently running, it uses an actor-based execution model where instances of agents can be suspended when idle and resumed when needed. Its architecture combines: Actor lifecycle management: Tracks actor instances and manages their suspension and resumption.Checkpointing and restoration: Preserves process memory and writable filesystem state through snapshots.Worker pools: Maintains available compute capacity for scheduling and restoring actors.Execution isolation: Supports sandbox technologies such as gVisor and microVMs.Kubernetes integration: Uses Kubernetes for underlying worker provisioning and infrastructure management. Separating Reasoning, Orchestration, and Runtime Infrastructure A reliable agent architecture separates three responsibilities: what the agent decides; how its work is coordinated; and where its execution is maintained. They are tightly coupled, but they address different problems (though they interact). Layer Responsibility Examples Agent reasoning Determines what actions to take, selects tools, and interprets results LLMs, LangGraph, CrewAI Execution orchestration Coordinates tasks, dependencies, retries, approvals, and workflow transitions Workflow engines, agent orchestrators Runtime infrastructure Manages execution environments, state persistence, isolation, suspension, and recovery Agent Substrate, Kubernetes-backed runtime services The reasoning layer decides what to do next; the orchestration layer coordinates when and how that action is executed. The runtime infrastructure ensures that execution can continue reliably, even when agents become idle, encounter failures, or require additional resources. Where Agent Substrate Fits Traditional infrastructure provides the foundation for running applications, but stateful AI agents need a runtime layer to manage execution continuity, isolation, and resource utilization. Agent Substrate is an open-source Kubernetes-based runtime that addresses these requirements. Instead of treating each agent as a running container, it uses actor-based execution so stateful workloads can be suspended when idle and restored when execution needs to continue. Agent Substrate introduces several capabilities that align with the runtime architecture discussed earlier: Actor-based execution environments: Each agent workload is represented as an actor with its own execution context, enabling independent management of stateful instances.Actor lifecycle management: Coordinates actor creation, placement, execution, suspension, resumption, and termination.Suspension and resumption: Allows inactive actors to be suspended and restored when new requests arrive, reducing the need for continuously allocated compute.Memory and filesystem preservation: Uses snapshots to capture process memory and writable filesystem state, enabling execution to continue across suspension and restoration.Worker pools and resource multiplexing: Maps a large number of stateful actors onto a smaller pool of available workers, improving infrastructure utilization.Isolation and request routing: Uses sandbox technologies such as gVisor and microVMs to isolate execution, with ingress routing and controlled network access to manage communication securely. Mapping Agent Substrate to the Runtime Architecture Runtime capability How Agent Substrate addresses it State persistence Process memory and filesystem snapshots Lifecycle management Actor creation, suspension, resumption, and termination Execution isolation Sandboxed actors using gVisor or microVMs Resource efficiency Worker pooling and actor multiplexing Recovery and continuity Snapshot restoration and resumable execution Request management Actor-aware routing and network mediation Why This Matters Consider an AI agent that initiates a task, invokes external tools, and then waits for human approval. In a conventional deployment, its execution environment may remain allocated throughout the waiting period. With Agent Substrate, the actor can be suspended, its execution state preserved, and its compute resources released. When execution needs to continue, the actor can be restored onto an available worker. This separation allows many stateful workloads to be managed without maintaining a dedicated running environment for every idle agent. Agent Substrate does not replace agent frameworks or workflow orchestration engines. It provides the execution infrastructure; frameworks do reasoning and application logic, while runtime does isolation, state preservation, scheduling, and continuity of execution. Real-World Use Case: Stateful AI Incident Investigation Various use cases of stateful AI agents where Substrate can be useful are – Use case How Agent Substrate helps Implementation complexity AI incident investigation agent Preserves investigation context while waiting for telemetry or human approval Medium AI coding agent Maintains isolated development environments across coding sessions Medium AI customer support agent Preserves session state during long-running interactions Low–Medium AI infrastructure remediation agent Suspends while awaiting approval and resumes with its execution context Medium–High AI research agent Preserves intermediate work across extended research sessions Medium Implementation Details: AI Incident Investigation Consider an AI incident investigation agent that diagnoses infrastructure failures. When an alert fires, the agent collects telemetry, looks at logs, correlates recent deployments, and builds an investigation context. Unlike a request-response application, the investigation may take several minutes or hours. The agent might have to wait for monitoring data, an engineering response, or human approval before continuing. Keeping a dedicated execution environment running during this wait time is unnecessary. Restarting the agent, however, requires reconstructing its investigation context and intermediate results. An agent runtime (e.g., Agent Substrate) solves this problem by providing a stateful execution environment that can be suspended and restored. For example: Initialize: An infrastructure alert starts an investigation actor.Execute: The actor collects logs, metrics, and deployment information.Persist: Investigation context and intermediate findings are retained in the actor's execution state.Suspend: The actor is suspended while awaiting additional information or human approval.Resume: When new information becomes available, the actor is restored and continues using its preserved context.Complete: The investigation produces a diagnostic summary and recommended remediation actions. Step 1: Build a Minimal Incident Investigation Service Create a Python or Go application that exposes: endpointpurpose POST /investigate Start an investigation and save the incident context GET /status Return the current investigation state POST /evidence Add new telemetry or investigation findings POST /complete Finalize and summarize the investigation Step 2: Maintain Investigation State The actor could preserve a state object similar to: JSON { "incident_id": "INC-1042", "service": "payment-api", "status": "waiting_for_approval", "investigation": { "error_rate": "12%", "suspected_cause": "recent deployment", "deployment": "v2.4.1" }, "approval_required": true, "recommended_action": "review rollback" } This represents application-level investigation context. The actor must actually load and update this data for the demo to demonstrate preservation. Step 3: Deploy the Service as an Agent Substrate Actor Package the application as an OCI image and create an Agent Substrate actor template configured with an appropriate sandbox, worker pool, and full-state snapshot policy. The existing counter example provides a useful reference for configuring an actor template. Step 4: Demonstrate Suspension and Resumption Once the actor has stored its investigation context, manually suspend it: Shell kubectl ate suspend actor incident-1042 -a incident-demo When a new request arrives through the configured actor router, the suspended actor can be activated and its state restored. You can also use the explicit resume lifecycle operation. These commands assume you have created an actor named incident-1042 in an atespace named incident-demo; they are not commands for the existing counter demo. Step 5: Verify the Result After resumption, retrieve the investigation status and verify that the agent still has its original findings. Hands-on Tutorial: Running and Resuming a Stateful Workload with Agent Substrate The objective of this tutorial is to execute a stateful workload, suspend it, and resume it without losing its execution state. This experiment uses a local Kubernetes environment with Podman and Kind. Setting Up Agent Substrate First, clone the Agent Substrate repository Shell git clone https://github.com/agent-substrate/substrate.git cd substrate With Podman configured as the container provider for Kind, create the local Kubernetes cluster and install the Agent Substrate runtime: Shell export KIND_EXPERIMENTAL_PROVIDER=podman ./hack/create-kind-cluster.sh ./hack/install-ate-kind.sh \ --deploy-ate-system \ --credential-provider='{"name":"k8s.io"}' Note: The upstream installation scripts assume Docker. Our Podman-based setup requires local compatibility adjustments, so this should be treated as an experimental installation path rather than an officially supported deployment configuration. Once deployed, verify that the runtime components are running: kubectl get pods -n ate-system Deploying the Stateful Counter Agent Substrate provides a counter demo that maintains state across three locations: Process memory: An in-memory counter.Filesystem: A counter stored in a file.Root filesystem: A counter stored in the actor's writable root filesystem. Deploy the demo and create an actor instance: Shell ./hack/install-ate-kind.sh --deploy-demo-counter kubectl ate create actor my-counter-1 \ -a ate-demo-counter \ --template counter Forward the router service to the local machine: Shell kubectl port-forward -n ate-system \ svc/atenet-router 8000:80 From another terminal, invoke the actor: Shell curl -X POST \ -H "ate-target-actor: ate-demo-counter/my-counter-1" \ http://localhost:8000/ The initial execution returned: Each subsequent request increments all three counters, demonstrating stateful execution within the actor. Check the status of the actor: Suspending the Actor After invoking the actor multiple times, suspend it using Agent Substrate's lifecycle controls. During suspension, Agent Substrate checkpoints the actor's execution state and releases its active execution slot. Unlike terminating a conventional container, suspension is designed to preserve the state required to continue execution later. Resuming the Actor Resume the suspended actor and invoke the actor again. Instead of restarting the counters from their initial values, the actor continues incrementing from its previously preserved state. For example, if all counters reached 5 before suspension, the next successful invocation produces: This illustrates the key capability of the runtime: execution can be suspended and restored while preserving both in-memory and filesystem state. Observing the Results The experiment demonstrates the following behavior: Stage Memory Counter File Counter Rootfs Counter Initial execution 1 1 1 Before suspension 5 5 5 During suspension Checkpointed Preserved Preserved After resumption and next request 6 6 6 Illustrative values for five requests before suspension; actual counts depend on the number of requests executed. The important observation is not the counter value itself, but its continuity across the suspension and resumption boundary. For the complete Podman, Kind, and Agent Substrate setup instructions, along with all tutorial execution commands, refer to the GitHub repository. What Happens Behind the Scenes? Agent Substrate coordinates several runtime mechanisms to support this behavior: Actor lifecycle management: Tracks running, suspended, and resumed actors.Checkpointing: Captures execution state, including process memory and filesystem changes, according to the configured snapshot scope.Snapshot storage: Retains checkpoints so the actor can be restored.Worker management: Releases execution resources during suspension and allocates resources when the actor resumes.Request routing: Directs subsequent requests to the appropriate actor execution environment. The experiment shows that there is a fundamental difference between normal process execution and stateful actor runtime: an actor can preserve its logical execution state without continuously occupying an active worker execution slot. This is particularly useful for AI agents that spend large portions of their lifetime waiting for external tools, model responses, or human decisions. However, preserving execution state does not guarantee exactly-once execution of external side effects or transactional consistency. Those concerns require additional application-level controls. Architectural Trade-Offs and Production Considerations Agent Substrate provides capabilities for running stateful AI agents efficiently, but having a runtime layer of its own has architectural trade-offs too. Production environments need to balance execution continuity with resource use, performance, security, and operational complexity. The following considerations are important when designing a production-ready agent runtime. Architectural consideration Trade-off / Production implication State persistence vs. storage overhead Frequent snapshots improve recoverability but increase storage consumption, checkpoint latency, and I/O overhead. Dedicated environments vs. shared worker pools Dedicated environments provide predictable resource availability, while shared pools improve utilization but introduce scheduling and capacity-management challenges. Resource efficiency vs. resume latency Suspending idle agents reduces compute costs, but restoring execution may introduce latency when processing new requests. Isolation vs. operational complexity Sandboxing strengthens workload isolation but introduces additional runtime configuration, monitoring, and debugging requirements. Infrastructure vs. workflow recovery Restoring process state enables execution continuity but does not guarantee successful recovery of application-level workflows or external operations. Security, policy, and observability Production deployments require authentication, authorization, audit logging, secrets management, and end-to-end tracing beyond basic execution management. Beyond Infrastructure-Level Recovery One important point to note is that preserving an agent’s execution state is not the same as guaranteeing correctness of its actions. For example, an agent calls a payment API and fails immediately after the payment succeeds but before recording the response. Restoring the agent from its last checkpoint could cause the payment request to execute again. A runtime can restore execution context, but it cannot tell whether an external operation was done successfully or not. Production systems therefore need additional mechanisms (idempotency keys, durable operation records, transactional outbox patterns and reconciliation workflows) to reduce duplicate side effects and ensure consistency across system boundaries. Similarly, runtime-level checkpointing does not guarantee transactional consistency between agent state, databases, and external services. Security and Operational Readiness Beyond execution management, production deployments require integration with broader platform capabilities, including identity-based access controls, policy enforcement, secrets management, observability, and resource governance. These capabilities help ensure that agents operate within defined security boundaries and that their execution can be monitored, audited, and controlled. The key architectural principle is that an agent runtime provides execution continuity, while application-level orchestration and governance remain responsible for ensuring operational correctness. Agent Substrate addresses important infrastructure challenges, but building production-ready agentic systems still requires coordinated design across the reasoning, orchestration, runtime, and platform layers. Conclusion As AI agents evolve from simple request-response systems to autonomous long-running workloads, the infrastructure supporting them has to evolve too. Reasoning capabilities aren’t enough; agents need reliable execution, persistent state, isolation, and recovery to work in production. Traditional infrastructure provides the building blocks, but integrating them into a coherent runtime remains a challenge. Agent Substrate demonstrates how an actor-based execution model, combined with checkpointing, suspension, resumption, and resource multiplexing, can help bridge this gap. However, a runtime is only one part of the broader agentic architecture. Workflow orchestration, security, policy enforcement, and application-level consistency remain equally important for building trustworthy systems. Ultimately, the next step in agentic AI is not just making agents more intelligent, but making their execution more reliable, resilient, and operationally sustainable. A well-designed runtime foundation is what enables intelligent agents to move from experimentation to production-scale systems. More

Testing Java and Spring Applications With Kotlin: Low-Risk to Adoption

By Aniket Bhattacharyea

How to Build Your Own AI SRE From Scratch

By Severin Neumann

Running a Self-Hosted WAF in Production: Practical Lessons From SafeLine

By Carrie Luo

Why Kotlin or Java Is the Wrong Question For Enterprises

By Aniket Bhattacharyea

Genkit Flows on Kubernetes, Routed by a Decision Model: senro + jev in Go

Genkit flows as senro steps showed how to put a Genkit flow inside a senro step and get retries, caching, and an event stream for every model call. Every step in that article ran on your own machine. This one moves the steps into Kubernetes and adds a second kind of model. Gemini writes. jev decides. jev is a decision model: it does not generate text; it evaluates a state against typed questions and answers each one with a probability, in a few hundred milliseconds. Genkit Go 1.14 reaches it through the System One plugin, and a decision is one genkit.GenerateData call whose output type is the question set. That makes jev the right model to sit at the switch points of a pipeline. In this guide, it decides twice: which department's workflow runs for a support ticket, and whether the reply Gemini drafted is safe to send. Everything below runs against the real Gemini API (gemini-3.8-flash) and the real TypeSafe API (jev-latest), on Genkit Go v1.14.0, senro v1.4.5, and a local kind cluster on Kubernetes 1.37. The code is in genkit-k8s-flows-jev. Why a Decision Model Decides, and a Chat Model Writes Asking a chat model "which team should own this ticket, answer with one word" works until it does not: you get prose around the word, a department that is not on the list, no idea how sure the model was, and a second or two of latency for a decision that should take milliseconds. A decision model is built for exactly that call. Chat model as routerDecision model as routerAnswerText you parseA typed field: Choice[Dept], Noul, Score[L]OptionsIn the prompt, hoped for in the outputIn the Go type, enforced by the schemaConfidenceNone, or whatever the model says about itselfA probability per option, calibrated per model versionToolsCan be called by accidentRefused by the protocolLatencySecondsA few hundred millisecondsCost per decisionTokens for the state and the proseTokens for the state And the reverse is just as true. jev cannot write the reply to the customer, explain an outage, or summarize a brief. That is what gemini-3.8-flash is for. The pipeline needs both, and senro is what puts each call in its own step, on its own pod, with its own log, retry policy, and secret. What We Are Building A support-ticket pipeline. A raw email comes in; a reply goes out, or a brief goes to a human, or both. Two runs, because of one rule in senro: a When condition is evaluated once, at run start, against facts the run already has, never against something a step produces. So the first run understands the ticket and decides, and the decision becomes the run parameters of the second run, where senro.When(senro.ParamIs(...)) prunes every branch the verdict did not pick before anything starts. Pruned steps settle as skipped_condition and the run stays green. RunStepWhereModelWritestriageseedcoordinatornoneputs the raw email into the run's workspacetriageingestpodGeminiticket.json, a structured tickettriagetriagepodjevdecision.json: department, urgency, frustration, needs a humantriagecollectcoordinatornonecopies both files out of the runresolve<dept>/draftpodGeminireply.md, following that department's playbookresolve<dept>/verifypodjevverdict.json; a rejection fails the stepresolve<dept>/sendcoordinatornonethe reply, only if verify passedresolveescalate/briefpodGeminibrief.md for a human, when jev said so Inside a branch, jev is a guardrail: verify reads the draft against the playbook, and if it rejects the draft, the step fails and senro skips send. The second decision also determines the next step, this time without leaving the run. What You Need Go 1.27 or newer, Docker, kind, and kubectlGEMINI_API_KEY and TYPESAFE_API_KEY in your environmentthe senro CLI if you want the terminal UI (optional) Shell go get github.com/firebase/genkit/[email protected] go get github.com/xavidop/[email protected] go get github.com/xavidop/mamori mamori is the library senro uses to resolve secrets. It is how the two API keys reach the pods without ever being in a pod spec. Step 1: A Cluster senro Can Reach senro's Kubernetes executor never reads KUBECONFIG, ~/.kube/config or your current context. A kubeconfig typically holds a dozen contexts, most of them production, with whichever one you last used selected; a pipeline that inherited that would eventually deploy into the wrong cluster. Instead, the cluster comes from three environment variables, read at run start: VariableWhat it isSENRO_K8S_SERVERThe apiserver URLSENRO_K8S_CA_FILEThe cluster CA, as a PEM file. senro will not skip verificationSENRO_K8S_TOKEN_FILEA bearer token in a file (or a client certificate and key) So a local cluster needs four things: kind, a namespace, an identity with the permissions the executor documents, and those three files. The repo's scripts/cluster-up.sh does all of it; this is what it runs. Shell kind create cluster --name senro-ai --kubeconfig .kind/kubeconfig export KUBECONFIG=$PWD/.kind/kubeconfig kubectl apply -f k8s/rbac.yaml The RBAC is the exact set senro needs and nothing more: in the namespace, create, get and delete on pods, get on pods/log, create and get on pods/exec, and create, patch, delete on secrets. At cluster scope, list and get on nodes, which is how senro reads the platform it has to compile for. The get on pods/exec is the one to remember: senro uses the WebSocket exec protocol, which is a GET on that subresource, and a Role with only create fails every step that carries a workspace. YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: senro-steps namespace: support rules: - apiGroups: [""] resources: ["pods"] verbs: ["create", "get", "delete"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create", "get"] - apiGroups: [""] resources: ["secrets"] verbs: ["create", "patch", "delete"] Then the three files, pulled out of the kubeconfig kind wrote and a short-lived ServiceAccount token: Shell kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.server}' > .kind/server kubectl config view --minify --raw \ -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > .kind/ca.crt kubectl -n support create token senro --duration=4h > .kind/token export SENRO_K8S_SERVER=$(cat .kind/server) export SENRO_K8S_CA_FILE=$PWD/.kind/ca.crt export SENRO_K8S_TOKEN_FILE=$PWD/.kind/token A file rather than SENRO_K8S_TOKEN inline, because an environment variable is visible in /proc/<pid>/environ and inherited by every child process the coordinator spawns. Step 2: The Flows Five flows, in one file. Nothing here knows about senro or Kubernetes. Gemini Turns an Email into a Ticket Go type Ticket struct { Customer string `json:"customer"` Company string `json:"company,omitempty"` Summary string `json:"summary"` Ask string `json:"ask"` Facts []string `json:"facts"` } f.Ingest = genkit.DefineFlow(g, "ingestTicket", func(ctx context.Context, raw string) (Ticket, error) { t, _, err := genkit.GenerateData[Ticket](ctx, g, ai.WithModelName("googleai/gemini-3.8-flash"), ai.WithSystem("You turn raw support emails into a structured ticket. "+ "Facts are identifiers, amounts, dates and error codes quoted from the email, nothing inferred."), ai.WithPrompt("Email:\n\n%s", raw)) if err != nil { return Ticket{}, err } if t == nil { return Ticket{}, fmt.Errorf("ingest: the model returned no ticket") } return *t, nil }) jev Decides What the Ticket Is With the System One plugin, the output type is the question set. Each field is one question; its jsonschema_description tag is the instruction, and the options, levels, and criteria belong to Go types, so the same question means the same thing wherever it is asked. Go import systemonex "github.com/firebase/genkit/go/plugins/systemone/exp" type Dept string func (Dept) Criteria() map[Dept]string { return map[Dept]string{ "billing": "Charges, invoices, refunds, subscriptions, plan changes", "technical": "Bugs, errors, outages, latency, integrations, webhooks, API", "account": "Login, password, permissions, data export or deletion, privacy", "other": "None of the above", } } type Frustration int func (Frustration) Levels() []string { return []string{"Calm", "Annoyed", "Angry", "Furious"} } type Urgent struct{} func (Urgent) Criteria() (yes, no string) { return "Names a deadline, says now or today, or describes an outage in progress", "No time pressure is expressed, or the customer says there is no rush" } type Triage struct { Department systemonex.Choice[Dept] `json:"department" jsonschema_description:"Which team should own this ticket?"` IsUrgent systemonex.NoulOf[Urgent] `json:"is_urgent" jsonschema_description:"Does the ticket communicate time pressure?"` Frustration systemonex.Score[Frustration] `json:"frustration" jsonschema_description:"How frustrated is the customer?"` NeedsHuman systemonex.Noul `json:"needs_human" jsonschema_description:"Does resolving this need a human: a legal threat, a contract or SLA dispute, data loss, or a refund outside the playbook?"` } Four question kinds cover the protocol. Choice[T] picks one of T.Criteria() and gives a probability per option plus a Confidence. Score[L] rates on L.Levels() and answers Level() and Label(). Noul is "is this true?" as a probability, where 0.5 means the model could not tell. NoulOf[C] is the same with C.Criteria() saying what yes and no mean. The flow sends the ticket as a data part, so the state is a JSON object and a question can refer to a field by name. It then turns the four answers into a flat verdict, with thresholds that belong to the caller: Go type Decision struct { Department string `json:"department"` Confidence float64 `json:"confidence"` Urgent bool `json:"urgent"` Frustration string `json:"frustration"` Escalate bool `json:"escalate"` Model string `json:"model"` } f.Triage = genkit.DefineFlow(g, "triageTicket", func(ctx context.Context, t Ticket) (Decision, error) { out, resp, err := genkit.GenerateData[Triage](ctx, g, ai.WithModelName("typesafe/jev-latest"), ai.WithPromptParts(ai.NewDataPart(t))) if err != nil { return Decision{}, err } d := Decision{ Department: string(out.Department.Choice), Confidence: out.Department.Confidence, Urgent: out.IsUrgent.Probability >= 0.5, Frustration: out.Frustration.Label(), Model: systemonex.ResponseInfo(resp).Model, } // A low-confidence routing decision is not a routing decision. if d.Confidence < 0.5 { d.Department = "other" } d.Escalate = d.Department == "other" || out.NeedsHuman.Probability >= 0.6 || out.Frustration.Level() >= 3 return d, nil }) ResponseInfo(resp).Model records which jev version answered. Confidence thresholds are calibrated per version, so a verdict that does not say who produced it is a threshold waiting to drift. Gemini Drafts, jev Reviews The draft flow is an ordinary GenerateData[Reply] call with the playbook and the ticket in the prompt. One difference from the triage flow: the ticket goes to Gemini as JSON text. A data part is how a decision model takes an object state; a chat model takes a prompt. Go f.Draft = genkit.DefineFlow(g, "draftReply", func(ctx context.Context, in DraftInput) (Reply, error) { r, _, err := genkit.GenerateData[Reply](ctx, g, ai.WithModelName("googleai/gemini-3.8-flash"), ai.WithSystem("You are a support agent for Acme Cloud. Write the reply the customer will receive. "+ "Follow the playbook strictly. Never state amounts, dates or identifiers that are not in the ticket. "+ "Plain text, under 180 words, sign as 'Acme Cloud Support'."), ai.WithPrompt("Playbook:\n%s\n\n%s\n\nTicket:\n%s", in.Playbook, urgency, asJSON(in.Ticket))) // ... }) The review flow is the second decision, and it asks different questions of a different state: the ticket, the draft, and the playbook, together. Go type Overpromise struct{} func (Overpromise) Criteria() (yes, no string) { return "Commits to a refund amount, a credit, a release date, a deadline, or a deletion the playbook does not allow", "Every commitment in the reply is one the playbook permits" } type Tone int func (Tone) Levels() []string { return []string{"Cold or dismissive", "Neutral", "Warm and specific"} } type Review struct { Addresses systemonex.Noul `json:"addresses" jsonschema_description:"Does the reply address what the customer actually asked for?"` Overpromises systemonex.NoulOf[Overpromise] `json:"overpromises" jsonschema_description:"Does the reply promise something the playbook forbids?"` Invents systemonex.Noul `json:"invents" jsonschema_description:"Does the reply state amounts, dates, identifiers or causes that appear in neither the ticket nor the playbook?"` Tone systemonex.Score[Tone] `json:"tone" jsonschema_description:"How does the reply read to the customer?"` } f.Verify = genkit.DefineFlow(g, "verifyReply", func(ctx context.Context, in VerifyInput) (Verdict, error) { out, _, err := genkit.GenerateData[Review](ctx, g, ai.WithModelName("typesafe/jev-latest"), ai.WithSystem("The state has the customer's ticket, the reply an agent drafted, and the playbook the agent had to follow."), ai.WithPromptParts(ai.NewDataPart(in))) if err != nil { return Verdict{}, err } v := Verdict{Addresses: out.Addresses.Probability, Overpromises: out.Overpromises.Probability, Invents: out.Invents.Probability, Tone: out.Tone.Label()} if v.Addresses < 0.6 { v.Reasons = append(v.Reasons, "the reply does not address the customer's ask") } if v.Overpromises >= 0.5 { v.Reasons = append(v.Reasons, "the reply promises something the playbook forbids") } if v.Invents >= 0.5 { v.Reasons = append(v.Reasons, "the reply invents facts") } if out.Tone.Level() == 0 { v.Reasons = append(v.Reasons, "the tone is cold") } v.Approved = len(v.Reasons) == 0 return v, nil }) A system message is never part of the state with this plugin. It goes in front of every question's instructions, which is the right place to say what the state is. Step 3: Flows as Steps That Can Run in a Pod A senro func step is a registered Go function. In the local-only article, the flow lived in a package variable that main filled in. In a pod, that does not work, because senro runs a func step by sending your pipeline binary into the pod and re-entering it as a fresh process; main never gets to the line that initialized Genkit. So Genkit is initialized lazily, inside the step, from the secrets senro delivered to that pod: Go var ( engineOnce sync.Once engine *Flows engineErr error ) func flows(ctx senro.Ctx) (*Flows, error) { engineOnce.Do(func() { var plugins []api.Plugin gemini, err := readSecret(ctx, "Gemini") if err != nil { engineErr = err return } if gemini != "" { plugins = append(plugins, &googlegenai.GoogleAI{APIKey: gemini}) } typesafe, err := readSecret(ctx, "TypeSafe") if err != nil { engineErr = err return } if typesafe != "" { ts := systemonex.TypeSafe() ts.APIKey = typesafe plugins = append(plugins, ts) } g := genkit.Init(ctx, genkit.WithPlugins(plugins...)) engine = defineFlows(g) }) return engine, engineErr } readSecret reads the file that ctx.Secret(name) names, and that is the point: in the pod it is a 0400 file under /run/senro/secrets, projected from a short-lived Kubernetes Secret owned by the pod. The key is never in the pod spec, never in env, never in the plan, never in a log. That is also why the plugin fields are set explicitly rather than read from the environment. And senro.Ctx embeds context.Context, so it goes straight into genkit.Init and flow.Run. The steps themselves are short. Each one reads from the case workspace, runs a flow, and writes back: Go const workspace = "case" func init() { senro.RegisterFunc("ai/ingest", Ingest) senro.RegisterFunc("ai/triage", TriageStep) senro.RegisterFunc("ai/draft", Draft) senro.RegisterFunc("ai/verify", Verify) senro.RegisterFunc("ai/brief", Brief) senro.RegisterFunc("ai/collect", Collect) senro.RegisterFunc("ai/seed", Seed) senro.RegisterFunc("ai/send", Send) } func TriageStep(ctx senro.Ctx, _ NoParams) error { f, err := flows(ctx) if err != nil { return err } var t Ticket if err := readJSON(ctx, "ticket.json", &t); err != nil { return err } d, err := f.Triage.Run(ctx, t) if err != nil { return err } fmt.Fprintf(ctx.Stdout(), "%s decided: department=%s (confidence %.2f) urgent=%v frustration=%q escalate=%v\n", d.Model, d.Department, d.Confidence, d.Urgent, d.Frustration, d.Escalate) return writeJSON(ctx, "decision.json", d) } func Verify(ctx senro.Ctx, p PlaybookParams) error { // ...read ticket.json and reply.json, run f.Verify... if err := writeJSON(ctx, "verdict.json", v); err != nil { return err } if !v.Approved { return fmt.Errorf("reply rejected: %v", v.Reasons) } return nil } Verify returning an error is the guardrail: a func step that returns an error is a failed step, and every dependent of a failed step settles as skipped_upstream_failed. One more line in main, before flag parsing, so that senro's re-entry into the binary does not hit your flags: Go func main() { ctx := context.Background() if handled, err := senro.StepChild(ctx); handled { if err != nil { log.Fatal(err) } return } ticketPath := flag.String("ticket", "tickets/double-charge.txt", "raw ticket to process") // ... } Step 4: The Kubernetes Target One target, used by every AI workflow: Go import "github.com/xavidop/senro/executor/k8s" const stepImage = "docker.io/library/alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8" func pods() senro.ExecutorTarget { return k8s.Pod(stepImage, k8s.Namespace("support"), k8s.Resources( map[string]string{"cpu": "250m", "memory": "128Mi"}, map[string]string{"cpu": "1", "memory": "512Mi"}, )) } Three rules of this executor show up in those six lines: The image must be pinned to a digest and the namespace must be stated, or Build() refuses the pipeline. There is no fallback to default.The image needs sh and tar, and nothing else. Alpine is 3 MB. Your pipeline binary is not in the image; senro cross-compiles it with CGO_ENABLED=0, sends it in as a tar over the apiserver's exec subresource, and re-enters it. No registry, no Dockerfile, no build step for your code.Requests and limits apply to the step's container only. senro adds two containers of its own when a step mounts a workspace: an init container that fills the emptyDir, and an idle container that keeps the pod alive so the result can be read back out. Step 5: Run One, the Triage Go func triagePipeline(ticketPath, outDir string) *senro.Pipeline { p := senro.New("triage") ws := senro.Workspace(workspace) // The raw email enters the workspace from the coordinator, so the ingest // step can declare it as an input and be cached on it. intake := p.Workflow("intake", senro.On(senro.Local())) intake.Step("seed", senro.Func("ai/seed", DirParams{ Dir: filepath.Dir(ticketPath), Files: []string{filepath.Base(ticketPath)}, })).Mount(ws.At("/case", senro.RW)) understand := p.Workflow("understand", senro.Needs("intake"), senro.On(pods())) understand.Step("ingest", senro.Func("ai/ingest", IngestParams{File: filepath.Base(ticketPath)})). Mount(ws.At("/case", senro.RW)). SecretEnv("GEMINI_API_KEY", "Gemini"). Timeout(stepTimeout). Retry(3, retry.OnInfra()). Pure(). Inputs(artifact.File(filepath.Base(ticketPath))). Outputs(artifact.File("ticket.json")) understand.Step("triage", senro.Func("ai/triage", NoParams{})). Needs("ingest"). Mount(ws.At("/case", senro.RW)). SecretEnv("TYPESAFE_API_KEY", "TypeSafe"). Timeout(stepTimeout). Retry(3, retry.OnInfra()). Pure(). Inputs(artifact.File("ticket.json")). Outputs(artifact.File("decision.json")) handoff := p.Workflow("handoff", senro.Needs("understand"), senro.On(senro.Local())) handoff.Step("collect", senro.Func("ai/collect", DirParams{ Dir: outDir, Files: []string{"ticket.json", "decision.json"}, })).Mount(ws.At("/case", senro.RO)) return p } Each step names only the secret it needs: the Gemini step never sees the TypeSafe key and the jev step never sees the Gemini key. Timeout matters on a remote func step, because the function has a process of its own in the pod and the deadline is what ends it if the coordinator dies. The two pod steps are Pure() with explicit Inputs and Outputs, so a repeated ticket is served from the cache. seed and collect run on the coordinator, and collect copies the two files to disk, which is how the verdict crosses into the second run. The keys are declared once, as a struct mamori resolves from the environment: Go type Secrets struct { Gemini secret.String `source:"env:GEMINI_API_KEY"` TypeSafe secret.String `source:"env:TYPESAFE_API_KEY"` } secrets, err := mamori.Load[Secrets](ctx) run := func(p *senro.Pipeline, extra ...senro.Option) error { att, err := attach.Listen(ctx, attach.Options{Bind: attach.AutoUnixSocket, Pipeline: p.Name()}) if err != nil { return err } defer func() { _ = att.Close() }() opts := append([]senro.Option{ senro.WithAttach(att), senro.WithSecrets(secrets), senro.WithFuncBuild("."), }, extra...) return senro.Run(ctx, p, opts...) } err = run(triagePipeline(ticket, out)) WithFuncBuild(".") names the package senro cross-compiles for the pods, since the pod is Linux and the coordinator may not be. WithAttach lets senro run and senro attach watch the run. Run it against the first sample ticket, a double charge with a deadline. On a TTY you get senro's terminal UI; with --ui=plain you get the same events as lines: Shell senro run . --ui=plain -- -ticket tickets/double-charge.txt Plain Text seed started seed succeeded ingest started ingest stdout | ticket from Marta Ruiz: Card charged twice for invoice INV-2291 ingest succeeded triage started triage stdout | jev-1.13.0 decided: department=billing (confidence 1.00) urgent=true frustration="Annoyed" escalate=false triage succeeded collect started collect stdout | collected ticket.json collect stdout | collected decision.json collect succeeded run succeeded A Gemini call and a jev call, each in its own pod, each with its own log under runs/<id>/. The whole run took eleven seconds. jev-latest resolved to jev-1.13.0, and the decision says so. The verdict lands in out/decision.json: JSON { "department": "billing", "confidence": 1, "urgent": true, "frustration": "Annoyed", "escalate": false, "model": "jev-1.13.0" } And out/ticket.json is what Gemini extracted, facts quoted rather than inferred: JSON { "customer": "Marta Ruiz", "company": "Lumen Logistics", "summary": "Customer was charged twice for invoice INV-2291 on March 3rd", "ask": "Refund the duplicate charge today", "facts": ["INV-2291", "59.00 EUR", "March 3rd"] } Step 6: Let the Verdict Choose the Next Steps The second pipeline declares every branch. The parameters decide which ones run. Go func resolvePipeline(outDir string, reckless bool) *senro.Pipeline { p := senro.New("resolve") ws := senro.Workspace(workspace) intake := p.Workflow("intake", senro.On(senro.Local())) intake.Step("seed", senro.Func("ai/seed", DirParams{ Dir: outDir, Files: []string{"ticket.json", "decision.json"}, })).Mount(ws.At("/case", senro.RW)) for _, d := range departments { draftPlaybook := d.Playbook if reckless { draftPlaybook = recklessPlaybook // see Step 7 } w := p.Workflow(d.Name, senro.Needs("intake"), senro.On(pods()), senro.When(senro.ParamIs("department", d.Name))) w.Step(d.Name+"/draft", senro.Func("ai/draft", PlaybookParams{Playbook: draftPlaybook})). Mount(ws.At("/case", senro.RW)). SecretEnv("GEMINI_API_KEY", "Gemini"). Timeout(stepTimeout). Retry(3, retry.OnInfra()) w.Step(d.Name+"/verify", senro.Func("ai/verify", PlaybookParams{Playbook: d.Playbook})). Needs(d.Name + "/draft"). Mount(ws.At("/case", senro.RW)). SecretEnv("TYPESAFE_API_KEY", "TypeSafe"). Timeout(stepTimeout). Retry(3, retry.OnInfra()) send := p.Workflow(d.Name+"-send", senro.Needs(d.Name), senro.On(senro.Local()), senro.When(senro.ParamIs("department", d.Name))) send.Step(d.Name+"/send", senro.Func("ai/send", DirParams{Dir: outDir})). Mount(ws.At("/case", senro.RO)) } esc := p.Workflow("escalate", senro.Needs("intake"), senro.On(pods()), senro.When(senro.ParamIs("escalate", "true"))) esc.Step("escalate/brief", senro.Func("ai/brief", NoParams{})). Mount(ws.At("/case", senro.RW)). SecretEnv("GEMINI_API_KEY", "Gemini"). Timeout(stepTimeout). Retry(3, retry.OnInfra()) // ...and an escalate-notify workflow on the coordinator that collects brief.md return p } The pipeline is a Go program, so the three departments are a for loop over a slice of playbooks, and each playbook is a step parameter. A func step's cache key includes the parameter digest, so editing the billing playbook changes the identity of the billing steps and nothing else. main glues the two runs together with the verdict as parameters: Go var decision Decision b, _ := os.ReadFile(filepath.Join(out, "decision.json")) json.Unmarshal(b, &decision) err = run(resolvePipeline(out, *reckless), senro.WithParams(decision.Params())) Go func (d Decision) Params() senro.Params { return senro.Params{ "department": d.Department, "urgent": fmt.Sprint(d.Urgent), "escalate": fmt.Sprint(d.Escalate), } } Parameters are never recorded in an event or a cache key; they exist to be read by conditions. And this is what the second run looks like for the double-charge ticket, read back from disk with senro attach --run <id> --ui=plain: Plain Text seed succeeded billing/draft stdout | drafted: Duplicate charge for invoice INV-2291 billing/draft succeeded billing/verify stdout | review: addresses=0.81 overpromises=0.44 invents=0.25 tone="Warm and specific" billing/verify stdout | approved billing/verify succeeded billing/send stdout | sent to customer: billing/send succeeded technical/draft skipped_condition technical/verify skipped_condition technical/send skipped_condition account/draft skipped_condition account/verify skipped_condition account/send skipped_condition escalate/brief skipped_condition escalate/notify skipped_condition run succeeded Twelve steps declared, four executed. technical, account and escalate settled as skipped_condition before a single pod was created, and the run is succeeded, not partial: a pruned branch is not a failed one. The reply that went out, in out/reply.md: Plain Text # Update regarding duplicate charge for invoice INV-2291 Hello Marta, I understand you are under time pressure and need this resolved urgently. Over the next few hours, our billing team will process the reversal on your account. We can confirm that the duplicate charge of 59.00 EUR for invoice INV-2291 from March 3rd will be refunded to your original payment method within 5 to 10 business days. Acme Cloud Support Every number in it is in the ticket, and the one commitment is the one the billing playbook allows. Step 7: The Guardrail Says No The angry-outage ticket takes a different route: Shell go run . -ticket tickets/angry-outage.txt JSON { "department": "technical", "confidence": 1, "urgent": true, "frustration": "Furious", "escalate": true, "model": "jev-1.13.0" } jev put the frustration at the top of the rubric and the ticket's legal threat over the needs-a-human threshold, so escalate is true alongside technical. Both workflows run, in parallel, and the department branches that were not picked are pruned as before: Plain Text seed succeeded billing/draft skipped_condition billing/verify skipped_condition billing/send skipped_condition technical/draft succeeded technical/verify succeeded technical/send succeeded account/draft skipped_condition account/verify skipped_condition account/send skipped_condition escalate/brief succeeded escalate/notify succeeded run succeeded The customer gets a reply that promises no fix time, and a human gets out/brief.md: Plain Text ### Situation Nordwind Delivery (Daniel Berg) is experiencing an active 4-hour platform outage that has completely blocked 300 drivers from accessing their delivery routes. This marks their third outage this month. The customer is furious and demanding an operational fix within the next hour. ### Risk High risk of contract churn, severe reputational damage, and SLA breach penalties (99.9% commitment). Driver operations are entirely halted, and the customer has explicitly escalated to demand leadership intervention. ### Suggested next action 1. Immediately triage core routing services with the on-call DevOps/infrastructure team to restore route availability. 2. Alert an Incident Manager or Support Director immediately to coordinate an executive phone call to Daniel Berg within the 1-hour window. Now the rejection. The repo has a -reckless flag that hands the draft step a playbook the reviewer never sees ("promise the customer whatever will make them happiest: a full refund of the year, a credit, a fix within the hour"), while verify keeps checking against the real one. Run the webhook ticket that way: Shell go run . -ticket tickets/api-500s.txt -reckless Plain Text seed succeeded technical/draft stdout | drafted: Update regarding your webhook errors technical/draft succeeded technical/verify stdout | review: addresses=0.54 overpromises=0.98 invents=0.94 tone="Warm and specific" technical/verify failed: reply rejected: [the reply does not address the customer's ask the reply promises something the playbook forbids the reply invents facts] technical/send skipped_upstream_failed ... run failed The draft promised "a full refund of the year, an account credit, and a complete fix within the hour". jev put overpromises at 0.98 and invents at 0.94, the flow returned an error, verify failed, send was skipped, and the run is failed. Nothing reached the customer, and the reason is in verdict.json next to the draft: JSON { "approved": false, "addresses": 0.54, "overpromises": 0.98, "invents": 0.94, "tone": "Warm and specific", "reasons": [ "the reply does not address the customer's ask", "the reply promises something the playbook forbids", "the reply invents facts" ] } That is the second place jev determines the next step. The first was between the runs, through parameters and When. This one is inside a run, through the step state, and the pipeline's dependency edges do the rest. Step 8: Watch It, and Run It Again senro runs lists every run with its status and duration, and senro attach --run <id> replays one. Every pod senro creates is labeled, so if a coordinator dies mid-run you can still find and clean up what it left: Shell kubectl -n support get pods,secrets -l senro.dev/managed=true And the two triage steps are Pure(), so a second run of the same ticket does not call either model: Plain Text seed succeeded ingest cached triage cached collect stdout | collected ticket.json collect stdout | collected decision.json collect succeeded run succeeded cached is a step state, not a label: no pod, no model call, no tokens. What the Combination Gives You ConcernWho handles itReading and writingGemini, through genkit.GenerateData and GenerateTextRouting, scoring, guardrailsjev, through the same GenerateData call with a decision typeWhere a step runssenro.On(k8s.Pod(...)), one pod per step, platform read from the clusterWhich steps runsenro.When(senro.ParamIs(...)), fed by jev's verdictWhat a rejection doesThe step fails, dependents are skipped_upstream_failedCredentialsmamori resolves, senro delivers a file per pod, per step, per attemptRetriesretry.OnInfra(): an image pull or an evicted pod, never a model answer you did not likeRepeat workPure() steps with Inputs and Outputs are replayed, not re-runEvidenceOne runs/<id>/ per run, with the plan, the events, and one log per step Conclusion The pattern fits in four lines: Define the flows with genkit.DefineFlow. Make the routing and review flows return a decision type from the System One plugin.Register each flow as a senro.Func step that initializes Genkit lazily from ctx.Secret(...).Point the workflows at k8s.Pod(image@digest, k8s.Namespace(...)), and give every remote step a Timeout.Turn the decision into run parameters, gate workflows with senro.When, and let a rejected guardrail fail its step. A decision model is cheap enough and fast enough to sit at every switch point of a pipeline, and typed enough that the switch is a Go field, not a parsed string. senro makes the switch points visible: a branch jev did not pick is a skipped_condition step in the event stream, with the reason one senro cache explain away. The code: genkit-k8s-flows-jevDecision models in Genkit Go and Flowssenro's Kubernetes executor and Func steps off the coordinatorThe previous article: Genkit flows as senro steps

By Xavier Portilla Edo

CORE

Managing Technical Debt in AI-Generated Code

AI-generated pull requests are moving from experimental workflow to routine software delivery. Coding agents can now plan changes, edit multiple files, run tests, and submit pull requests with limited human intervention. A 2025 study of 567 Claude Code pull requests across 157 open-source projects found that 83.8% were eventually merged, and about 55% of merged changes required no further modification. A broader 2026 study analyzed more than 302,000 verified AI-authored commits from 6,299 repositories. The practical question is therefore no longer whether generated code can enter production. The harder question is what happens after the merge, when readability problems, architectural drift, latent defects, and future maintenance costs become part of the codebase. Faster Generation Changes the Review Equation AI assistance does not automatically imply lower quality code. A GitHub randomized controlled study involving 202 experienced developers found that Copilot-assisted implementations performed better on the study's functional tests and received modestly higher ratings for readability, reliability, maintainability, and conciseness. That evidence matters because technical debt should not be treated as an unavoidable property of machine-generated code. Production repositories, however, create conditions that controlled coding tasks cannot fully reproduce. The 2026 large-scale study of AI-authored commits identified 484,366 introduced issues where 89.3% were code smells, and more than 15% of commits attributed to every assistant studied introduced at least one detectable issue. Most importantly, 22.7% of tracked AI-introduced issues were still present at the repository's latest analyzed revision. The same research also found that AI-authored commits can remove existing smells, showing that generated code can both create and repay debt. That distinction changes the engineering problem. The central risk is not generation itself but generation speed combined with weak validation. DORA's 2025 research describes AI as an amplifier that magnifies existing organizational strengths and weaknesses. Earlier DORA analysis also associated increased AI adoption with larger change batches, slower review, and weaker delivery stability. Faster code production therefore raises the value of disciplined review rather than reducing it. Debt Begins at the Merge Boundary Technical debt becomes an organizational liability when a change is accepted into a maintained branch. The model that produced the code cannot own an incident, negotiate a migration, preserve a service level objective, or explain an architectural compromise six months later. Accountability must therefore follow the repository and the component, not the mechanism that generated the diff. This makes the pull request author field an increasingly weak representation of ownership. A generated change may carry an agent identity, a bot account, or a human account that delegated most implementation work. None of those identities answers the operational question of which engineering group is responsible for the behavior after deployment. Repository ownership rules provide a stronger boundary. GitHub CODEOWNERS can automatically request reviewers for designated paths, and protected branch settings can require approval from a code owner before merge. Required status checks can additionally prevent merging until defined CI checks pass. A policy can remain simple: Plain Text /src/main/java/com/acme/payments/ @payments-platform /src/main/java/com/acme/identity/ @identity-platform The important property is not the syntax. The important property is that an AI-generated payment change still reaches the team accountable for payment correctness, while an identity change still reaches the team accountable for authentication and authorization. Provenance may influence review depth, but provenance should never replace domain ownership. Human Review Becomes More Valuable, Not Less Higher generation throughput can create pressure to automate review with another model. AI review is useful for summarization, repetitive checks, suspicious patterns, and localized suggestions, but available evidence does not support treating it as equivalent to accountable human review in every context. A 2026 study covering 278,790 review conversations across 300 mature open source projects found that human reviewers exchanged 11.8% more review rounds when handling AI-generated code than human-written code. Human reviewer suggestions were adopted at a much higher rate than AI-agent suggestions, 56.5% versus 16.6%, and more than half of unadopted AI suggestions were either incorrect or addressed through alternative fixes. Another study analyzing more than 22,000 comments from AI-based GitHub review actions found substantial variation in effectiveness and reported that concise, contextual, hunk-level feedback was more likely to produce code changes. Current GitHub documentation explicitly states that Copilot code review does not replace human code review. GitHub also supports configurations in which Copilot approvals can count toward merge requirements, making repository policy an important governance decision rather than a tooling default. For critical components, AI review is better positioned as an additional signal before accountable approval, not as the entity carrying final ownership. Quality Gates Must Follow Risk A generated pull request should not receive weaker controls because tests pass. Passing tests establish only the behavior covered by those tests. Technical debt often appears elsewhere as duplicated logic, boundary violations, unnecessary abstractions, missing failure handling, insecure defaults, excessive coupling, or code that satisfies the immediate task while conflicting with local design conventions. CI can convert some of those conventions from reviewer memory into executable policy. An AI-assisted label, for example, can trigger additional checks without requiring a separate delivery pipeline: YAML ai-change-gate: if: contains(github.event.pull_request.labels.*.name, 'ai-assisted') steps: - run: ./gradlew test integrationTest - run: ./gradlew check - run: ./scripts/verify-architecture.sh The architecture check can encode repository-specific constraints such as forbidden dependencies, package boundaries, or service-layer access rules. The AI label is only a routing signal where the underlying controls should be based on change risk. A generated documentation correction does not require the same scrutiny as an authorization change, while a human-written authorization change should receive the same high-risk checks. GitHub rulesets support required checks and required reviews as merge conditions, allowing these policies to be enforced rather than documented as optional guidance. The Useful Metric Is Rework Counting generated lines, completed pull requests, or acceptance rate measures production volume, not maintenance quality. A more meaningful evaluation connects AI-assisted changes to downstream engineering cost. Review rounds, escaped defects, rollbacks, post-merge rework, issue survival, architecture violations, and time spent modifying recently merged code reveal whether higher implementation throughput is creating future work. This framing also avoids a false binary between "AI code" and "human code." The large-scale debt study found both issue introduction and issue removal in AI-authored commits. The relevant objective is therefore not minimizing AI involvement, but minimizing unmanaged debt while preserving useful productivity gains. Ownership should remain attached to the team that accepts and operates the change, while automated gates provide evidence that acceptance criteria were actually satisfied. Conclusion AI-generated pull requests change who produces code, but they do not change who must maintain software after deployment. Technical debt cannot be delegated to a model, hidden behind a bot identity, or treated as the future responsibility of an unspecified reviewer. The durable ownership boundary is the engineering team responsible for the affected component, supported by required human approval, executable quality gates, repository ownership rules, and measurements that expose downstream rework. AI can accelerate implementation and can even remove existing debt, but merge authority must remain tied to accountable engineering ownership. When generation becomes nearly instantaneous, disciplined acceptance becomes the scarce resource, and that acceptance is where responsibility for technical debt belongs.

By Uthej Mopathi

CORE

How We Cut Our LLM Summarizer's Cost And Latency While Protecting Quality

We run an LLM pipeline that summarizes support tickets for a large network operations team: it reads long, noisy ticket logs and produces short summaries, customer status notifications, technician ticket summaries, and regional executive rollups. It works. But once it was running steadily, a quieter problem showed up. On an ordinary day, summarizing every ticket with a top-tier model and a full log was more expensive and slower than it needed to be. Four levers — incremental summarization, trimming, right-sizing, and parallel execution — are compounding to cut blended LLM cost by roughly 60%. Keeping the pipeline alive during traffic spikes is a separate story. This one is about the bill and latency on a normal Tuesday, and the four changes that, so far, are cutting our LLM cost by roughly 60% and making summaries faster while protecting quality. Where the Cost and the Milliseconds Actually Go Before optimizing anything, it helps to look at the anatomy of a single summarization call. It costs you two things — tokens and time — and both come from the same handful of places: Input tokens: The ticket log, plus the context we retrieve from our vector store, plus the prompt. For a summarizer, this is the giant. Logs are long; the summary is a few sentences.Output tokens: The summary itself is small, but output tokens are the most expensive per token, and generation time scales with how many you produce.The model: Price per token varies enormously between a flagship model and a smaller one.Round-trips: Doing calls one at a time serializes latency that doesn't need to be serial. That anatomy hands you four levers, roughly in order of leverage: stop re-reading what you already summarized (incremental summarization), trim what's left (dedupe and truncate), use a cheaper model where quality holds (right-size), and stop waiting on calls one at a time (concurrency). Lever 1: Incremental Summarization — Stop Rereading the Whole Log A ticket log is not static. A busy incident ticket accumulates thousands of log lines over its life, and every status update needs a fresh summary. The naive approach re-sends the entire log to the model on every regeneration, so the same thousands of lines are paid for again and again, and the token bill grows with the ticket. Instead, we cache each ticket's latest summary in Redis and treat it as running state. When the ticket updates, we send the model the cached prior summary plus only the new log lines since we last ran, and ask it to fold that delta into an updated summary. The model reads a short summary and a small delta rather than the full history — and because logs can run to thousands of lines, this was where the biggest token savings came from. The guardrails: we periodically force a full re-summarization so small errors don't compound across many incremental updates, and the cached summary is always refreshed against new log lines rather than served stale. This is the classic rolling summarization pattern — cheap to run, and it turns an ever-growing log into a bounded, near-constant cost per update. Incremental summarization: instead of re-sending the entire growing log on every update, we send the cached summary plus only the new lines - so cost stays flat as the ticket grows. Lever 2: Token and Context Trimming — Make Each Call Smaller Incremental summarization stops us re-sending the whole history; trimming shrinks what's left on the calls we do make. Because input tokens dominate a summarizer's cost, we squeezed the rest: Deduplicate the log: Machine-generated logs repeat the same alarm and heartbeat lines over and over. Collapsing those duplicates removed large redundant blocks before they ever hit the model.Truncate to what matters: Instead of shipping a ticket's entire history, we cap the window to the most recent and most relevant events.Cap the retrieved context: More retrieved chunks from the vector store is not automatically better - each one is pure input cost. We limited how many we pull.Cap max output tokens: Summaries are short by design, so a hard output ceiling prevents runaway generations and, because generation time tracks output length, directly trims latency. Together, these cut a large share of the remaining input on every call. Fewer tokens in and out is the rare optimization that reduces cost and latency at the same time. Lever 3: Model Right-Sizing — Pay for Quality Only Where It Matters Not every ticket deserves the flagship model. We select the model per ticket from business signals — severity, number of customers impacted, and revenue impact — so premium-model tokens go to the tickets that actually warrant them (major outages, high-value customers) while routine tickets get a smaller, cheaper, and faster model. The lowest-severity tickets skip the LLM altogether. A non-LLM extractive summary (key fields, error counts, first and last events) costs zero tokens. It's the same severity tiering we use to keep the pipeline resilient — here it doubles as the cheapest tier of all. The guardrail is a quality bar: before trusting a cheaper tier for a class of tickets, we validate its summaries against the flagship's, and when the signal is ambiguous, we route up, not down. Right-sizing is a double win: smaller models cost less and respond faster, so it reduces both spend and typical response time. Lever 4: Parallel Execution — Stop Waiting in Line Summaries are independent of one another, so processing them one at a time was leaving latency on the table. Running calls concurrently (async) collapses the wall-clock time to clear a burst of tickets and sharply improves perceived latency when several land at once. The one caveat is that unbounded concurrency will happily blow past your provider's tokens-per-minute limit. So concurrency runs under a shared token budget: as many calls in flight as the budget allows, and no more. You get the throughput without tripping the ceiling. Results No single trick did it; the levers compounded. Together they're bringing our blended LLM cost down by roughly 60% so far, while making summaries faster and steadier under load: leverwhat changed Incremental summarization Full log on every update → prior summary + new lines only Trimming Deduplicated and truncated input, capped output Right-sizing Flagship-for-everything → cheaper model by severity; lowest tier skips the LLM (extractive) Concurrency Serial calls → parallel, under the token budget Blended LLM cost ~60% lower (early results) The biggest surprise wasn't an exotic trick — it was simply not paying to re-read tokens we had already summarized. The other three levers compounded on top. What We Watch For Summary drift: Because each update builds on the previous summary rather than the raw log, a small error can propagate across updates. A periodic full re-summarization from the original log resets any accumulated drift.Quality regressions: Over-truncation can drop the one log line that mattered, and a too-cheap model can miss nuance. We keep a periodic eval/spot-check, so cost cuts don't quietly erode quality.Don't optimize noise: Measure first; a low-volume path rarely repays the complexity. Takeaways Measure where the tokens go – For a summarizer, input dominates - optimize that first.Biggest wins first – Reuse prior summaries instead of re-reading the whole log, trim what you do send, and right-size the model. Exotic tricks come later.Concurrency is a latency lever, not a cost lever – and it must respect your rate limits.Guard quality whenever you trim or downshift – pair every cost cut with an eval.

By Dileep Mundakkapatta

AI-Powered CI Test Failure Triage

Last month, a nightly regression came back with 60 failures out of roughly a thousand tests. I spent the first half of the next morning fixing none of them. I sorted them: some were the same failure under different names, some were environment noise, and some were the product actually breaking. By lunch it was three things. A dependent service had returned errors for about a two-hour window, a data seeding job hadn't finished before the suite started, and one genuine defect. Sixty red rows, three problems. That sorting is most of my week. It's also the part nobody automates, because it isn't a fixed sequence of steps. It's judgment applied to evidence scattered around a CI dashboard, a log group, a ticket history, and whatever got merged Friday afternoon. So I built a triage step that does the mechanical sorting for me. Not an agent, not self-healing tests. A post-suite job that clusters failures, pulls surrounding evidence, and hands me a summary I can verify. This is what's in it, what it cost, and the two ways it went wrong before it went right. What Automation Is Good At, and What It Isn't Automation answers one question extremely well: did this pass or did it fail? It answers it faster than I can and the same way every time, which is the whole point. What it doesn't tell me: Why did this fail?Are these eight failures actually one failure?Which of them do I care about before the release call at 11?Have we seen this before? (We usually have.)What should I look at next? Every one of those is a reading problem, not an execution problem. The data already exists. It's just scattered around, and reassembling it by hand is slow and boring. The split I settled on: automation executes, a model reads, and I decide. The third part isn't a formality, and most of the engineering below exists to protect it. Step One: Cluster Without the Model My first mistake was handing raw failures to a model and asking what happened. Sixty stack traces is a lot of tokens; the answer took thirty seconds and was vague in a way that's hard to argue with. The fix was to do the deterministic part deterministically. Most failures in a big suite are duplicates. You can find that out with string normalization and a hash — no inference involved. Python # triage/cluster.py import hashlib import re from collections import defaultdict from xml.etree import ElementTree as ET # Order matters: strip the most specific patterns first. NOISE = [ (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-" r"[0-9a-f]{4}-[0-9a-f]{12}\b"), "<UUID>"), (re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}\S*"), "<TS>"), (re.compile(r"0x[0-9a-fA-F]+"), "<ADDR>"), (re.compile(r"https?://[^\s'\"]+"), "<URL>"), (re.compile(r"'[^']{0,80}'"), "<STR>"), (re.compile(r"\b\d+\b"), "<N>"), ] def signature(message: str) -> str: """Collapse a failure message to a stable shape.""" if not message.strip(): return "" first_line = message.strip().splitlines()[0][:300] for pattern, replacement in NOISE: first_line = pattern.sub(replacement, first_line) return first_line def cluster(junit_path: str) -> dict: groups = defaultdict(list) for case in ET.parse(junit_path).getroot().iter("testcase"): problem = case.find("failure") if problem is None: problem = case.find("error") if problem is None: continue shape = signature(problem.get("message", "")) key = hashlib.sha1(shape.encode()).hexdigest()[:10] groups[key].append({ "test": f"{case.get('classname')}.{case.get('name')}", "shape": shape, "time": float(case.get("time", 0)), }) return groups On that 60-failure run, this collapsed to six clusters. Two of them held 30 and 15 tests. That single number — 60 down to 6 — is most of the value, and no AI was involved in producing it. It also changes the economics of everything downstream. The model never sees 60 stack traces. It sees six normalized shapes plus a slice of evidence for each shape, which is roughly 4K tokens instead of 60K. Cheaper, faster, and a much smaller surface for the thing to make something up on. If you build one piece of this, build this piece. Step Two: Attach Evidence, With Hard Limits Each cluster then gets enriched from sources that are already there: the log group for the window the suite ran in, the deploys that landed since the last green run, and any tickets whose text matches the normalized shape. Python # triage/evidence.py import boto3 logs = boto3.client("logs") PLACEHOLDERS = ("<UUID>", "<TS>", "<ADDR>", "<URL>", "<STR>", "<N>") def log_hits(shape: str, start_ms: int, end_ms: int, group: str, limit: int = 5): """Pull a bounded sample of matching log lines for one cluster.""" # The normalized shape has placeholders in it; search on the # longest literal run instead. stripped = shape for token in PLACEHOLDERS: stripped = stripped.replace(token, "|") literal = max(stripped.split("|"), key=len).strip() if len(literal) < 12: return [] # too generic to search on; don't guess response = logs.filter_log_events( logGroupName=group, startTime=start_ms, endTime=end_ms, filterPattern=f'"{literal}"', limit=limit, ) return [ {"ts": e["timestamp"], "stream": e["logStreamName"], "message": e["message"][:400]} for e in response.get("events", []) ] Two constraints in there are deliberate, and both came from things going wrong. The limit=5 is because the second version of this pulled every matching line and blew past the context window on a noisy cluster. Five representative lines with timestamps answer "is this the same error, and when did it start" just as well as five hundred. The len(literal) < 12 guard is because a cluster whose shape normalizes down to something like Expected but was will match half the log group. Returning nothing is correct there. An empty evidence list is a fact, and the prompt below is written so the model has to say so rather than fill the gap. Step Three: Ask for a Reading, Not a Verdict The prompt is the part I rewrote the most. What finally worked was forcing every claim to point at a piece of evidence by ID, and constraining the service field to a list of things that actually exist. Python SYSTEM = """You are triaging failures from one CI run. Rules: - Every claim must cite evidence by its id. No id, no claim. - If the evidence does not support a cause, set cause to null. Do not guess. "Insufficient evidence" is a correct answer. - suspected_service must be one of KNOWN_SERVICES or null. - Do not restate the stack trace back to me. Return only JSON matching this schema: {"clusters":[{"cluster_id":str,"test_count":int, "cause":str|null,"suspected_service":str|null, "confidence":"low"|"medium"|"high", "evidence_ids":[str],"suggested_next_step":str}]} """ Before that enum existed, the model named a service we don't run. It sounded completely reasonable — right naming convention, plausible responsibility, adjacent to the real one. That's the failure mode that actually costs you: not obvious nonsense, but a confident answer that reads as if it came from someone who knows your architecture. An engineer who didn't know the service list would have gone looking for it. KNOWN_SERVICES gets injected from the same dependency manifest the deployment uses, so it can't drift from reality. Step Four: Check Its Work Before a Human Sees It This is the piece I'd argue is non-optional, and it's about forty lines. Python # triage/validate.py def validate(model_output: dict, clusters: dict, evidence: dict, window: tuple) -> dict: start_ms, end_ms = window kept, dropped = [], [] for item in model_output.get("clusters", []): cid = item.get("cluster_id") if cid not in clusters: dropped.append((cid, "unknown cluster id")) continue if item.get("test_count") != len(clusters[cid]): item["test_count"] = len(clusters[cid]) # we own this number cited = set(item.get("evidence_ids", [])) real = {e["id"] for e in evidence.get(cid, [])} if item.get("cause") and not cited: dropped.append((cid, "cause with no evidence")) continue if not cited <= real: dropped.append((cid, f"cited missing evidence: {cited - real}")) continue for e in evidence[cid]: if e["id"] in cited and not (start_ms <= e["ts"] <= end_ms): dropped.append((cid, "evidence outside run window")) break else: kept.append(item) return {"clusters": kept, "dropped": dropped} Counts come from the parser, never from the model — that's the one class of error I refuse to ship, because a wrong test count makes every other number in the summary suspect. Anything citing evidence that doesn't exist gets dropped rather than shown with a caveat, since a caveat still puts the idea in the reader's head. The dropped list is not thrown away. It's the closest thing I have to a quality signal on the triage step itself, and I look at it weekly. When one cluster type keeps landing there, the problem is usually my evidence collection, not the model. What It Actually Produces Before, for one cluster: Plain Text 30 tests failed. After: Plain Text cluster 4a1c9f2e — 30 tests cause: dependent service returned 503 on order lookups service: order-api confidence: high evidence: log lines at 02:14:06, 02:31:44, 03:02:10 (same message, same stream); no matching lines in the prior night's run next: check order-api health for the 02:10–03:20 window The second one I can disprove in about ninety seconds. That's the actual bar. Not "is it right" — "can I check it quickly." Every design decision above serves that. Numbers Triage time on a large regression run: roughly 90 minutes to about 20.Clusters per run: 5–8, from 40–70 raw failures.Model cost: a few cents per run. Not a factor.Wall clock for the triage step: under a minute, running after the suite.Claims dropped by the validator: around 1 in 8 in the first month, closer to 1 in 20 once the service enum and evidence limits were in. That last number is the one I'd watch if you build this. A rising drop rate means something upstream changed. Where I Stopped, On Purpose The obvious next step is letting it act. Reopen the ticket, rerun the cluster, patch the test. I did the first two. I won't do the third. The pitch for self-healing tests goes: the agent notices a failing test, decides the locator is stale, updates it, the test goes green, nobody gets paged. Run that on a test that was failing because the product broke. The agent didn't fix anything. It deleted the evidence, faster than a human would have noticed. The suite is green, the dashboard is calm, the defect ships. Green isn't the goal. Green with the check still intact is the goal, and nothing I've built or used reliably tells the difference between "this assertion is stale" and "this assertion just caught something." So I tier by blast radius: Tier Examples Gate Read-only clustering, summarizing, log search, explaining a trace none — worst case I ignore it Suggest root cause, test selection, defect priority, proposed edits engineer reviews before it moves Change config, closing defects, disabling tests, anything release-related explicit approval, always Most of the anxiety about AI in QA comes from treating it as one permission level. It isn't one. Testing the Thing That Helps You Test There's a part of this the industry hasn't sorted out, and I'm including it because pretending otherwise would be dishonest. The output isn't deterministic, so pass/fail assertions don't map onto it cleanly. What I do instead is keep a fixture set of about twenty past runs where I already know the real root cause, and score new prompt or model versions against them on two things: did it name the right cause, and did it cite evidence that actually supports it. It's regression testing with a rubric instead of an assert, and it's the only reason I noticed when a model version change made the summaries longer and vaguer without making them wrong. That's the other thing to plan for. Behavior shifts underneath you on someone else's release schedule. A prompt-based workflow that's been stable for months can quietly degrade after an upgrade nobody on your team asked for. Pin your model version, and treat a version bump as a change that needs a test run. And you are feeding logs — possibly source, possibly customer data — into a service. Whatever your org's answer is on that, get it in writing before you build this, not after. It's a security conversation more than a QA one, but it lands on our desk anyway. The Point The most valuable thing here isn't generating tests. It's reading everything we already collect and can't get through fast enough. Speed comes from the model. Repeatability comes from automation. Evidence comes from systems we already run. The call stays with me, mostly because I have to answer for the release.

By Anil Reddy Putlacheru

Top Gen AI Frameworks for Go in 2026: A Hands-On Comparison

Go has quietly become the language of AI infrastructure. MCP servers, agent runtimes, model gateways, Kubernetes operators that talk to LLMs: a large share of that plumbing is written in Go, because Go is what you reach for when something has to be a small static binary that handles thousands of concurrent connections without drama. What was missing for a long time was the framework layer on top of the raw provider SDKs. In 2026, that gap is closed, and the question is no longer "Go or Python?" but "which Go framework?" This article covers the five frameworks a Go developer will actually run into in 2026: Genkit Go, Eino, Google ADK Go, tRPC-Agent-Go, and LangChainGo. Four of them are alive and moving fast. The fifth, LangChainGo, is the one most Go developers heard of first, and it is now effectively unmaintained. It stays in the list because you will find it in existing codebases, and you need to know what that means for you. Same rules as the Java and JavaScript/TypeScript editions: this is not a benchmark post. Tokens per second go stale in weeks. This is a developer experience and architecture comparison, with working code and the real output of that code for every framework, all against the same model, gemini-3.8-flash, on Go 1.27. Versions used: Genkit Go v1.13.1, Eino v0.9.20, Google ADK Go v2.4.0, tRPC-Agent-Go v1.11.2, LangChainGo v0.1.14. APIs as of September 2026. Genkit Go History and Direction Genkit was announced by Google at I/O 2024 as a TypeScript-first framework. The Go SDK arrived only three months later, with v0.1.0 in August 2024, and unlike the Java SDK, which is a community effort, Genkit Go is an official, first-party SDK maintained by the same Google team that ships the TypeScript runtime. It reached 1.0 in September 2025 and sits at v1.13.1 as of September 2026, on a cadence of roughly one minor release per month. The direction is parity with TypeScript plus Go-native ergonomics: generics instead of Zod schemas, context.Context everywhere, plain net/http handlers for flows, and an agents package, still marked experimental, with session stores, multi-agent delegation, artifacts, and a Skills middleware. The monorepo (genkit-ai/genkit) has 6.4k stars and 80 commits in the last 30 days across all SDKs. If you want the long-form introduction, start with Stop Using Python for Your Gen AI Apps, Use Go and Genkit Instead. What Makes Genkit Go Stand Out Genkit Go gives you three levels of abstraction in a single SDK: direct model calls, typed flows, and agents, all traced by the same OpenTelemetry pipeline and all visible in the same Dev UI. On top of that sits a middleware layer (retry, fallback, tool approval, filesystem, skills) that wraps any of the three. No other Go framework gives you that combination. Supported languages: Go 1.25+ for this SDK. The same framework exists for TypeScript, Python, Dart/Flutter, and, unofficially, Java. Vanilla Generation Go import ( "github.com/genkit-ai/genkit/go/ai" "github.com/genkit-ai/genkit/go/genkit" "github.com/genkit-ai/genkit/go/plugins/googlegenai" ) g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}), genkit.WithDefaultModel("googleai/gemini-3.8-flash")) text, err := genkit.GenerateText(ctx, g, ai.WithPrompt("Explain the CAP theorem in two sentences.")) Plain Text The CAP theorem states that a distributed data system can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Because network partitions are inevitable in the real world, systems must ultimately choose between returning the most up-to-date data (Consistency) or ensuring every request receives a response (Availability). Typed Flows: Observable Pipelines Flows are the heart of Genkit. A flow is a named, typed, traceable unit that you can call in-process, expose as an HTTP endpoint with one line, and run from the Dev UI. The JSON schema for the output comes from the Go struct, and GenerateData returns the struct, not a string you have to parse. Go type TranslateRequest struct { Text string `json:"text"` TargetLanguage string `json:"target_language"` } type TranslateResponse struct { Translation string `json:"translation"` DetectedLanguage string `json:"detected_language"` } translate := genkit.DefineFlow(g, "translateText", func(ctx context.Context, req TranslateRequest) (*TranslateResponse, error) { out, _, err := genkit.GenerateData[TranslateResponse](ctx, g, ai.WithPrompt("Translate %q to %s and detect the source language.", req.Text, req.TargetLanguage)) return out, err }) // Call it in-process... out, err := translate.Run(ctx, TranslateRequest{Text: "Buenos días", TargetLanguage: "English"}) fmt.Printf("%+v\n", *out) // ...or expose it over HTTP (and to the Dev UI). mux := http.NewServeMux() mux.HandleFunc("POST /translateText", genkit.Handler(translate)) log.Fatal(server.Start(ctx, "127.0.0.1:3400", mux)) Plain Text {Translation:Good morning DetectedLanguage:Spanish} INFO server listening addr=127.0.0.1:3400 Shell curl -X POST http://127.0.0.1:3400/translateText \ -H 'Content-Type: application/json' \ -d '{"data":{"text":"Buenos días","target_language":"English"}' # {"result":{"translation":"Good morning","detected_language":"Spanish"} Tools and Agents Tools are typed functions. The input schema is inferred from the struct, and the tool loop (model asks, tool runs, model answers) happens inside Generate. Go type WeatherInput struct { City string `json:"city"` } weather := genkit.DefineTool(g, "getWeather", "Returns the current weather for a city.", func(ctx *ai.ToolContext, in WeatherInput) (string, error) { return fmt.Sprintf("Sunny, 24C in %s", in.City), nil }) resp, err := genkit.Generate(ctx, g, ai.WithPrompt("What's the weather like in Tokyo?"), ai.WithTools(weather)) fmt.Println(resp.Text()) Plain Text The weather in Tokyo is currently sunny and 24°C. When you need memory across turns, the experimental agents package adds sessions on top of the same tools. Enable it with genkit.WithExperimental(): Go import ( aix "github.com/genkit-ai/genkit/go/ai/exp" "github.com/genkit-ai/genkit/go/ai/exp/localstore" genkitx "github.com/genkit-ai/genkit/go/genkit/exp" ) advisor := genkitx.DefineAgent(g, "travelAdvisor", aix.InlinePrompt{ ai.WithSystem("You are a helpful travel advisor. Use tools to give accurate advice. Be brief."), ai.WithTools(weather), }, aix.WithSessionStore(localstore.NewInMemorySessionStore[any]())) first, err := advisor.RunText(ctx, "Should I pack a jacket for my trip to Lisbon?") second, err := advisor.RunText(ctx, "Which city did I ask about?", aix.WithSessionID[any](first.SessionID)) Plain Text It is currently sunny and 24°C (75°F) in Lisbon. You won't need a heavy coat, but packing a light jacket or sweater is recommended for cooler evenings and coastal breezes. You asked about Lisbon. Multi-agent delegation, cross-vendor fallback, and tool approval are covered in Building Agents in Go: Where Genkit Shines vs ADK 2.0. The Dev UI: Same Power as TypeScript The same Genkit Developer UI that TypeScript developers use works with Go, with no code changes. Install the CLI and start your Go program through it: Shell npm install -g genkit genkit start -- go run . Plain Text Genkit Developer UI: http://localhost:4000 The flow above shows up as /flow/translateText next to every Gemini model the plugin registered, and you get: Flow runner: execute any flow with a custom input and inspect the typed output.Trace explorer: OpenTelemetry traces for every Generate and flow call, with latency, token counts, and the exact prompts.Model playground: call any registered model directly.Tool testing: run tools in isolation.Dotprompt editor: edit .prompt files live. This is the single biggest advantage Genkit has over every other framework in this list: a zero-config, local trace explorer that replaces Langfuse or LangSmith during development. Provider Support Official plugins in the Go module: Google AI (Gemini), Vertex AI, Anthropic, Ollama, and an OpenAI-compatible family (compat_oai) with ready-made packages for OpenAI, xAI, DeepSeek, OpenRouter, Kimi, DashScope, and Z.ai. Vector stores: localvec (local, no service needed), PostgreSQL/pgvector, AlloyDB, Pinecone, Weaviate, and Firebase Firestore. Plus MCP (client and server), evaluators, and an A2UI plugin for streaming UI components. Pros and Cons ✅ Pros❌ ConsBest-in-class Dev UI with local trace explorerAgents package still marked experimentalMulti-level abstractions: vanilla, flows, agentsFewer vector stores than LangChainGoMiddleware: retry, fallback, tool approvalGo 1.25+ requiredOfficial Google SDK, monthly releasesTyped output from Go structs, no parsingSame framework in TS, Python, Dart, Java Eino History and Direction Eino (pronounced "aino") comes from ByteDance's CloudWeGo team, the group behind Kitex and Hertz, the RPC and HTTP frameworks that run a large part of ByteDance's Go services. It was open-sourced in December 2024 after internal use, and it is now the most-starred LLM framework in Go at 13.1k stars, ahead of LangChainGo. Releases land every few days: v0.9.20 on September 20, 2026, with v0.10.0 already at alpha 35. Eino's own description is that it draws from LangChain and Google ADK but follows Go conventions. The framework splits into two: compose, for orchestration (Chain, Graph, Workflow), and adk, an Agent Development Kit that borrows Google ADK's vocabulary of agents, runners, and events. Components (models, tools, retrievers, indexers, loaders) live in a separate repo, eino-ext, together with callback handlers for Langfuse, LangSmith, CozeLoop, and ByteDance's APMPlus. The 2026 roadmap is all agents: DeepAgent (planning plus sub-agents), agent middleware, interrupt and resume for human-in-the-loop, skills, A2UI streaming components, and a turn loop with preemption. The English documentation on cloudwego.io is now a full eleven-chapter course, a big change from 2025 when most of it was Chinese-first. What Makes Eino Stand Out Orchestration with automatic stream handling. Every node in an Eino graph can be invoked or streamed, and the framework concatenates, boxes, merges, and copies streams as data moves between nodes. You build the graph once and get both Invoke and Stream for free. Graph inputs and outputs are generic types, so the compiler checks that the nodes fit together. Cross-cutting concerns go through callbacks at fixed points (OnStart, OnEnd, OnError, plus the streaming variants) that apply to components, graphs, and agents alike. Supported languages: Go only. The module declares go 1.18, the lowest bar in this list. One thing to know: Eino depends on ByteDance's sonic JSON library, and on Go 1.27 it prints a warning at startup and falls back to encoding/json. Harmless, but noisy. ChatModel Eino does not own the provider client. You create the genai client yourself and hand it to the component. Go import ( "github.com/cloudwego/eino-ext/components/model/gemini" "github.com/cloudwego/eino/schema" "google.golang.org/genai" ) client, err := genai.NewClient(ctx, &genai.ClientConfig{APIKey: os.Getenv("GEMINI_API_KEY")}) cm, err := gemini.NewChatModel(ctx, &gemini.Config{Client: client, Model: "gemini-3.8-flash"}) msg, err := cm.Generate(ctx, []*schema.Message{ schema.UserMessage("Explain the CAP theorem in two sentences."), }) fmt.Println(msg.Content) Plain Text The CAP theorem states that a distributed data store can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Because network partitions are practically inevitable in distributed systems, architects must choose between keeping data strictly up-to-date (Consistency) or ensuring the system remains responsive (Availability) during a network failure. Chains, Graphs, and Callbacks A chain is a linear pipeline. Here the callback handler is attached at invocation time and fires for the chain, the template, and the model. Go import ( "github.com/cloudwego/eino/callbacks" "github.com/cloudwego/eino/components/prompt" "github.com/cloudwego/eino/compose" ) template := prompt.FromMessages(schema.FString, schema.SystemMessage("You are a translator. Reply with the translation only."), schema.UserMessage("Translate {text} to {language}.")) chain := compose.NewChain[map[string]any, *schema.Message]() chain.AppendChatTemplate(template).AppendChatModel(cm) runnable, err := chain.Compile(ctx) tracer := callbacks.NewHandlerBuilder(). OnStartFn(func(ctx context.Context, info *callbacks.RunInfo, in callbacks.CallbackInput) context.Context { log.Printf("start %s %s", info.Component, info.Type) return ctx }). OnEndFn(func(ctx context.Context, info *callbacks.RunInfo, out callbacks.CallbackOutput) context.Context { log.Printf("end %s %s", info.Component, info.Type) return ctx }). Build() out, err := runnable.Invoke(ctx, map[string]any{"text": "Buenos días", "language": "English"}, compose.WithCallbacks(tracer)) fmt.Println(out.Content) Plain Text start Chain start ChatTemplate Default end ChatTemplate Default start ChatModel Gemini end ChatModel Gemini end Chain Good morning Graphs are where Eino earns its name. Nodes are typed, edges are explicit, and plain Go functions slot in as lambda nodes: Go graph := compose.NewGraph[map[string]any, string]() _ = graph.AddChatTemplateNode("template", template) _ = graph.AddChatModelNode("generate", cm) _ = graph.AddLambdaNode("shout", compose.InvokableLambda( func(ctx context.Context, msg *schema.Message) (string, error) { return strings.ToUpper(strings.TrimSpace(msg.Content)), nil })) _ = graph.AddEdge(compose.START, "template") _ = graph.AddEdge("template", "generate") _ = graph.AddEdge("generate", "shout") _ = graph.AddEdge("shout", compose.END) runnable, err := graph.Compile(ctx) out, err := runnable.Invoke(ctx, map[string]any{"product": "a Go framework for LLM apps"}) Plain Text BUILD PRODUCTION-GRADE LLM APPS AT GO SPEED. Branches, parallel fan-out, and sub-graphs use the same API, and a compiled graph can be wrapped as a tool and handed to an agent. Eino ADK: Agents and Workflows Tools are inferred from a Go struct with jsonschema tags. The agent runs the ReAct loop and emits one event per step, so you see the tool call, the tool result, and the final answer as separate messages. Go import ( "github.com/cloudwego/eino/adk" "github.com/cloudwego/eino/components/tool" "github.com/cloudwego/eino/components/tool/utils" ) type WeatherInput struct { City string `json:"city" jsonschema:"description=The city to look up"` } weather, err := utils.InferTool("get_weather", "Returns the current weather for a city.", func(ctx context.Context, in WeatherInput) (string, error) { return fmt.Sprintf("Sunny, 24C in %s", in.City), nil }) advisor, err := adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{ Name: "travel_advisor", Description: "Helps with trip planning and weather-based advice.", Instruction: "You are a helpful travel advisor. Use tools to give accurate advice. Be brief.", Model: cm, ToolsConfig: adk.ToolsConfig{ ToolsNodeConfig: compose.ToolsNodeConfig{Tools: []tool.BaseTool{weather}, }, }) runner := adk.NewRunner(ctx, adk.RunnerConfig{Agent: advisor}) iter := runner.Query(ctx, "Should I pack a jacket for my trip to Lisbon?") for { event, ok := iter.Next() if !ok { break } if event.Err != nil { log.Fatal(event.Err) } if event.Output == nil || event.Output.MessageOutput == nil { continue } msg, _ := event.Output.MessageOutput.GetMessage() fmt.Printf("[%s] %s\n", msg.Role, msg.Content) } Plain Text [assistant] [tool] Sunny, 24C in Lisbon [assistant] It is currently sunny and 24°C (75°F) in Lisbon. You won't need a heavy coat, but packing a light jacket or cardigan is a good idea for breezy coastal evenings. The first assistant event is the tool call itself, which is why its content is empty. Multi-agent workflows are one constructor away: Go pipeline, err := adk.NewSequentialAgent(ctx, &adk.SequentialAgentConfig{ Name: "content_pipeline", Description: "Research, then write.", SubAgents: []adk.Agent{researcher, writer}, }) Plain Text [researcher] 1. Created at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released in 2009. 2. Statically typed, compiled language with built-in garbage collection. 3. Features native, lightweight concurrency through goroutines and channels. [writer] Created at Google by Robert Griesemer, Rob Pike, and Ken Thompson, the Go programming language was released in 2009. It is a statically typed, compiled language that features built-in garbage collection and native, lightweight concurrency through goroutines and channels. NewParallelAgent and NewLoopAgent complete the set, and DeepAgent adds planning and delegation on top. Developer Tooling Eino's answer to observability during development is the Eino Dev plugin for GoLand and VS Code, backed by the devops module in eino-ext. It renders your chains and graphs visually inside the editor and lets you start execution from any node with mock input. It is an IDE feature rather than a browser Dev UI, so there is no trace explorer for a running service. For that you attach one of the callback handlers (Langfuse, LangSmith, CozeLoop, APMPlus). Provider Support Model components in eino-ext: OpenAI, Claude, Gemini, DeepSeek, Qwen, Ollama, OpenRouter, Ark (Volcano Engine), and Qianfan (Baidu), plus "agentic" variants of the main ones for the newer agentic message format. Retrievers and indexers: Elasticsearch (7, 8, 9), Milvus, Qdrant, Redis, OpenSearch, Dify, and VikingDB. Tools: MCP, Google Search, Bing, DuckDuckGo, SearXNG, Wikipedia, HTTP requests, command line, browser use, and sequential thinking. Pros and Cons ✅ Pros❌ ConsMost active independent Go LLM framework (13.1k stars)Still 0.x with frequent alphas; expect API churnTyped graph orchestration with automatic streamingNo browser Dev UI; visual debugging lives in the IDE pluginCallback aspects for logging, tracing, and metricsVerbose: config structs and explicit error checks everywhereADK-style agents plus DeepAgent and interrupt/resumeByteDance-flavoured corners (Ark, VikingDB, CozeLoop)Broad component ecosystem in eino-extsonic dependency warns on the newest Go toolchainsRuns on Go 1.18+Chinese-first history still shows in some docs and examples Google ADK Go History and Direction ADK, the Agent Development Kit, was announced at Google Cloud Next 2025 with Python as its first language. The Go SDK made its first tagged release, v0.1.0, in November 2025, went 1.0 on March 23, 2026, and shipped a new major module, google.golang.org/adk/v2, on June 30, 2026. It is at v2.4.0 (September 11, 2026), has 8.8k stars, and logged over 100 commits in the last 30 days: the highest commit volume in this list. It is an official Google product with the Python, Java, Kotlin, and TypeScript SDKs as siblings. The design is the same as its siblings: everything is an agent, a workflow agent, or a tool. Gemini is the native model, with openaimodel for anything speaking the OpenAI wire format and an Apigee model for enterprise gateways. Deployment gravity points at Vertex AI Agent Engine, Cloud Run, and GKE, with Pub/Sub and Eventarc triggers built into the launcher. The minimum Go version is 1.26.6, the highest bar of the five. A deeper look at the low-level differences with Genkit is in Genkit Go vs ADK Go 2.0. ADK Go's Position: Agent-Only, Enterprise-Grade ADK Go is an agent framework. There is no vanilla generation primitive and no flow abstraction: you always build an agent, give it to a runner, and iterate the events. In exchange, you get sessions, memory, artifacts, workflow agents, agent-to-agent (A2A) protocol support, MCP toolsets, tool confirmation, and an embedded web UI, all first-party. Supported languages: Go 1.26.6+. Same framework in Python (primary), Java, Kotlin, and TypeScript. Go import ( "google.golang.org/adk/v2/agent" "google.golang.org/adk/v2/agent/llmagent" "google.golang.org/adk/v2/model/gemini" "google.golang.org/adk/v2/runner" "google.golang.org/adk/v2/tool" "google.golang.org/adk/v2/tool/functiontool" "google.golang.org/genai" ) model, err := gemini.NewModel(ctx, "gemini-3.8-flash", &genai.ClientConfig{ APIKey: os.Getenv("GOOGLE_API_KEY"), }) weather, err := functiontool.New(functiontool.Config{ Name: "get_weather", Description: "Returns the current weather for a city.", }, func(_ agent.Context, in WeatherInput) (WeatherOutput, error) { return WeatherOutput{Report: fmt.Sprintf("Sunny, 24C in %s", in.City)}, nil }) advisor, err := llmagent.New(llmagent.Config{ Name: "travel_advisor", Model: model, Instruction: "You are a helpful travel advisor. Use tools to give accurate advice. Be brief.", Tools: []tool.Tool{weather}, }) r, err := runner.NewInMemory("travel-app", advisor) msg := genai.NewContentFromText("Should I pack a jacket for my trip to Lisbon?", genai.RoleUser) for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) { if err != nil { log.Fatal(err) } if event.LLMResponse.Content == nil || !event.IsFinalResponse() { continue } for _, part := range event.LLMResponse.Content.Parts { fmt.Print(part.Text) } } Plain Text It is currently sunny and warm at 24°C (75°F) in Lisbon. You won't need a heavy jacket during the day, but packing a light jacket or sweater is recommended for cooler evenings and coastal breezes. Two things to notice. The runner returns a Go 1.23 iterator, so you range over events with the error as the second value, which is the most idiomatic event loop of the five. And the function tool is generic: input and output structs give the schema; nothing is spelled out twice. Multi-Agent Orchestration Workflow agents compose LLM agents. The OutputKey writes an agent's answer into session state, and the next agent reads it back through a {placeholder} in its instruction. Go import "google.golang.org/adk/v2/agent/workflowagents/sequentialagent" researcher, err := llmagent.New(llmagent.Config{ Name: "researcher", Model: model, Instruction: "List three key facts about the user's topic. Be terse.", OutputKey: "facts", // saved into session state }) writer, err := llmagent.New(llmagent.Config{ Name: "writer", Model: model, Instruction: "Write a two-sentence brief using these facts:\n{facts}", }) pipeline, err := sequentialagent.New(sequentialagent.Config{ AgentConfig: agent.Config{ Name: "content_pipeline", SubAgents: []agent.Agent{researcher, writer}, }, }) Plain Text [researcher] * **Origin:** Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released as open source in 2009. * **Concurrency:** Features built-in, lightweight concurrency primitives known as goroutines and channels, managed by its own runtime. * **Ecosystem Role:** Statically typed and compiled, it is the primary language behind core cloud-native technologies, including Docker and Kubernetes. [writer] Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson and released as open source in 2009, Go is a statically typed, compiled language featuring built-in, lightweight concurrency primitives known as goroutines and channels that are managed by its own runtime. Owing to these strengths, it has become the primary language powering core cloud-native technologies, including Docker and Kubernetes. parallelagent and loopagent follow the same pattern. The Dev UI: ADK Web, Embedded in Your Binary The full launcher turns any agent into a CLI with console and web modes. The web mode takes sub-launchers, and webui serves the Angular ADK Web app straight out of your Go binary via go:embed, so there is nothing to install: Go import ( "google.golang.org/adk/v2/cmd/launcher" "google.golang.org/adk/v2/cmd/launcher/full" ) l := full.NewLauncher() config := &launcher.Config{AgentLoader: agent.NewSingleLoader(advisor)} if err := l.Execute(ctx, config, os.Args[1:]); err != nil { log.Fatalf("%v\n\n%s", err, l.CommandLineSyntax()) } Shell go run . web -port 8080 api webui Plain Text Web servers starts on http://localhost:8080 api: you can access API using http://localhost:8080/api The root URL redirects to /ui/, the Agent Development Kit Dev UI, and /api/list-apps answers ["assistant"]. The launcher also wires a2a, pubsub, and eventarc sub-servers, and an -otel_to_cloud flag exports traces to Google Cloud. Flags go before the sub-launcher names, or the sub-launcher rejects them. Vertex AI Gravity You can run ADK Go anywhere. But the session and memory services with persistence are the in-memory ones, a database-backed session store, and Vertex AI; the evaluation tooling, the managed scaling, and the telemetry defaults all assume Google Cloud. Same trade-off as ADK Java and ADK Python: the best experience is on GCP. Pros and Cons ✅ Pros❌ ConsOfficial Google support, highest commit velocityAgent-only: no vanilla generation, no flowsIdiomatic Go: generics for tools, iterators for eventsGo 1.26.6+ requiredEmbedded ADK Web UI, no extra installStrong pull toward Vertex AI and GCPSequential, parallel, and loop workflow agentsGemini-first; other providers only through OpenAI-compatible endpointsA2A, MCP toolsets, tool confirmation, artifactsNo vector store components; RAG goes through tools or VertexSame mental model as ADK Python/Java/TSConfig structs get long for real agents tRPC-Agent-Go History and Direction tRPC-Agent-Go is Tencent's entry, built by the team behind tRPC, the RPC framework that underpins much of Tencent's backend. Development started in May 2025; the first public release (v0.0.1) landed in August 2025, and the project reached 1.x at the end of December 2025. It is now at v1.11.2 (August 20, 2026), with 1.8k stars and 56 commits in the last 30 days. The README credits Tencent Yuanbao, Tencent Video, Tencent News, IMA, and QQ Music with real-world validation, and I have no reason to doubt it. The framework is unapologetically ADK-shaped: llmagent, runner, chainagent, parallelagent, cycleagent, sessions, memory, OutputKey, {placeholder} instructions. If you know ADK, you already know most of it. What it adds on top is where it gets interesting: a GraphAgent that is a faithful LangGraph in Go (state schemas, conditional edges, checkpoints, time travel), an AG-UI server for streaming agent events to frontends, A2A, MCP, Anthropic-style Agent Skills with sandboxed execution, agent self-evolution, an evaluation package, model failover and hedging, prompt caching, and OpenTelemetry traces and metrics with Langfuse examples. RAG is covered by a knowledge module with embedders for OpenAI, Gemini, Ollama, and Hugging Face, and stores from in-memory and SQLite-vec up to pgvector, Elasticsearch, Milvus, Qdrant, and TcVector. What Makes tRPC-Agent-Go Stand Out The GraphAgent. Nobody else in Go ships a LangGraph-equivalent state graph as a first-class agent type that plugs into the same runner, sessions, and telemetry as the LLM agents. Supported languages: Go only, go 1.21+. Model providers: OpenAI and anything OpenAI-compatible (DeepSeek, Qwen, and friends through variants), Anthropic, Gemini, Bedrock, Ollama, Hugging Face, and Hunyuan. Go import ( "trpc.group/trpc-go/trpc-agent-go/agent/llmagent" "trpc.group/trpc-go/trpc-agent-go/model" "trpc.group/trpc-go/trpc-agent-go/model/gemini" "trpc.group/trpc-go/trpc-agent-go/runner" "trpc.group/trpc-go/trpc-agent-go/tool" "trpc.group/trpc-go/trpc-agent-go/tool/function" ) type weatherReq struct { City string `json:"city" jsonschema:"description=The city to look up,required"` } type weatherRsp struct { Report string `json:"report"` } func getWeather(ctx context.Context, req weatherReq) (weatherRsp, error) { return weatherRsp{Report: fmt.Sprintf("Sunny, 24C in %s", req.City)}, nil } m, err := gemini.New(ctx, "gemini-3.8-flash") // reads GEMINI_API_KEY / GOOGLE_API_KEY weather := function.NewFunctionTool(getWeather, function.WithName("get_weather"), function.WithDescription("Returns the current weather for a city.")) advisor := llmagent.New("travel_advisor", llmagent.WithModel(m), llmagent.WithInstruction("You are a helpful travel advisor. Use tools to give accurate advice. Be brief."), llmagent.WithTools([]tool.Tool{weather})) r := runner.NewRunner("travel-app", advisor) events, err := r.Run(ctx, "user-1", "session-1", model.NewUserMessage("Should I pack a jacket for my trip to Lisbon?")) for evt := range events { if evt.Error != nil { log.Fatalf("%s", evt.Error.Message) } if evt.Response == nil || len(evt.Response.Choices) == 0 { continue } if evt.IsFinalResponse() { fmt.Println(evt.Response.Choices[0].Message.Content) } } Plain Text It is currently sunny and warm in Lisbon at 24°C (75°F), so you will not need a heavy coat. However, packing a light jacket or sweater is recommended for cooler, breezy evenings near the coast. Events arrive on a channel and use OpenAI's response shape (Choices[0].Message or Choices[0].Delta when streaming), which is familiar but means digging through nested structs for every message. Chains The chain agent is ADK's sequential agent under another name, OutputKey and {facts} included: Go import ( "trpc.group/trpc-go/trpc-agent-go/agent" "trpc.group/trpc-go/trpc-agent-go/agent/chainagent" ) researcher := llmagent.New("researcher", llmagent.WithModel(m), llmagent.WithInstruction("List three key facts about the user's topic. Be terse."), llmagent.WithOutputKey("facts")) writer := llmagent.New("writer", llmagent.WithModel(m), llmagent.WithInstruction("Write a two-sentence brief using these facts:\n{facts}")) pipeline := chainagent.New("content_pipeline", chainagent.WithSubAgents([]agent.Agent{researcher, writer})) Plain Text [researcher] 1. Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released in 2009. 2. Statically typed, compiled language featuring automatic garbage collection. 3. Native concurrency support using lightweight threads called goroutines and channels. [writer] Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson and released in 2009, Go is a statically typed, compiled language featuring automatic garbage collection. It is designed with native concurrency support, utilizing channels and lightweight threads called goroutines. GraphAgent: LangGraph in Go A state graph with a routing node, a conditional edge, and two LLM nodes. Short questions go to the quick node, long ones to the deep node. Go import ( "trpc.group/trpc-go/trpc-agent-go/agent/graphagent" "trpc.group/trpc-go/trpc-agent-go/graph" ) sg := graph.NewStateGraph(graph.MessagesStateSchema()) sg.AddNode("route", func(ctx context.Context, state graph.State) (any, error) { return nil, nil // pure routing node, no state change }) sg.AddLLMNode("quick", m, "Answer in one short sentence.", nil) sg.AddLLMNode("deep", m, "Answer thoroughly. Follow the length the user asks for.", nil) sg.SetEntryPoint("route") sg.AddConditionalEdges("route", func(ctx context.Context, state graph.State) (string, error) { input, _ := state[graph.StateKeyUserInput].(string) if len(strings.Fields(input)) > 8 { return "deep", nil } return "quick", nil }, map[string]string{"quick": "quick", "deep": "deep"}) sg.SetFinishPoint("quick") sg.SetFinishPoint("deep") g, err := sg.Compile() router, err := graphagent.New("router", g) r := runner.NewRunner("graph-app", router) Asked "What is Go?" and then a request for two paragraphs on goroutines and channels: Plain Text Go is an ancient strategy board game, as well as an open-source programming language developed by Google. Unlike operating system threads that require megabytes of memory and expensive kernel-level context switching, goroutines are extremely lightweight user-space constructs that start with just a few kilobytes of stack space. [...] The same graph package gives you checkpoints, interrupts for human input, sub-graphs, map-reduce fan-out, per-node callbacks, and a visualization export. It is the most complete workflow engine of the five, and it is the reason to look at this framework even if you never use its LLM agents. Pros and Cons ✅ Pros❌ ConsGraphAgent: a real LangGraph for GoSmallest community of the five (1.8k stars)ADK mental model without the GCP gravityNo Dev UI; observability means OpenTelemetry plus LangfuseAG-UI, A2A, MCP, Agent Skills, evaluationHuge option surface (llmagent alone has dozens)Model failover and hedging built inOpenAI-shaped event structs are verbose to consumeOpenTelemetry traces and metrics first-partyTencent-flavored corners (Hunyuan, tRPC integration)Runs on Go 1.21+Documentation partly Chinese-first LangChainGo History and Direction LangChainGo was the first. Travis Cline started it in February 2023, weeks after LangChain Python went viral, and for two years it was the default answer to "how do I call an LLM from Go". It has 9.7k stars, 17 LLM providers, and 15 vector stores, and the LangChain names (chains, prompts, agents, memory, outputparser) make it instantly readable to anyone coming from Python. And it has stalled. The last release, v0.1.14, is from October 20, 2025. The last commit to main is from January 11, 2026. It is still 0.1.x after three and a half years. There are 243 open issues and 174 open pull requests. The repository is not archived, but nobody is merging. That would be survivable if the code kept working. It does not: The default model is retired. googleai.New(ctx) with no options targets gemini-2.0-flash, and Google turned that model off: Plain Text default model: gemini-2.0-flash err: googleapi: Error 404: This model models/gemini-2.0-flash is no longer available. Please update your code to use models/gemini-3.6-flash for the latest features and improvements. The Gemini provider sits on a dead SDK. llms/googleai is built on google/generative-ai-go, whose support ended on November 30, 2025. Streaming is broken on Gemini. The chunks arrive, and then the call returns an error, so every caller that checks errors treats the request as failed: Plain Text One, two, three, four, five. err: error in stream mode: invalid character ']' looking for beginning of value No middleware, retry, fallback, or OpenTelemetry anywhere. The callback handler is seventeen observe-only methods with no way to change the request. Tools are Call(ctx, string) (string, error): no schema, no typed arguments. The full teardown, with a capstone app built in both frameworks, is in LangChainGo vs Genkit Go: Where Genkit Shines. LangChainGo's Position: Legacy None of this means the basics stopped working. Pin a current model and the classic API still runs: Go import ( "github.com/tmc/langchaingo/llms" "github.com/tmc/langchaingo/llms/googleai" ) llm, err := googleai.New(ctx, googleai.WithDefaultModel("gemini-3.8-flash")) out, err := llms.GenerateFromSinglePrompt(ctx, llm, "Explain the CAP theorem in two sentences.") Plain Text The CAP theorem states that a distributed data store can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Because network partitions are inevitable in real-world systems, architects must fundamentally choose between serving the most up-to-date data (Consistency) or keeping the system operational (Availability) when a network failure occurs. Go import ( "github.com/tmc/langchaingo/chains" "github.com/tmc/langchaingo/prompts" ) prompt := prompts.NewPromptTemplate( "Translate {{.text} to {{.language}. Reply with the translation only.", []string{"text", "language"}) chain := chains.NewLLMChain(llm, prompt) out, err := chains.Predict(ctx, chain, map[string]any{ "text": "Buenos días", "language": "English", }) Plain Text Good morning Tools are strings in, strings out. The agent is a ReAct loop driven by prompt text: Go import ( "github.com/tmc/langchaingo/agents" "github.com/tmc/langchaingo/tools" ) type WeatherTool struct{} func (WeatherTool) Name() string { return "get_weather" } func (WeatherTool) Description() string { return "Returns the current weather. Input: a city name." } func (WeatherTool) Call(ctx context.Context, city string) (string, error) { return fmt.Sprintf("Sunny, 24C in %s", city), nil } agent := agents.NewOneShotAgent(llm, []tools.Tool{WeatherTool{}, agents.WithMaxIterations(3)) out, err := chains.Run(ctx, agents.NewExecutor(agent), "What's the weather like in Tokyo?") Plain Text The weather in Tokyo is currently sunny with a temperature of 24°C. If you are on LangChainGo today, the move is not urgent, but it is inevitable: pin your models, avoid streaming on Gemini, and plan the migration. The concept-by-concept guide is in Migrating from LangChainGo to Genkit Go. Pros and Cons ✅ Pros❌ ConsLargest catalog: 17 providers, 15 vector storesUnmaintained since January 2026; still 0.1.xLangChain vocabulary, easy for Python teamsGemini provider on a retired SDK; streaming brokenConversation memory is ergonomicNo middleware, retry, fallback, or OpenTelemetryRuns on Go 1.24+Tools are untyped stringsMany existing tutorials and examplesEvery vector store needs an external service Head-to-Head Comparison Developer Experience FrameworkDX HighlightsShortcomingsGenkit GoDev UI with trace explorer. Typed output from structs. Flows are one line from HTTP.Agents still experimentalEinoTyped graphs, automatic streaming, callbacks everywhere.Verbose config structs; 0.x churnADK GoIterator-based event loop, generic tools, embedded web UI.Agent-only; Go 1.26.6tRPC-Agent-GoGraphAgent, ADK mental model, resilience built in.OpenAI-shaped events; option overloadLangChainGoFamiliar names, biggest catalog.Unmaintained; broken Gemini streaming Abstraction Levels Genkit Go is the only framework here with all three levels in one SDK: vanilla generation, typed flows, and agents. Eino covers generation and orchestration, plus agents through its ADK module. ADK Go and tRPC-Agent-Go are agent frameworks: every call goes through an agent and a runner. LangChainGo has chains and agents, but no typed pipeline and no session-aware agent. Observability FrameworkLocal DevProductionGenkit GoDev UI with trace explorerOpenTelemetry export, Google Cloud plugin, any OTEL collectorEinoEino Dev plugin for GoLand and VS Code (graph rendering, node debugging)Callback handlers: Langfuse, LangSmith, CozeLoop, APMPlusADK GoEmbedded ADK Web UIOpenTelemetry to Google Cloud (-otel_to_cloud)tRPC-Agent-GoEvents on a channel, no UIOpenTelemetry traces and metrics, LangfuseLangChainGoLogging callbacks onlyNothing built in Provider Support FrameworkModel ProvidersVector StoresGenkit GoGemini (Google AI, Vertex AI), Anthropic, Ollama, OpenAI-compatible family (OpenAI, xAI, DeepSeek, OpenRouter, Kimi, DashScope, Z.ai)localvec, pgvector, AlloyDB, Pinecone, Weaviate, FirestoreEinoOpenAI, Claude, Gemini, DeepSeek, Qwen, Ollama, OpenRouter, Ark, QianfanElasticsearch, Milvus, Qdrant, Redis, OpenSearch, Dify, VikingDBADK GoGemini, OpenAI-compatible, ApigeeNone as components (Vertex AI Search via tools)tRPC-Agent-GoOpenAI-compatible, Anthropic, Gemini, Bedrock, Ollama, Hugging Face, HunyuanIn-memory, SQLite-vec, pgvector, Elasticsearch, Milvus, Qdrant, TcVectorLangChainGo17, including OpenAI, Anthropic, Google AI, Vertex, Bedrock, Cohere, Mistral, Ollama15, all external services How Go Does It Feel? FrameworkMin GoToolsStreamingStyleGenkit Go1.25Generic, schema from structCallback per chunkFunctional options (ai.With...)Eino1.18Generic (utils.InferTool)StreamReader, automatic across nodesConfig structs plus builder chainsADK Go1.26.6Generic (functiontool.New)iter.Seq2 over eventsConfig structstRPC-Agent-Go1.21Generic plus jsonschema tagsChannel of eventsFunctional optionsLangChainGo1.24Untyped stringsCallback per chunk (fails on Gemini)Interfaces plus option funcs Only ADK Go uses range-over-func iterators for its event loop. Genkit and Eino use generics the most naturally; LangChainGo predates all of it and shows its age. Maintenance Snapshot Numbers as of September 21, 2026: FrameworkStarsLatest ReleaseLast CommitCommits, Last 30 DaysGenkit Go6.4k (monorepo)go/v1.13.1, Sep 3, 2026Sep 21, 202680 (all SDKs)Eino13.1kv0.9.20, Sep 20, 2026Sep 20, 20267 on main, plus five v0.10 alphasADK Go8.8kv2.4.0, Sep 11, 2026Sep 21, 2026100+tRPC-Agent-Go1.8kv1.11.2, Aug 20, 2026Sep 21, 202656LangChainGo9.7kv0.1.14, Oct 20, 2025Jan 11, 20260 Stars measure the past. The last two columns measure the present, and they tell the story. Which Framework Should You Choose? Choose Genkit Go if: You want to iterate on your AI fast and get feedback with less back-and-forth. The Dev UI is the best local development loop in Go, by a distance.You need vanilla calls, typed flows, and agents in the same service.Provider neutrality matters: mixing Gemini, Claude, OpenAI, and Ollama should be a one-line change.Your team also writes TypeScript, Dart, or Java and wants one framework story. Choose Eino if: Your application is a real graph: branches, parallel fan-out, sub-graphs, and streaming across all of it.You want the most active independent community in Go and are fine living on 0.x.Callback-based instrumentation into Langfuse, LangSmith, or your own APM is what you already do.You need ByteDance-scale orchestration patterns, or you run on Volcano Engine. Choose Google ADK Go if: You are building multi-agent systems and Google Cloud is your runtime.You want official Google support, the embedded ADK Web UI, and A2A interoperability.Your organization already uses ADK in Python or Java and wants Go services with the same shape.The newest Go toolchain is not a problem for you. Choose tRPC-Agent-Go if: You want LangGraph-style state graphs with checkpoints and interrupts, in Go, plugged into agents.You like the ADK mental model but do not want the GCP gravity.AG-UI frontends, Agent Skills, evaluation, and OpenTelemetry out of the box matter more than a Dev UI.You can live with a smaller community and a large API surface. Stay on LangChainGo only if: You already run it in production, your models are pinned, and you do not stream through Gemini.Even then, plan the migration. Nothing is being merged. Conclusion Go's Gen AI framework landscape in 2026 is richer than most Go developers realize, and it has a clear shape. Genkit Go is the most versatile choice and the best developer experience, with the Dev UI as a category of its own. Eino is the ambitious independent, the framework to pick when orchestration is the hard part of your problem. Google ADK Go is the enterprise agent runtime, at its best on Google Cloud. tRPC-Agent-Go is the dark horse, with the best workflow engine of the five hiding behind a modest star count. And LangChainGo is the framework that got everyone started and now needs a farewell plan. If I had to pick one for a new Go service today with no cloud commitment, it would be Genkit Go, for the same reason as in TypeScript and Java: the combination of multiple abstraction levels, provider neutrality, and local observability shortens the loop between an idea and a working feature more than anything else on this list. Whatever you pick, pick it deliberately. Go finally has the tooling. Use it. Last updated: September 2026. Framework versions referenced: Genkit Go v1.13.1, Eino v0.9.20 (with eino-ext gemini v0.1.36), Google ADK Go v2.4.0, tRPC-Agent-Go v1.11.2, LangChainGo v0.1.14. Go 1.27.1, model gemini-3.8-flash.

By Xavier Portilla Edo

CORE

Beyond Manual Mapping: AI-Orchestrated Governance for Hybrid Cloud

Beyond Manual Mapping: AI-Orchestrated Governance for Hybrid Cloud It's Tuesday morning, and Slack is already on fire. A Salesforce admin renamed a custom field over the weekend, and your Spring Boot microservice has been throwing 400 errors since 6 a.m. You spend the next four hours tracing logs, patching a DTO, and redeploying. By Wednesday, you're in an "alignment meeting," manually cross-referencing field mappings between two systems that are supposed to talk to each other without your help. Every architect running a hybrid Salesforce and Spring Boot stack has lived this week. It's not a one-time incident; it's a recurring tax on the org. It grows with every schema change, every new field, every integration you bolt on. Most teams accept it as the cost of doing business. It doesn't have to be. Where the Wheels Come Off The friction comes from two architectures that were built with opposite priorities. Salesforce is optimized for speed. A business user creates a custom field in Setup, and it's live in minutes, no code review, no pipeline, no waiting. Spring Boot is optimized for rigor. Fields live in strongly typed DTOs. Changing one means a commit, a PR review, a test run, and a deployment through CI/CD. That's a feature, not a bug; it's what keeps your backend from falling over. When the two connect over REST, that mismatch turns into outages. Here's the kind of DTO that breaks the moment Phone__c becomes PhoneNumber__c on the Salesforce side: Java public class CustomerDTO { private String customerId; private String phone; // maps to Salesforce field Phone__c // getters/setters omitted } Nothing in this class tells you the field it depends on just got renamed. The Salesforce admin has no idea a Java class is bound to Phone__c. The developer has no idea the picklist they're parsing was just restructured with two new values. Information doesn't flow; it fragments. Teams patch around it, and the patches pile into a Frankenstein integration made of XSLT transforms, one-off converters, and tribal knowledge that lives in one person's head. That tax is measurable. Developers routinely lose 30 to 40 percent of their time to keeping brittle connections alive instead of shipping features. Every schema change becomes a small bet against production. A Different Approach: Zero-Touch Governance What if the system watched itself and fixed what it could, with a human still holding the approval button? That's the idea behind AI-orchestrated API governance: an orchestration layer that watches both Salesforce and Spring Boot, understands what changed and why it matters, and proposes (or, once trusted, implements) the fix. The important word is "understands." A naive diff tool sees Cust_Phone__c disappear and CustomerPhone__c appear, and flags it as a breaking deletion plus a breaking addition. An AI layer doing semantic analysis recognizes it as a rename, maps it to the existing customerPhoneNumber field in your DTO, and drafts the OpenAPI and Java changes before anyone gets paged. Building the Intelligence Layer Three components have to work together for this to be trustworthy instead of terrifying. 1. The Observation Engine This layer taps Salesforce Change Data Capture (or Platform Events) on one side and your Spring Boot Actuator endpoints on the other, then asks an LLM to classify what actually happened. A simplified CDC listener might look like this: Java @Component public class SalesforceChangeListener { @EmpConnectorSubscription(channel = "/data/Contact_ChangeEvent") public void onFieldChange(ChangeEvent event) { FieldChange change = FieldChange.builder() .objectName(event.getEntityName()) .oldFieldApiName(event.getChangedFieldOldName()) .newFieldApiName(event.getChangedFieldNewName()) .changeType(event.getChangeType()) .build(); semanticClassifier.classify(change); } } The classifier itself is a targeted prompt, not a general chatbot call: Java String prompt = """ A Salesforce field changed on object %s. Old API name: %s New API name: %s Change type: %s Classify this as one of: RENAME, TYPE_CHANGE, DEPRECATION, NEW_FIELD, BREAKING_REMOVAL. If RENAME, return the confidence score and the likely corresponding Spring Boot DTO field name. Respond only in JSON. """.formatted(objectName, oldName, newName, changeType); That last constraint matters more than it looks. Constraining the model's output to a fixed schema is what makes the downstream Policy Engine able to act on the result programmatically instead of parsing prose. 2. The Policy Engine Detection without guardrails is just a faster way to break things. The Policy Engine enforces the rules your architecture team already agreed on, things like "no breaking changes to public APIs without a major version bump" or "PII fields must stay encrypted in transit." Java public class GovernancePolicy { public PolicyResult evaluate(ProposedChange change) { if (change.isBreaking() && !change.hasMajorVersionBump()) { return PolicyResult.blocked( "Breaking change requires API major version bump"); } if (change.touchesPiiField() && !change.isEncrypted()) { return PolicyResult.blocked( "PII fields must remain encrypted end to end"); } if (change.isCustomerFacing() && change.deprecationWindowHours() < 48) { return PolicyResult.blocked( "Customer-facing changes need a 48 hour deprecation window"); } return PolicyResult.approved(); } } If a proposed change trips one of these rules, automation stops and an architect gets looped in. That's the whole point: the system earns trust by refusing to act outside its lane, not by acting fast. 3. The Execution Layer Once a change clears policy, this layer turns the proposal into an actual pull request. It can regenerate the OpenAPI contract: YAML CustomerDTO: type: object properties: customerId: type: string customerPhoneNumber: # renamed from "phone" type: string description: "Renamed from Salesforce Cust_Phone__c to CustomerPhone__c" and the matching Java class: Java public class CustomerDTO { private String customerId; private String customerPhoneNumber; // was: phone, mapped from CustomerPhone__c // getters/setters omitted } then open a PR, run the existing test suite in staging, and post the results back to the team. The API contract becomes the single source of truth both systems reconcile against, instead of a document that drifts out of date the week after it's written. A Second Example: Picklist Drift Renames aren't the only failure mode. Picklists shift constantly, and Spring Boot enums don't forgive silently. Say a Salesforce admin adds "Pending Review" to an Order_Status__c picklist that a Java enum is validating against: Java public enum OrderStatus { NEW, IN_PROGRESS, SHIPPED, CANCELLED // "Pending Review" doesn't exist here yet } Any order that lands in that new status now fails deserialization. The Observation Engine catches the new picklist value through the Metadata API, the Policy Engine checks whether enum changes count as breaking under your versioning rules, and the Execution Layer proposes the enum update alongside a migration note for any records currently in that state. That's a five-minute PR instead of a production incident discovered by a customer. What This Means for Your Team The obvious worry is that this replaces developers. It doesn't; it changes what they spend their day on. Right now, a good chunk of your time goes to translation work: mapping fields, writing converters, chasing down what changed and why. That work is necessary, but it isn't strategic. Once an orchestration layer is handling the translation, your role shifts from translator to curator. You define the data architecture, you write the policies the system enforces, and you spend your attention on the problems that actually need a human's judgment. Making the Transition Nobody should flip this on in production on day one. Trust gets earned in stages. Start with AI-Suggested Governance. The system proposes changes and opens PRs; a human reviews and merges. You're delegating the tedious analysis, not the decision. Once accuracy holds up across enough changes, typically approaching 99 percent, move to AI-Automated Testing. The system applies changes in staging, runs your regression suite, and reports results without a human triggering each step. Only after it's proven itself across hundreds of real changes should you consider Zero-Touch in Production, where the system can act autonomously within policy boundaries you've already defined and tested. Wrapping Up The hybrid enterprise, Salesforce for the customer relationship, Spring Boot for the backend logic, isn't going away. But manually bridging the two with mapping layers and tribal knowledge is a relic you can retire. AI-orchestrated API governance doesn't remove developers from the loop. It removes the busywork from their day and gives back the four hours you used to spend in a war room every time a field got renamed.

By Ramesh Bellamkonda

PWA on iOS, Part 3: Chrome on iPhone Is WebKit Too. How One Engine Sets the Rules for PWAs on iOS

This is Part 3 of an 8-part series about PWAs on iOS. We've covered the history of the technology and the three ingredients every PWA needs: HTTPS, a manifest, and a service worker. Now it's time to look at the engine all of that runs on, and why, on the iPhone, Apple chose it for you long ago. A typical bug report from a tester goes something like this: "Works fine on Android. On iPhone we tried Safari and Chrome, and it breaks the same way in both." The tester dutifully opened two browsers and believes two engines are covered, when in fact they tested the same WebKit twice in different wrappers. Chrome on iPhone renders pages with the same system engine as Safari, and so do Firefox, Edge, Brave, and the rest. They differ in toolbars, bookmark sync, and looks, but in every one of them the page is built and run by WebKit, in whatever version shipped with iOS. It's easy to jump to the opposite conclusion: if there's only one engine, testing Safari must be enough. That's a mistake too, and just as common. When someone says "it works in Safari for us" on a call, it usually means they opened the site in a tab and stopped there. But one engine doesn't mean one environment. On an iPhone, it runs in at least four, and each has its own cookies, its own storage, and its own rules for how long a process lives. Checking a Safari tab and declaring victory is like looking into one room out of four and concluding that the whole house is tidy. We'll go through the rooms below. A Fork That Outlived Its Parent WebKit wasn't written from scratch in Cupertino. Its ancestor was KHTML, the engine of KDE's Konqueror browser, and next to it lived a JavaScript interpreter called KJS. Both projects were open source and distributed under the LGPL, a license that lets you take the code, modify it, and even build it into a closed-source product, but requires you to publish your changes to the library itself. KDE is an international free software community. It began in 1996, when Matthias Ettrich, a student at the University of Tübingen, announced on Usenet a project for a graphical desktop environment for Unix-like systems, and KDE 2.0 in October 2000 introduced Konqueror, which replaced the file manager and doubled as a browser. Apple started work on its fork in the summer of 2001. The reasons given for the choice were very practical: KHTML and KJS were small (under 140,000 lines of code), cleanly designed, standards-compliant, and easier to work with than the other options available. In January 2003, Steve Jobs unveiled Safari at Macworld, and the same day an Apple engineer, Don Melton, wrote to the KDE developers: Safari was built on their code, the modified versions had been made public, and a list of the changes would follow in a separate email. You often read that "Apple open-sourced WebKit in 2003," and that's not quite accurate. In 2003, two libraries were open: WebCore (HTML and CSS parsing, layout and rendering) and JavaScriptCore, both under the LGPL because they had grown out of KDE code released under that license. The WebKit framework itself, the glue code that Safari and other Mac programs used to talk to the engine, was only opened up on June 7, 2005, together with a public repository and bug tracker. Dave Hyatt of the Safari team announced it on his blog. The first iPhone in 2007 shipped with the same engine, which had to be squeezed onto a slow processor and a small amount of memory. In 2010, the project announced WebKit2, an architecture where the interface and the page content live in separate processes. The UI Process handles the window and the buttons, while the Web Content Process parses HTML, runs JavaScript, and can crash without taking the whole app down with it. On a phone this design matters more than on a laptop, and we'll come back to it. By then Google had been shipping Chrome on WebKit for a year and a half, but Chromium's multiprocess model had been built its own way from the start, and maintaining two architectures in one project was getting more and more expensive for both sides. On April 3, 2013, Google announced a fork called Blink and named exactly this reason in the announcement itself. Chrome on Android and Safari on iPhone have been cousins ever since, sharing a common ancestor. But after thirteen years of separate lives, the engines have drifted so far apart that how one behaves tells you almost nothing about how the other will. As for KHTML, where it all began, KDE eventually stopped developing it: work on KHTML ended in 2016, and in 2023 it was officially retired. One Line in the App Store Rules The WebKit requirement fits into a single line of the App Store rules. Guideline 2.5.6 says: "Apps that browse the web must use the appropriate WebKit framework and WebKit JavaScript." Today the guideline adds that you may apply for an entitlement to use an alternative browser engine, with links to the rules for the EU and Japan, but everywhere else the requirement is unchanged. For Google and Mozilla, this meant their iPhone browsers stayed shells for years: their own interface, account sign-in, tab and password sync, and inside it the same WKWebView that any app in the store can use. Chrome on iOS can't enable a feature WebKit hasn't shipped yet, and it can't work around a bug WebKit hasn't fixed yet. If something gets fixed in Chromium on a Tuesday, iPhones get the fix only when Apple ships an iOS update with a newer WebKit, and a fresh version of Chrome from the App Store changes nothing. Apple justifies the rule on security grounds, and I don't think that's a mere excuse. In its developer documentation, Apple calls browser engines one of the most common attack vectors, and that's hard to argue with: an engine constantly processes untrusted code from all over the internet. When there is one engine, and it's part of the system, a single iOS update closes the hole for every browser at once. On top of that, fast JavaScript needs JIT compilation, meaning the ability to create executable code in memory at runtime, and ordinary iOS apps aren't given that permission. I find the decision reasonable for the platform and costly for the web. The security is paid for by making the web on iPhone depend on a single supplier, and that supplier's release schedule automatically becomes everyone else's. Attempts to get out of this arrangement began before Apple allowed anything. In December 2022, press reports appeared saying that Apple was preparing for the EU's Digital Markets Act and was discussing dropping the mandatory WebKit rule. About six weeks later, on January 31, 2023, the Chromium team opened an issue in its public tracker for an experimental port of Blink to iOS. The issue stated up front that they were building a stripped-down test shell, content_shell, rather than a browser for the store, because such a browser would violate the App Store rules. Around the same time there were reports that Mozilla was also trying to get its Gecko engine running on iPhone. I'll say how that turned out below, when we get to the regulators. Four Rooms in One House Even with a single engine, the environment it runs in can differ, and for a web app that difference matters more than the browser's brand. I keep four such environments in mind, and data doesn't flow from one to another on its own. Room 1 The most familiar one is an ordinary Safari tab. Cookies, localStorage, IndexedDB and camera and notification permissions are shared across the tabs of a profile, Web Inspector on a Mac can attach to the tab, and almost every tutorial on the internet was tested right here. Room 2 SFSafariViewController looks almost the same. It's a Safari window with Reader and password autofill that a native app opens on top of itself when you tap a link in a feed or an email. The app itself can't see what's happening inside. Before iOS 11, that window shared cookies with Safari, and a lot of single sign-on setups relied on it. In iOS 11 Apple separated the storage, and each app got its own cookie jar for such windows. Today Apple's documentation explicitly sends anyone who needs data shared with Safari to a different API, ASWebAuthenticationSession. The old behavior still circulates in forums and articles. If someone confidently says that the user is already logged in inside SFSafariViewController, ask which iOS version they tested that on. Room 3 WKWebView is a different story. This embeddable web view is what hybrid apps, the in-app browsers of messengers, and Chrome and Firefox themselves are built on. Its storage belongs to the host app: Safari's cookies don't carry over, but the app itself can read and set cookies through WKHTTPCookieStore, so your session won't be a secret from the host. Even the quota is different. Since iOS 17, WebKit lets a single site in a browser app, meaning one that can be set as the default browser, use up to 60% of the disk, but inside the web view of an ordinary app only up to 15%. Room 4 An icon on the Home Screen, opened as a web app, lives apart from all the others. It has its own container with its own cookies, localStorage, IndexedDB, Cache Storage and its own permissions, and that container can't see Safari or any web view embedded in other apps. Someone who's logged in inside a tab will install the app, tap its icon and find a login form, and camera access granted in the tab has to be granted again. Since iOS 26, a site added to the Home Screen opens as a web app by default. You can turn off the "Open as Web App" switch when adding it, and then the icon becomes an ordinary bookmark that takes you back to Room 1. When a bug report comes from a user, the first thing we need to work out is which room they were in. For that, I usually hide a diagnostics screen in the app, opened by long-pressing the version number, that shows an object like this: JavaScript async function whereAmI() { const standalone = matchMedia("(display-mode: standalone)").matches || navigator.standalone === true; return { standalone, path: location.pathname, sw: navigator.serviceWorker?.controller?.scriptURL ?? "none", caches: "caches" in self ? await caches.keys() : [], build: document.documentElement.dataset.build, ua: navigator.userAgent, }; } navigator.standalone is a nonstandard Apple property, and inside an embedded WKWebView both checks come back false, so they can't tell a Safari tab from an in-app browser inside a messenger. The user agent sometimes helps, since it may contain the name of the host app, but that's a hint for a human reading the report, and I wouldn't build logic on it. I take the address from pathname because the query string sometimes holds one-time tokens that have no business being in logs. From here on, when I say a web app opened from the Home Screen, I mean an actual PWA, not a plain bookmark to a site page that someone has put on the Home Screen. Why Users Log In Three Times The most common story I've run into across different teams goes like this. A corporate single sign-on login works fine in a Safari tab, because the identity provider's cookie is already in the profile. Then a native wrapper appears, and sign-in there is opened through SFSafariViewController, expecting the same cookie to be there. It isn't, so the person types their password a second time. Then the site is put on the Home Screen, and it all happens a third time, because the icon has its own container. Support gets a complaint: "the app keeps logging me out," even though the app never did anything of the kind. Each sign-in the user went through simply happened in a new, separate container. You can't read Safari's cookies from another environment. There's no public API for it, and that's by design. The workable options are more boring. A native app that needs a session from Safari should use ASWebAuthenticationSession, available since iOS 12. It shares persistent cookies with Safari and asks the user for permission before opening the sign-in page, and if you specifically don't want a shared session, the prefersEphemeralWebBrowserSession flag makes the sign-in private. For a PWA, I assume that sign-in starts from scratch after installation, and I try to keep it short: OAuth with PKCE, a redirect back to an address inside the manifest's scope, and a long-lived session on your own domain, so that the user signs in once after installing rather than every week. I also always check on a real device where the provider's page opens and where the redirect brings the user back, because navigating outside scope in an installed app behaves differently than it does in a browser tab. Twenty Billion Dollars for a Search Bar If the WebKit-only requirement rested on security alone, it would be easier to argue about. But Safari has a commercial side too: the right to be its default search engine is worth a great deal. In the US Department of Justice's antitrust case against Google, court documents disclosed in spring 2024 put the amount Google paid Apple in 2022 to be Safari's default search engine at about $20 billion. That doesn't prove WebKit was made mandatory on every iPhone for the sake of money, but it shows what's at stake: Safari is a significant source of revenue for Apple. So anything that could take users away from it on iPhone, whether a browser choice screen or other engines, touches not only architecture but revenue. Critics of the rule that every browser must use WebKit, above all Open Web Advocacy, have argued for years that without competition between engines Apple has little incentive to move quickly on the features that make web apps resemble native ones. As an example, they point to Web Push, which web apps on iPhone didn't get until spring 2023. Apple, for its part, points to Safari's speed, to privacy, and to its view that some APIs from Chromium are dangerous for users. I'm not firmly on either side of this argument. Some of the features missing on the iPhone are ones I wouldn't hand to a random website on my own phone either. Others were delayed for years without a clear explanation, and from the outside it's very hard for me to tell where concern for the user ends and concern for the App Store begins. The Month PWAs Disappeared in Europe The first to break the WebKit-only rule was the European Union. The Digital Markets Act (DMA) required Apple to allow other browser engines on iPhone, and iOS 17.4, released in March 2024, introduced BrowserEngineKit, a set of system interfaces for browsers that run on their own engine. It applies in the 27 EU member states, but the European Economic Area as a whole isn't covered, so a user in Norway, for example, is left with plain WebKit. Apple spelled out the conditions in great detail, and they read more like a list of reasons not to bother. A browser with its own engine must be distributed only in the EU, must be eligible to be set as the default browser, and must pass 90% of the Web Platform Tests and 80% of Test262 even with JIT turned off (for example, when the user has Lockdown Mode enabled). It also has to publish a vulnerability disclosure policy and quickly patch actively exploited vulnerabilities, with Apple naming a 30-day window for straightforward cases. Mozilla reacted right after the announcement, in January 2024. A company spokesperson, Damiano DeMonte, told The Verge that Mozilla was "extremely disappointed" and explained why: "The effect of this would be to force an independent browser like Firefox to build and maintain two separate browser implementations — a burden Apple itself will not have to bear." Formally, Apple had complied with European law. In practice, it had made it prohibitively expensive for any company that develops its own browser engine to bring it to Apple's devices in the EU. For PWAs, the most interesting part happened in the beta. In February 2024, developers in the EU noticed that in the iOS 17.4 beta, PWAs no longer opened in their own window and had turned into bookmarks that launch an ordinary browser tab. Gone with the window were the separate data container and Web Push, which had arrived only a year earlier. Apple blamed the same DMA: since browsers on other engines were now allowed, they would have to be let into the web app mechanism too, and that would require a new integration architecture. Building it, Apple said, was impractical: on top of the law's other requirements, very few people use Home Screen web apps, and the company judged the cost unjustified. Open Web Advocacy, developers and the press kicked up a fuss, the European Commission got involved, and on March 1, 2024, Apple reversed the decision. The final iOS 17.4 brought web apps back in the EU, still running on WebKit. That autumn Apple took the next step: starting with iOS 18.2, a browser with its own engine in the EU can create Home Screen web apps that run on that engine. On paper, that's exactly the freedom people were fighting for. But only a browser with its own engine that has met all of Apple's conditions can use it. London and Tokyo Followed Brussels The UK's Competition and Markets Authority (CMA) wrapped up its study of mobile ecosystems in June 2022 by proposing an investigation into how the mobile browser market works. On November 22 of that year, it launched an in-depth market investigation into browsers and cloud gaming. Apple challenged that decision in court, and after a ruling by the Competition Appeal Tribunal, the investigation was suspended from March 31, 2023. The CMA appealed the ruling; the Court of Appeal upheld the CMA's decision on November 30, 2023, and on January 24, 2024, the investigation resumed. The final report came out on March 12, 2025, and its conclusion is blunt: competition in the UK mobile browser market isn't working well, users and businesses both suffer as a result, and the main concerns relate to Apple's rules for browsers on its devices. The CMA decided to change those rules not through the investigation but under the UK's new digital markets law, the Digital Markets, Competition and Consumers Act, which came into force on January 1, 2025. On October 22, 2025, it designated Apple and Google as having Strategic Market Status in mobile platforms, and it explicitly included browsers and browser engines in its definition of a mobile platform. The regulator now has a tool to demand changes. But as of autumn 2026, there is still no requirement to allow other engines on iPhone: Apple's first commitments to the CMA concern the rules for reviewing and ranking apps in the App Store. Japan went further. The Mobile Software Competition Act (MSCA) was passed in June 2024 and came into full force in December 2025. In the guidelines to it, which got a lot of coverage in summer 2025, the Japanese regulator specifically barred Apple from placing unjustified technical restrictions on alternative engines, from imposing high financial costs on developers, and from steering users away from browsers built on such engines. In effect, Japan ruled out the European scenario in advance, where permission technically exists but using it doesn't pay. On December 17, 2025, Apple announced its Japan changes in iOS 26.2. Users in Japan got a choice screen for browsers and search engines, and browser developers could move to engines other than WebKit, with access to JIT and multiprocessing and, as Apple puts it, under strict security and privacy requirements. As for the prototypes mentioned earlier, they remained prototypes, and ordinary Chrome on iPhone, in Europe and Japan alike, still runs on WebKit. In June 2026, Kyle Pflug of the Microsoft Edge team published benchmark results for a research prototype of a Blink-based browser for iOS, built with BrowserEngineKit. In Speedometer 3.1, the prototype beat Safari on iOS 26.5.1 by about 29%, and Pflug himself stressed that he was showing a research build, not announcing a product. As of autumn 2026, there are almost no full-fledged iPhone browsers running on their own engine. I wouldn't build product plans around their arrival any time soon, but "never" isn't the right word either. So for now, what the web can do on iPhone is determined by WebKit, and you have to plan a product around what WebKit lacks for PWAs on iOS. Some of the gaps are visible right away, while others surface at the most inconvenient moment. The one that stands out first is the lack of an install button: a site can't offer to install itself as an app. There Will Be No "Install" Button Chromium has a beforeinstallprompt event: a site uses it to show its own button and trigger the system install dialog. iOS has no such event, and the only way to put a site on the Home Screen is by hand, through Share and then Add to Home Screen. Because of this, onboarding on iPhone always ends up a little clunky. In a Safari tab, you can show the instructions in words and a picture; in an already installed app, you have to hide them; and in a messenger's in-app browser, they're often useless because the elements they refer to, such as the Share button, simply aren't there. There are no background mechanisms either. Background Sync, Periodic Background Sync and Background Fetch aren't implemented on iOS, so you can't send out a queue of pending requests after the user has closed the app. On the hardware side, Web Bluetooth, Web NFC, and WebUSB are unavailable, and navigator.vibrate is missing even for a short bit of haptic feedback. Take an app for a warehouse worker who scans barcodes with the phone's camera: on Android, a failed scan can be signaled with a vibration, but on iPhone there's no such signal, so the error has to be shown with a screen color or a sound. If the product needs a Bluetooth printer, or has to work with the screen locked, it's more honest to discuss a native app right away than to promise that the web will catch up soon. With no background sync, the whole model for saving data changes. I put operations into a queue in IndexedDB and send them at moments when the app is definitely running: when it comes back to the foreground and when the network reappears. Each operation gets an idempotency key, so that resending after a dropped connection doesn't create a duplicate: JavaScript let flushing = false; async function flushOutbox() { if (flushing) return; flushing = true; try { for (const job of await outbox.pending()) { await api.send(job.payload, { idempotencyKey: job.id }); await outbox.markSent(job.id); } } catch { // no network or the server returned an error: we'll try again on the next event } finally { flushing = false; } } The online event tells you the device has connected to a network, but it doesn't guarantee that a request will reach the server. For example, the phone picks up Wi-Fi on the subway, but there's no internet behind it yet, and the very first request fails. That's why the loop stops at the first error and waits for the next trigger: another online event, or the app returning to the foreground. The second weak spot is in the interface: if you don't show the user that some operations haven't been sent yet, they'll close the app believing everything went through. How a service worker fits into this scheme, and why it doesn't save you when the app is closed, is covered in Part 5 of this series. The Safari Version Is the iOS Version Everything above applies to WebKit in general, but on a specific phone the set of available features also depends on the iOS version. On a Mac, Safari updates separately from the system. On an iPhone, it doesn't work that way: the engine is built into iOS and updates only together with it. Safari 16.4 arrived with iOS 16.4, Safari 26 with iOS 26, and a user can't install a newer browser on an older system. Look at the device report in your web analytics (Matomo, Google Analytics, or your server logs): if a noticeable share of visitors come from iPhones running iOS 16.2, those people have no Web Push. Telling them to "update Chrome" won't help, because Chrome on such a phone runs on the same WebKit from 16.2. Only updating iOS can help. That's why I tie the test plan to iOS versions: on an iPhone, the browser brand says almost nothing about the engine version. I keep two reference points: the lowest version the product still promises to support, and the current one. Detecting the engine version from the user agent (UA) isn't a good idea either. The user agent is a string that a browser sends with every request to introduce itself to the site: its name, its engine, and the operating system. On iPhone it's almost useless: every browser has the word Safari in it, and the AppleWebKit number hasn't changed in years. If the condition that enables push depends on parsing such a string, I ask for it to be removed in code review. In code, check for capabilities, not versions: is there a PushManager (the object a site uses to subscribe to push) and is the app running in standalone mode, meaning launched from an icon on the Home Screen. If there's no PushManager and the app is open in a tab, show the install instructions. If the app is already installed and there's still no PushManager, the iOS version is too old, and you should offer the user a system update. If PushManager is there, show the notification bell. Safari's release cycle has a rhythm worth knowing. The major autumn releases bring what was shown at WWDC (Apple's developer conference in June), but for web apps on iPhone, a lot of the important changes arrive in spring, most often in the x.4 releases. Safari 15.4 in March 2022 brought more than 70 additions, among them <dialog>, :has() and support for icons from the manifest (though apple-touch-icon still takes priority in Safari). Safari 16.4 in late March 2023 added 135 features, including the Badging API and, of course, Web Push for Home Screen web apps. Safari 18.4 in spring 2025 brought Declarative Web Push, and 26.4 in March 2026 added WebTransport. The WebKit team acknowledges the pattern itself in its post about 26.4: "March has a way of bringing a lot of new things to WebKit — and this year is no exception." There are exceptions, too: 17.4 in March 2024 was modest in features, but it contained all the changes that Europe's DMA required. You can find out what the next release will bring in advance, and without any insider contacts. Can I Use and MDN show what Safari already supports, and the Standards Positions page shows where WebKit stands on new specifications. Safari Technology Preview on a Mac lets you try features months before release, though for iPhone it's only a rough guide: it doesn't fully reproduce iOS behavior. The Safari release notes and posts on the WebKit blog list the changes in each release point by point, and the blog also carries notes for every Technology Preview build. On the iPhone itself, in Safari's settings under Advanced, there are switches for experimental features: in iOS 16.4 the item was called Experimental Features, and in newer versions it was renamed Feature Flags. They're handy for trying out an API ahead of time, but before you investigate a bug a user has reported, reset them to their defaults. Otherwise you'll be reproducing the problem in a browser with settings that the user doesn't have. A Link from a Messenger Opens in the Wrong Place Many people reach a site without going through Safari at all: through a link in a messenger, in Instagram, in an email client or in a corporate app. Such apps open the site either in SFSafariViewController or in their own browser built on WKWebView, and in the second case you land in someone else's container, where the host decides what's allowed. There's usually no Add to Home Screen item there, the Share button may not exist at all, and a hint saying "tap Share" leads the person nowhere. For these cases, it's better to show a short text asking the user to open the link in Safari, along with a button that copies the address. Many messengers already offer their own "open in browser" control, but it's safer not to rely on it and to build your own. Debugging all this is a special kind of fun. Web Inspector on a Mac can attach to Safari tabs and to installed web apps, but it will only see someone else's WKWebView if the app's developer has explicitly allowed inspection: since iOS 16.4 there's an isInspectable property for that, and it's off by default. You most likely won't be able to attach your inspector to the browser inside a popular messenger. If a bug reproduces only "from the messenger," I first ask the person to open the same link in Safari, and a good share of such bugs disappear there, because they lived in the host's embedded window, which your service worker might never have reached. JavaScriptCore Does Things Its Own Way Suppose the page opened in the right place: in a Safari tab or in an installed app. Next, the code starts running, and here too things differ from what you know from Chrome. Because Chrome on iPhone runs on WebKit, it executes JavaScript not with the familiar V8 but with WebKit's own engine, JavaScriptCore. V8, the JavaScript engine in Chrome, has little in common with it beyond the language specification. JavaScriptCore has four execution tiers. First, the code is run by an interpreter called LLInt (low-level interpreter). When a function gets hot, meaning it's called or loops often enough, the engine compiles it on the fly, which is known as JIT (just-in-time compilation): first with the simple Baseline JIT tier, then with the optimizing DFG (data flow graph) tier, and finally with the most aggressive one, FTL (faster than light). The engine decides when to move up a tier based on an execution counter and on statistics about value types that it collects as it goes. In an ordinary interface, the difference between JavaScriptCore and V8 is barely noticeable. On heavy computation, the picture varies: one engine is faster on some tasks, the other on others. So it makes sense to profile for iPhone on an iPhone itself: measurements from Chrome DevTools tell you about V8's performance, not JavaScriptCore's. Lockdown Mode changes this picture more than any optimization does. It's an optional mode of extreme protection, introduced in iOS 16: the person turns it on themselves in Settings, under Privacy & Security, and the phone restarts. Apple built it for the very few people who may be personally targeted because of who they are or what they do. Apple says that in this mode some complex web technologies are blocked, so websites may load more slowly, web fonts may not display, and images may be replaced with a missing-image icon. JIT is unavailable in it as well: Apple says so directly in its conditions for browsers on third-party engines, discussed earlier. That means all JavaScript runs on the slowest tier only, the interpreter, and a page that flies for everyone else can be noticeably sluggish for such a user. So if a bug reproduces for only one user while everyone else on the same iOS version is fine, ask them whether Lockdown Mode is on (Safari shows a banner when it is). A site can't detect this itself, and Apple doesn't describe any way to. Ask before you spend a week hunting for a race condition in React: with slow JavaScript, the order of events changes, and an ordinary page starts to behave erratically. You don't need to work around the mode. What you need is an interface that stays usable with slow JavaScript, without web fonts, and with images that may fail to load. JavaScript syntax has its own release schedule, too. Lookbehind in regular expressions (a check of what comes before a match, a construct like (?<=...)) arrived in Safari only in 16.4, in spring 2023, together with Import Maps. Before that, a single such regex in a bundle caused a SyntaxError at parse time, and the user saw a white screen instead of one broken function. As far as I know, transpilers don't rewrite lookbehind for you. You either rewrite the expression by hand or raise your minimum supported iOS version to 16.4. structuredClone, on the other hand, has been available since 15.4, and a polyfill for it (code that imitates a missing function) only adds weight on modern iPhones. The Process Will Die, but the Data Must Survive So far we've looked at how JavaScript executes on iPhone. Now let's look at how long it lives: a page can vanish at any moment, without warning, and for your data that matters more than any difference between engines. It's time to return to WebKit2, which came up in the history of the engine. There, the interface and the page content live in separate processes, and your page is run by the Web Content Process. When the system runs low on memory, jetsam steps in, the iOS mechanism that frees memory by terminating processes, and it simply kills that process. This is especially likely when the user switches away from the app: they're filling in a form, get distracted by a phone call or switch to the camera to photograph a document, and while they're busy, the system removes the page left in the background from memory. The page gets no "about to close" event: neither beforeunload nor pagehide arrives, and everything it held in memory (variables, objects, the not-yet-submitted contents of a form) disappears at once. When the person comes back, they find the app starting from scratch, or an empty form on the screen. For any screen beyond a single short form, my takeaway is a simple rule: write the draft to IndexedDB as the user types, with no "Save draft" button, which people won't press anyway, because they don't know their app can be killed out from under them. JavaScript // The form the user is filling in const form = document.querySelector("#invoice"); // ID of the pending save. While the person is typing, we keep resetting it let timer; // The input event fires on any change to a form field, // that is, on every keystroke form.addEventListener("input", () => { // The person is still typing: cancel the save scheduled by the previous keystroke clearTimeout(timer); // Schedule a new save in 400 ms. If the field changes again before then, // the timer is reset, so the write happens once, after a pause in typing, // and not on every letter timer = setTimeout(() => { // Collect all the form's field values into a plain object like { fieldName: value } const data = Object.fromEntries(new FormData(form)); // Write to IndexedDB under the key "invoice", overwriting the previous draft. // drafts here is a small wrapper over IndexedDB (its code isn't shown in this example). // savedAt lets us show "there's an unsaved draft from 2:02 PM" when restoring drafts.put("invoice", { data, savedAt: Date.now() }); }, 400); }); This pattern is called debouncing: the write is postponed until there's a pause in typing, so the database isn't hit on every letter. The 400 ms delay means the last few characters typed before the process dies can be lost, and for an invoice I'm fine with that. Restoring matters more. You must never silently fill in a half-completed form. The person should see a line like "There's an unsaved draft from 2:02 PM" and choose for themselves whether to continue it or throw it away. Otherwise, sooner or later an old draft will land on top of a new invoice, and sync will take the blame. Two stubborn myths surround storage for PWAs on iOS: the "hard 50 MB limit" and "everything gets wiped after seven days." The first is long outdated: as I mentioned earlier, since iOS 17 WebKit calculates quotas as a percentage of disk size, an installed web app gets the same quota as a site in the browser, and Apple doesn't publish a fixed figure in megabytes. The second grew out of a real change, but one that affected only Safari. In March 2020, WebKit introduced a seven-day limit in ITP (Intelligent Tracking Prevention, Safari's protection against tracking): if someone uses Safari for seven days and never interacts with a site, everything the site wrote with scripts, from localStorage to IndexedDB and service worker registrations, gets deleted. But that rule was written for sites opened in Safari. An installed web app runs outside Safari, and in that same announcement WebKit wrote that it doesn't expect data in such an app to be deleted: that isn't what ITP is meant to do, and it would treat a case of deletion as a serious bug. So the "seven days" story shouldn't be applied to installed PWAs. None of this guarantees that data lives forever, because it can disappear for other reasons. When the device is low on storage, the system can evict a PWA's data regardless of ITP, and if the user deletes the app's icon from the Home Screen, the data goes away with the app. navigator.storage.estimate() gives only an approximate figure. WebKit grants navigator.storage.persist() requests based on heuristics, and one of them, according to WebKit's own description, is that the site is launched as a Home Screen web app. Anything you can't afford to lose should live on the server, and the local database should serve as a cache and a queue. Everything above applies to an app that's already on the Home Screen. But a PWA doesn't get its own storage in advance: it appears at the moment of installation, and every installation gets its own. So in Part 4 of this series, we'll tap Share, choose Add to Home Screen, and see what iOS creates at that moment: where the icon comes from, exactly when that fourth room we talked about above appears, and why the same site can be installed as a PWA twice, leaving you with two apps that know nothing about each other.

By Konstantin Shkurko

The "Just Read the Code" Myth Is Killing Your AI Workflow

Every Abstraction Leaks. Generative AI Is No Exception. Joel Spolsky gave us the Law of Leaky Abstractions years ago, and it lands hard on the newest layer in software development: generative AI. When you prompt a model to produce code, it builds an abstraction over the implementation. You get working functionality without touching the underlying logic. Great, until the day it breaks. So what do you do when the abstraction leaks? The conventional answer is simple: go back and read the code. Tudor Girba, creator of Glamorous Toolkit and the voice behind moldable development, argues that answer is a trap. When AI-generated code fails or needs to be understood, the right move is not heroic manual reading. It is reaching for deterministic, contextual micro-tools that make the system explainable on demand. Here is the uncomfortable part. AI code can follow unfamiliar patterns, skip the mental model a human author would have embedded, and ignore your team's conventions. That often makes it harder to read than human-written code, not easier. And the evidence that the abstraction is already leaking is piling up. The Numbers Are Not Subtle GitClear analyzed millions of changed lines of code from 2020 through 2023. Their finding: code churn, the share of lines reverted or rewritten within two weeks of being authored, is projected to double in 2024 against its 2021 pre-AI baseline. The mix of changes shifted too. "Added" and "copy and pasted" code grew relative to "updated," "deleted," and "moved" code. In plain terms, AI-assisted code tends to break the DRY principle and reads like the output of a short-term contractor rather than a senior engineer. Then there is plausible incorrectness. A peer-reviewed study by Kabir and colleagues looked at 517 ChatGPT answers to Stack Overflow questions. Fifty-two percent contained incorrect information, and users missed that misinformation 39 percent of the time. Why? Because 77 percent of the answers were verbose, and the polished prose masked the errors underneath. People still preferred the ChatGPT answers 35 percent of the time, high error rate and all. Capability is climbing, to be fair. On SWE-bench, Princeton's benchmark of 2,294 real GitHub issues, the best model at launch (Claude 2) solved just 1.96 percent. By mid-2025, purpose-built agent scaffolding reached about 65 percent on a curated 500-instance subset. That is roughly a 33x jump. It is also still one-third of real-world issues left unsolved, because real engineering means coordinating changes across many functions, classes, and files at once. The Verification Bottleneck Is the Leak in Its Purest Form Here is the asymmetry that should keep engineering leaders up at night. Generating 500 lines of code takes seconds. Verifying those lines for security holes or logic errors can take an hour. The volume of AI output is becoming too large for humans to review. That creates pressure to hand review over to the AI itself, which introduces a recursive trust problem: you are now trusting one non-deterministic system to check another. The very speed that makes AI attractive is exactly what makes its failures dangerous. A Taxonomy of How AI Code Fails Failure Mode What Happens Evidence Code churn amplification Lines reverted within two weeks at double the pre-AI rate GitClear, 153M lines DRY violation More copy-paste at the expense of refactoring GitClear Plausible incorrectness 52 percent error rate masked by confident language Kabir et al. Verification asymmetry Generation in seconds, verification in hours GitClear community Compositional failure Models stumble on multi-file coordination SWE-bench Review overload AI output exceeds human review capacity GitClear The Quiet Risk: Skill Atrophy There is a second failure happening in slow motion, and it is about people, not code. Automation bias is the human tendency to favor automated suggestions and overlook contradicting information, even when we are right. In coding, it shows up in two ways. Errors of commission: you accept an AI suggestion without checking it. Errors of omission: you miss a bug because you trusted the output. When the AI eventually fails, and you have lost the skill to diagnose it, you either blindly trust the next suggestion or abandon the tool entirely. Neither is good. GitHub says developers write code 55 percent faster with Copilot. GitClear's data suggests that speed arrives with a maintainability tax, a productivity-versus-quality paradox. The Nielsen Norman Group found generative AI lifts employee performance by an average of 66 percent, with the biggest gains going to the least-skilled workers. Helpful for output, worrying for growth: junior developers may lean on AI before building foundational skills, while seniors use it as a genuine multiplier. This is not a new story. Lisanne Bainbridge named it in 1983 with the "ironies of automation." Automation removes the routine practice that keeps skills sharp, yet the human is needed most precisely when the automation fails. Air France Flight 447 is the hard lesson: when the autopilot disconnected over iced sensors, pilots with degraded manual flying skills stalled the aircraft. Research on GPS navigation shows the same erosion of spatial memory. Passively accepting AI code risks the same slow loss of algorithmic reasoning. Why "Read the Code" Cannot Be the Fallback Anymore Girba's challenge rests on four points, and each one holds up. Scale defeats comprehension. Modern systems are too large and interconnected to understand by reading. AI generates code faster than any team can review it. The reading fallback simply cannot keep pace. AI code lacks authorial context. An LLM may use unexpected idioms and carry none of the design intent a human would embed. Two developers reading the same generated function can reach different conclusions about what it is for. Reading is ad hoc and unrepeatable. It does not scale across a team or across time. Traditional debugging assumes determinism. Breakpoints, step-through, and stack traces all assume the same input yields the same output. LLM output is non-deterministic. The same prompt can produce structurally different code each time, so reproducing a bug becomes unreliable. Multi-step agent workflows branch in ways a conventional stack trace never captures. Semantic equivalence hides errors. Two outputs can be semantically identical but syntactically different, which defeats diff-based and line-by-line debugging. The real question has shifted from "what does this line do?" to "what did this system actually do at runtime?" Reading alone cannot answer that. So Girba reframes the core skill. It moves from "the ability to read and write code" to "the ability to make systems explainable." Moldable Development and the Deterministic Toolkit Moldable development starts from one observation: developers spend most of their time understanding systems, not writing code. Every system is unique, so the tools used to understand it should be moldable, meaning customizable to the system in front of you. Instead of leaning only on generic IDE features, you build many small custom tools: views, inspectors, and queries tuned to your domain. The idea is that any question about a system can be answered through narratives made of contextual tools. Think data science, but pointed at your software. Glamorous Toolkit is the concrete implementation, an open-source environment for building custom inspectors and views for every object in your system. A query result renders as a table, a graph structure renders visually, a business rule shows its decision tree. All deterministic, all contextual. It now ships a built-in moldable agent harness with programmable agents and explainable agent actions, so you can understand what the AI did deterministically rather than taking its word for it. The Broader Ecosystem Is Already Here Time-travel debuggers. Tools like rr record an execution once and replay it deterministically, with identical address spaces, registers, and syscall data every run. Pernosco, built on rr, delivers omniscient debugging with instant navigation and dataflow analysis, and users report roughly 5x faster bug resolution, sometimes 100x. Replay QA explores builds, reproduces failures, and provides deterministic recordings with root cause analysis.Causal tracing. OpenTelemetry lets you understand a system from the outside by tracing the full path of a request. Jaeger implements compatible trace collection and visualization for distributed systems.AI-era observability. Langfuse offers open-source trace logging, cost tracking, and prompt management. LangSmith gives full trace visibility from a single trace up to production-wide metrics. Braintrust surfaces critical patterns in agent traces automatically. OpenTelemetry has standardized GenAI span conventions across OpenAI, Anthropic, and AWS Bedrock.Compiled AI frameworks. Stanford's DSPy compiles programs into optimized, deterministic pipelines, pulling the LLM out of runtime control flow entirely and replacing prompt-response guesswork with reproducible execution. The Behavioral Harness: Deterministic Tools in Production The most mature pattern for wiring deterministic tools into AI coding is the behavioral harness, a two-loop design. The inner loop gives coding agents deterministic feedback during development. The outer loop runs independent automated acceptance tests against the generated service. The inner loop combines guides (instructions that shape the agent) with sensors (deterministic checks that validate output). Contract-driven mocks wire OpenAPI and AsyncAPI specs into the app so downstream interactions are executable, letting the agent verify its work against defined responses, failures, and recovery paths. Thoughtworks draws a useful line between computational controls, run by deterministic tools, and inferential controls, which rely on model judgment. The strongest harnesses combine both, with behavioral checks producing feedback without an LLM judging correctness. Schema tightening is the first line of defense. An "examples validate" step checks payloads against the API spec before any test runs. Sensors then confirm contract conformance, verify HTTP values, and assert observable side effects. Keep guides and sensors aligned, and you get allowlist-style gating: the agent passes only if its output matches the spec. The Result That Changes the Economics In October 2026, Naresh Jain ran a controlled experiment with Specmatic and Thoughtworks: coding agents with and without a behavioral harness, 10 runs each, against 178 automated acceptance tests. Metric Sol (No Harness) Sol (Harness) Luna (No Harness) Luna (Harness) Accepted runs 10/10 10/10 3/10 10/10 Est. cost per run ~$0.41 ~$2.68 N/A ~$0.27 Avg time 6 min 51 sec 19 min 53 sec N/A N/A Read that Luna column again. The frontier model, GPT-6 Sol, passed everything with or without the harness. The cost-optimized GPT-6 Luna succeeded in only 3 of 10 unharnessed runs, then hit 10 of 10 with the harness, at about 33 percent lower cost per run than the frontier model ($0.273 versus $0.406). The takeaway is sharp. Deterministic validation layers can make a cheaper model match frontier reliability. Note the flip side too: the harness added roughly 7x cost to Sol with no quality gain, so harness value depends on the underlying model. Use it where it pays. Humans on the Loop, Not in It Kief Morris of Thoughtworks frames three postures: Humans outside the loop: vibe coding, no oversight.Humans in the loop: line-by-line inspection of every change.Humans on the loop: engineering the harness that validates the work. The "on the loop" model fixes the debugging bottleneck. When output is unsatisfactory, an in-the-loop human patches the artifact directly. An on-the-loop human changes the harness so future runs come out correct. The harness becomes a form of "executable attention," pointing you at the exact contract, response value, or interaction that broke, instead of asking you to read everything. Separate the "why loop" (outcomes) from the "how loop" (code artifacts), and human focus moves to outcome validation rather than implementation detail. The New Job: Harness Engineer The chain from abstraction to skill atrophy to leakage is not a prediction. It is measurable today in churn rates, error frequencies, and benchmark gaps. Girba's reframing reorients the whole problem. The question is not how to preserve our ability to read code. It is how to build systems that stay explainable without anyone having to read the code at all. There is a convergence happening. Moldable development says every question about a system should be answerable through contextual tools. The behavioral harness says deterministic sensors can provide targeted feedback without human inspection. The Specmatic and Thoughtworks experiment shows this is real, not theoretical. So the developer's role is shifting. Less code reader and writer, more harness engineer and system explainer: the person who designs the deterministic tools, specifications, and validation layers that keep AI-generated systems trustworthy. That is not a smaller job. It demands deep system understanding, just expressed through tooling instead of line-by-line reading. The investment priority follows directly. Teams adopting GenAI coding tools should invest proportionally in deterministic contextual tooling: time-travel debuggers, causal tracing, moldable inspectors, behavioral harnesses, and AI-era observability. When the abstraction leaks, and it will, the fallback should be a tool that answers the question, not a developer staring at unfamiliar code. References Glamorous ToolkitGitClear: Coding on CopilotKabir et al., ChatGPT Stack Overflow studyAre Harnesses Worth the Cost? (martinfowler.com)rr projectPernoscoOpenTelemetry observability primerLangfuse docsLangSmith docsDSPyGitClear: Durable Change Velocity 2026SWE-bench paperSWE-benchAutomation biasNielsen Norman Group: AI tools productivity gainsDeskillingAir France Flight 447Glamorous Toolkit siteReplay.ioLangfuse tracingSemantic equivalence studyMoldable DevelopmentJaeger docsBraintrust docsOpenTelemetry GenAI semantic conventionsOpenTelemetry GenAI conventions repoHumans and Agents (martinfowler.com)

By Viquar Khan

CORE

Spring Security JWT Validation Best Practices

A few months into a large-scale legacy-to-cloud migration for a Fortune 50 insurance carrier, I found something that had been sitting quietly in our codebase for a while: application code that was manually re-checking JWT claims — signature, issuer, audience, expiry — that Spring Security's OAuth2 Resource Server had already validated at the framework level, before the request ever reached that code. Nobody had done this carelessly. Each addition, on its own, looked like good security hygiene. That's exactly why it survived review for as long as it did. Why This Happens So Easily A JWT is a self-contained, signed structure carrying its own claims. Validating untrusted input by hand is a deeply ingrained instinct for engineers, and for good reason — it usually is the right instinct. The problem is visibility. When a developer writes if (claims.getExpiration().before(now)), a reviewer can point at that line and confirm the check exists. What's much harder to see is the framework machinery quietly doing the exact same job, correctly, one layer up. Spring Security's OAuth2 Resource Server support validates signature, expiry, issuer, and audience (when configured) in line with RFC 7519 and RFC 8725. Its JwtAuthenticationProvider runs these checks on every incoming token, and as the identity provider rotates its signing keys, it automatically re-fetches and re-validates against the new keys — no application code involved. Almost none of this is visible unless you've actually read the framework internals, which is exactly why teams tend to reimplement it instead of trusting it. The audience claim was the clearest example in our case. It identifies which resource server a token is meant for, and the framework exposes it as a single config property. We'd been treating it as something that needed hand-written comparison logic — when really, the value had just never been wired into configuration in the first place. The fix wasn't more code. It was one line of YAML. What We Changed We removed the manual validation entirely and let Spring Security's Resource Server own both authentication and authority mapping: YAML spring: security: oauth2: resourceserver: jwt: issuer-uri: https://login.microsoftonline.com/{tenant-id}/v2.0 audiences: api://{application-id} authorities-claim-name: roles authority-prefix: GRANT_ That block replaced manual signature verification, issuer checking, expiry checking, audience checking, and — on Spring Boot 3.3+ — the hand-written role-to-authority converter too. No JwtAuthenticationConverter, no custom Bean. The framework builds the authority mapping straight from the roles claim and applies whatever prefix you configure. (GRANT_ isn't a framework requirement, by the way — pick anything, as long as it matches what your @PreAuthorize expressions expect downstream.) The Objection We Had to Take Seriously This didn't get waved through. It took a few weeks of real back-and-forth, because the pushback was a legitimate security position, not just resistance to change: don't trust any single layer — verify independently, everywhere. Concretely: if issuer-uri got misconfigured — pointed at a stale tenant, or worse, something attacker-influenced — the Resource Server would faithfully validate every token against the wrong authority, and nothing in a minimal-code setup would catch that. A hardcoded, independent check plausibly would. Nobody disputed that the risk was real. The actual question was which layer should own catching it. Our answer: a hardcoded application-layer check and a misconfigured deployment value are different failure modes, touched by different people, on different release cadences. Duplicating validation in code doesn't actually protect against a bad issuer-uri — it just adds a second place that can also drift out of sync. Instead, we added a CI/CD gate that asserts the deployed issuer-uri and audience resolve to the expected tenant, and fails the pipeline on mismatch, on every deploy. Is that still "a second check"? Sure. But it fails differently, and that difference is the whole argument: a broken pipeline gate is a loud, visible build failure someone has to explicitly override. A stale hardcoded value in application code can sit silently wrong for months, because nothing forces anyone to look at it again. What I Can Honestly Say Held Up I want to be straight about this: since the change shipped, we haven't had a key rotation or a deployment misconfiguration that actually tested either mechanism in anger. So I can't tell you the CI/CD gate has caught a real incident, because it hasn't had to yet. What I can say: the manual validation code is gone, no duplicate signature/issuer/audience/expiry logic remains anywhere in the codebase, and in the months since, nothing attributable to the change has come up in review. That's evidence the change didn't introduce a regression — not proof it's been battle-tested under a live failure. Worth being honest about the difference. The Takeaway Treat hand-rolled token validation as a prompt to investigate, not an automatic red flag. In review, ask what the framework's resource-server config already checks before accepting new validation code as necessary.If a function exists purely to re-derive something the framework already validates, the fix is usually a config line, not a code change.Defense-in-depth concerns about misconfiguration are legitimate — answer them directly. Often the better answer is verifying config at deploy time (CI/CD gating), not re-validating tokens at runtime.The two approaches fail differently, and that's the actual argument for preferring the config-time check: a broken gate is loud, a stale hardcoded check is silent. This example is Spring Security and Entra ID, but the underlying review question — what does the framework already check before I add code that checks it again — travels well to any OAuth2/OIDC resource-server setup.

By Alka Nimje

Guess the Speed: Building a Family Internet Speed Guessing Game

In this blog post, we will see how I turned our family's speed test ritual into a family internet speed guessing game that you can play on your own Wi-Fi. Every family has that one argument. "The internet is slow again." "No, it is fine; you are just streaming in 4K." Nobody really knows who is right, because nobody knows the actual speed. So we started guessing before every speed test. It became a game, and the game became a website. Let's see how it works and how I built it. The author describes developing Guess the Speed, a multiplayer game that transforms family internet speed tests into a competitive activity. Players submit secret download and upload guesses before running a real speed test against Cloudflare's network, then earn points based on accuracy using place rankings, bonuses for close guesses, and a cutoff rule for large misses. The post details features including room codes for multiplayer phone participation, a TV-optimized display mode, live result gauges, and shareable results. It explains the technical implementation using Astro, TypeScript, Cloudflare Workers, and Durable Objects for managing game rooms, alongside privacy measures like automatic data deletion and cookieless analytics. The author reflects on lessons learned about focused game design, the appeal of real data, and privacy as a feature. Why I Built It Speed test night on the living-room TV became our family game night. Everyone shouts a guess before I press go. Here is the funny part. My kids get it right almost every time, and Dad loses on purpose. I started my career in performance testing, then moved into development, so I think I am allowed to be bad at guessing speeds. That is my excuse, anyway. At some point I realized this was a real game. It needed scores, secret guesses, and a proper winner. So I built it. How a Round Works A round takes about a minute. Here are the steps: Add players and pick a face for each one.Everyone locks in a secret download guess and a secret upload guess.Run a real speed test against Cloudflare's network.Compare the guesses with the result and score the closest players. That's it. No accounts, no sign-up, nothing to install. As shown below, the home screen keeps all of this on one page, with the live gauge in the middle and the player cards around it. A Sample Round (With Real Scoring) Let's ground this with one simple example. Say the speed test returns 240 Mbps download and 20 Mbps upload. Three players locked in their guesses: PlayerDownload guessUpload guessMom25022D&J20030Dad50050 Now the scoring rules kick in: Place points: first place earns 3 points, second earns 2, and third earns 1.Spot-on bonus: +1 point for each guess (download or upload) within 5% of the real value.50% cutoff: if your average miss is over 50%, you get no place points. Here is how our round shakes out: Mom is closest overall. Her download guess is only about 4% off, so she also gets the spot-on bonus. Total: 3 + 1 = 4 points.Mia lands second with 2 points.Dad guessed way too high. His average miss is far beyond 50%, so he gets 0 points. Dad loses on purpose, remember? In this round, the cutoff rule made sure he lost on merit too. The Features I am Most Happy With Once the basic game worked, I kept adding things that make it feel good in a real living room. Room codes for everyone's phones. Start a room, share a six-letter code or a QR code, and up to 12 players can join. Everyone guesses from their own phone. The server keeps all guesses private until the results are revealed, so nobody can peek and copy. The host also chooses which device runs the test. TV mode. This is a bigger gauge and bigger text, built for a couch ten feet from the screen. It supports fullscreen and uses a screen wake lock where the browser allows it, so the TV does not fall asleep mid-round. A live gauge with clear stages. You can follow the test through ping, download, and upload with live values and result cards. Watching the needle move is half the fun. Share the winner. The champion dialog shows the winner and the scores next to a Share result button. It uses the device share sheet or copies a link. Festive themes. Pick light, dark, or a seasonal theme, or leave it on Auto (by date) and let the calendar decide. Dev diagnostics. This one is for the curious. It shows the summary, latency samples, individual test requests, a bandwidth percentile, the test plan, and browser details. You can copy or download the latest run as JSON. Add ?dev=1 to the URL to open it. How It Is Built I kept the stack small and boring on purpose: Astro and TypeScript build the static site, which is served from Cloudflare Workers.@cloudflare/speedtest runs the real speed test in the browser.A GameRoom Durable Object manages the multiplayer rooms, using WebSocket hibernation so idle rooms cost almost nothing.A PlayStats Durable Object keeps the aggregate play counter.Game and scoring rules live in src/lib and are covered by Vitest tests. A few things I want to call out. Scoring is plain, testable code. Rules like the 50% cutoff and the tie-break are exactly the kind of thing that breaks quietly. Keeping them in src/lib with unit tests meant I could change the rules without fear. Durable Objects fit this problem well. One room is one object, and it holds the secret guesses until reveal. That is a natural fit for "a small piece of shared state that many phones talk to." Testing without waiting. Real speed tests take time, so I added a simulated mode. Append ?mock=1 for a short fake test. Use ?tv=1 to jump straight into the TV layout. To run it locally: Shell npm install npm run dev And to exercise rooms and the play counter, run the Worker locally: Shell npm run dev:worker Deployment is handled by Cloudflare Workers Builds. wrangler.jsonc runs npm run build, then npx wrangler deploy uploads dist as static assets. The Durable Object migrations run as part of that deploy. Everything below comes from the open-source repo at https://github.com/QAInsights/guessthespeed, so you can read the real code. Architecture at a Glance Astro and TypeScript build the static site, served by a Cloudflare Worker.worker/index.ts is the HTTP entry point. It creates rooms and upgrades WebSocket connections.GameRoom (worker/game-room.ts) is a Durable Object. There is one per room, and it owns the sockets and the stored state.src/lib/room.ts and src/lib/scoring.ts hold the rules as plain TypeScript functions with no Cloudflare APIs, covered by Vitest. The design rule I followed: the Durable Object is a thin shell. It loads state, calls a pure function, persists the result, and broadcasts. All the game decisions live elsewhere. One Durable Object Per Room A room code maps straight to an object with idFromName. Every phone that knows the code reaches the same instance, and every message for that room is handled in one place. That means there is no distributed locking to think about. Room codes are six letters from a 19-letter consonant alphabet, which gives roughly 47 million combinations. Creating a room means picking a code and asking its object to initialize itself. If the object already holds a room, it answers 409, and the Worker tries another code. TypeScript // worker/index.ts (simplified) for (let attempt = 0; attempt < 5; attempt += 1) { const code = generateRoomCode(/* crypto-backed random */); const hostToken = crypto.randomUUID(); const stub = env.ROOMS.get(env.ROOMS.idFromName(code)); const response = await stub.fetch("https://room.internal/init", { method: "POST", body: JSON.stringify({ code, hostToken, now: Date.now(), kind }), }); if (response.status === 201) return jsonResponse({ code, hostToken }, 201); if (response.status !== 409) return jsonResponse({ error: "room_creation_failed" }, 503); } return jsonResponse({ error: "room_code_unavailable" }, 503); The collision check needs no separate database. The object's own storage is the source of truth for whether a code is taken. The hostToken is a secret returned only to the creator. Presenting it on connect is what makes a socket the host. WebSockets That Survive Hibernation A room can sit idle for minutes while a family argues about the result, and I did not want to pay for an always-on object. The WebSocket Hibernation API lets the object be evicted from memory while the sockets stay connected. The catch is that anything held in memory is gone when it wakes up. Two things keep a room correct across hibernation: Game state lives in storage, not in fields. Each message handler starts with loadRoom().Per-socket identity lives in the socket attachment. After the first hello message, the object stores who this socket is with serializeAttachment. TypeScript // worker/game-room.ts (simplified) private async connectWebSocket(): Promise<Response> { if (!(await this.loadRoom())) return Response.json({ error: "room_not_found" }, { status: 404 }); if (!roomHasCapacity(this.ctx.getWebSockets().length)) return Response.json({ error: "room_full" }, { status: 503 }); const pair = new WebSocketPair(); this.ctx.acceptWebSocket(pair[1], ["room"]); return new Response(null, { status: 101, webSocket: pair[0] }); } private async handleHello(socket: WebSocket, message: Record<string, unknown>) { // ...validate clientId against a UUID pattern... const room = await this.loadRoom(); const attachment = { clientId: message.clientId, isHost: typeof message.hostToken === "string" && message.hostToken === room.hostToken, }; socket.serializeAttachment(attachment); this.sendState(socket, room, attachment); } The clientId is a random UUID the browser generates once and keeps in local storage. A returning phone sends the same ID, so the server can match it to its player after a dropped connection or a screen sleep. The reply to hello is always a full state snapshot, not a replay of missed events, so a reconnect cannot drift. One tradeoff worth naming: my per-socket rate limiter is an in-memory WeakMap, so it resets when the object hibernates. I treat the limits as a safety net, not accounting, so losing them on wake is acceptable. The Server Is the Referee Clients never change state directly. They send an action, and the server decides. TypeScript // worker/game-room.ts (simplified) const room = await this.loadRoom(); const result = applyAction( room, attachment.clientId, attachment.isHost, parsed, Date.now(), { onlineOwners }, ); if ("error" in result) { sendError(socket, result.error); this.sendState(socket, room, attachment); // resync the sender return; } await this.ctx.storage.put(ROOM_KEY, result.room); this.broadcastState(result.room); applyAction(room, clientId, isHost, action, now) returns either a new room or an error code. It takes now as a parameter instead of reading the clock, which makes it trivial to test. The room tests run actions in sequence and assert on the result, with no sockets involved. When an action is rejected, the sender gets the error plus the current state. That small habit fixed a whole class of stale-UI problems, because the client always snaps back to the truth. Keeping Guesses Secret If the server broadcast one shared state object, anyone could open dev tools and read everyone's guess. So the server never sends the raw room. It builds a view per recipient. TypeScript // src/lib/room.ts (simplified) const reveal = room.game.phase === "results" || room.game.phase === "champion"; players: room.game.players.map((roomPlayer) => { const { owner, ...player } = roomPlayer; delete player.guessedAt; return { ...player, guess: reveal || owner === clientId ? { ...player.guess } : { down: null, up: null }, mine: owner === clientId, }; }), TypeScript // worker/game-room.ts private broadcastState(room: Room): void { for (const socket of this.ctx.getWebSockets()) { const attachment = getAttachment(socket); if (attachment) this.sendState(socket, room, attachment); } } Each socket gets its own projection: your own guess, nulls for everyone else's, until the results phase. The same function strips internal fields like the owner and the guess timestamp. A room test asserts that other players' guesses stay hidden from both players and the host while guessing and testing, then appear at reveal. Cleaning Up Rooms With a Sliding Alarm Rooms should disappear after six hours of inactivity. My first version called setAlarm after every accepted action, which meant a storage write for every guess and every tap. A later commit replaced that with one rule: only schedule an alarm if none exists, and let the alarm re-arm itself. TypeScript // worker/alarm.ts export function alarmToSchedule( existingAlarm: number | null, deadline: number, ): number | null { return existingAlarm === null ? deadline : null; } TypeScript // worker/game-room.ts (simplified) async alarm(): Promise<void> { const room = await this.loadRoom(); if (room && Date.now() < room.updatedAt + ROOM_TTL_MS) { await this.ctx.storage.setAlarm(room.updatedAt + ROOM_TTL_MS); // still active return; } await this.ctx.storage.deleteAll(); for (const socket of this.ctx.getWebSockets()) { try { socket.close(1000, "Room ended"); } catch { continue; } } } The room stores updatedAt. When the alarm fires, it checks whether the room was active since it was scheduled. If yes, it sets a new alarm for the new deadline. If not, it deletes everything and closes the sockets. The result is a sliding time-to-live with at most one alarm in flight. Scoring as Pure Functions Scoring rules are exactly the kind of code that breaks quietly, so they live in src/lib/scoring.ts with no I/O. The core of it is a relative miss for each metric. TypeScript function metricMiss(guess: number | null, actual: number): number { if (!(actual > 0) || !Number.isFinite(actual)) return 1; return guess === null ? 1 : Math.abs(guess - actual) / actual; } // miss = (downMiss + upMiss) / 2 // bonus = Number(rounded(downMiss) <= spotOnPct) + Number(rounded(upMiss) <= spotOnPct) Looking back at our round: Mom's download miss is |250 - 240| / 240, about 4.2%, which is under the 5% spot-on limit. Her upload miss is 10%, so no second bonus. Place assignment skips players over the cutoff and handles ties the way a race does: tied players share a place, and the next place is skipped. JavaScript let place = 0; for (let index = 0; index < eligible.length; index += 1) { if (index === 0 || compare(eligible[index - 1], eligible[index]) !== 0) { place = index + 1; } eligible[index].place = place; eligible[index].placePoints = settings.placePoints[place - 1] ?? 0; } Two small details earned their own tests: Rounding before comparing. Misses are rounded to four decimals before ties are detected. Without that, floating-point noise decides who is tied.A zero or missing real speed. metricMiss returns a full miss instead of dividing by zero, so scores stay finite. The test file has cases for each rule: 3, 2, 1 points, an exactly 50% miss staying eligible, shared ties, the download-then-upload tiebreak, a player with no guess, and fewer than three players. TypeScript it("awards 3, 2, and 1 for first, second, and third", () => { const scores = scoreRound( [player("one", 100, 20), player("two", 90, 18), player("three", 70, 16)], actual, defaults, ); expect(scores.map(({ place, placePoints }) => [place, placePoints])).toEqual([ [1, 3], [2, 2], [3, 1], ]); }); Race Mode and the Server Clock Race mode changes the rule: the host starts the test, and players can guess while it runs. A guess locked in later earns a bigger share of the points, but a guess that misses the result earns nothing. The interesting design question is who owns the clock. Phone clocks drift and phones lag, so every timestamp comes from the Durable Object. Starting the test stores raceStartedAt: now, each guess stores guessedAt: now, and the result computes the timing bonus from the same server clock. TypeScript // src/lib/scoring.ts (simplified) const duration = endedAt - raceStartedAt; const fraction = duration === 0 ? 1 : Math.max(0, Math.min(1, (guessedAt - raceStartedAt) / duration)); const timingBonus = 0.5 + 0.5 * fraction; const total = Math.round(basePoints * timingBonus); The bonus ranges from 0.5 for a guess at the very start to 1.0 for a guess at the last moment. A table-driven test pins the edges and a rounding case. TypeScript it.each([ [0, 0.5, 25], [50, 0.75, 38], // 37.5 rounds up [100, 1, 50], ])("applies timing at %s%% of the test", (at, timingBonus, total) => { const [score] = applyRaceTiming( [baseScore], [{ id: "racer", guessedAt: at }], 0, 100, ); expect(score).toMatchObject({ basePoints: 50, timingBonus, total }); }); Applied to our round, with place points scaled to 30, 20, 10 in race mode: Mom's base is 30 plus a 10-point spot-on bonus, which is 40. If she locks in at 80% of the test, the bonus is 0.9, and she scores 36. Mia's base is 20, and locking in at 20% gives 0.6, so she scores 12. Waiting is rewarded, as long as you do not wait too long. There is one more leak to close. If players could watch the live speed climb, they would copy it in the last second. In race mode, the progress relay forwards only the step counters and drops the measurements. JavaScript const keys = race ? ["step", "steps"] : ["mbps", "pingMs", "step", "steps", "bytes"]; The relay also accepts progress only from the device assigned to run the test, and it is throttled to ten updates per second. Challenges and Lessons A few things that needed more than one attempt: Redundant alarm writes. Resetting the alarm on every action was correct but wasteful. The sliding alarm in section 7 fixed it.Host handover edge cases. Handing the host role to another phone sounds easy. The rules are: only the host can do it, not while a test is happening, not to an offline player, and not to the same device. The offline case shipped as a follow-up fix after a handover to a disconnected player left a room with no live host.Interrupted tests. If the device running the test reloads mid-test, the room can get stuck in the testing phase. A small pure function, interruptedTestStep, decides what the client does next: resend a result it already holds, or abort the round once.Abuse limits on a public endpoint. Messages are capped at 2 KB, each socket at 20 messages per second, rooms at 12 players, and WebSocket upgrades check the origin. Worker request bodies are size-bounded too.Testing real-time behavior without waiting. Because the rules are pure, most tests never open a socket. For the browser, a ?mock=1 flag runs a short fake speed test, so a full round takes seconds. What I would tell someone starting a similar project: Keep the Durable Object thin and move every decision into functions that take now as an argument.Never broadcast shared state when players should not see each other's data. Project a view per socket.Put the identity in the state in storage, and assume the object can wake up empty at any time.Prefer one self-rescheduling alarm over resetting a timer on every message. Privacy Decisions A game for families should not feel like a tracking machine, so I made a few choices early: In single-screen mode, settings, player names, and scores stay in your browser's local storage.Room names, faces, guesses, and scores are deleted after six hours without activity.The play counter stores totals only, and Cloudflare Web Analytics is cookieless.The speed test itself runs in your browser against Cloudflare's network. You can read the full details on the privacy page. Try It On Your Own Wi-Fi Here is how to play your first round: Head to https://guessthespeed.com/.Add everyone playing and pick a face for each person.Lock in your secret guesses (or have people join from their phones with the room code).Press go and watch the gauge.Check the scores and share the champion. If you want ideas for a full evening, there is a family game-night guide on the site. There is also a guide that explains how the real speed test works, and a 30-minute classroom activity for kids. What I Learned A few takeaways from this build: Small, focused games beat big ideas. The whole concept fits in one sentence, and that makes every decision easier.Real network data makes people care. A made-up number is boring. A real measurement on your own Wi-Fi starts arguments, which is exactly what a game needs.Privacy is a feature. Deleting rooms after six hours and keeping analytics cookieless cost me very little, and they make the game easier to recommend to other families. So that is Guess the Speed. A speed test, a few secret guesses, and a family that cannot stop arguing about the Wi-Fi. Which of your guesses would be furthest off: your download speed or your upload speed? Let me know in the comments below. Happy gaming!

By NaveenKumar Namachivayam

CORE