DZone Spotlight

The Policy That Started the Argument We were looking at a purple policy between ListingSold and StopListingFee. Read aloud, the flow expressed a simple business expectation: when a listing is sold, the applicable recurring listing fee should stop. The interesting question was where that policy belonged. If Billing owns it, Billing receives the fact that a listing was sold and determines what that means for its fee model. Listing reports the sale without naming the operation Billing should perform. If Listing owns the coordination, it requests that Billing stop the fee, which means Listing knows that capability exists. A third option is a separate process manager that receives the sale event and coordinates the response. At the level of a simple event-policy-command sequence, these versions can look almost identical. Once ownership is made explicit, the differences matter. Billing depends on the sale event’s contract in the first version. The coordinating component depends on Billing’s command contract in the others. Billing still validates changes to its own fee model in every version. Responsibility for detecting an incomplete outcome, such as a fee that continues accruing after a sale, also needs an explicit owner. We were still discussing the business rule, but we were now deciding how software responsibilities and contracts would support it. This was already a discussion about where coupling belongs and how it is managed. We had moved into architectural reasoning without changing the business outcome we wanted. I later discussed this on LinkedIn and encountered the objection that EventStorming is not software design at all. I disagree, but the objection helped me distinguish two claims. EventStorming can host software design. It does not follow that the arrangement of a board is sufficient justification for the boundaries of the resulting system. This article is about that gap. EventStorming Already Includes Design The official EventStorming material explicitly recognizes Big Picture, Process Modeling, and Software Design formats. The patterns collection also addresses collaborative design activities. EventStorming is not limited to collecting domain events on a long wall of paper. The title therefore makes a narrower claim than it might suggest. “Your Event Storming” means the model your group produced, with its particular clusters, policies, and arrows. That model may contain exploratory ideas, agreed decisions, and reasons for those decisions. What cannot be assumed is that the notation alone captures all the reasoning needed to justify the resulting software structure. A policy between two notes expresses a relationship: when this happens, that response should follow. Unless ownership and the relevant assumptions are also made explicit, the sequence leaves important questions unanswered. Who owns the rule? Which contract does the response depend on? What happens if the response is rejected or never completes? A Software Design session can address those questions. It can expose invariants, clarify responsibilities, and help participants explore alternative models. The policy discussion above is an example of that work. The problem is automatic inference: treating a cluster as sufficient reason for a service, a purple note as a prescribed message handler, or a temporal sequence as an execution topology. This is related to the architectural decisions a domain model leaves open. Modeling can establish constraints and support architectural choices without uniquely determining how the system should be deployed or operated. Those choices can be discussed during the workshop. They still need reasons beyond the appearance of the board. Same Flow, Different Ownership For this example, Listing owns the listing lifecycle, and Billing owns the fee lifecycle. A sold listing should no longer accrue the applicable recurring fee. The precise rule, especially when stopping accrual becomes effective, still needs clarification. For now, consider three ways to allocate responsibility for the response. Alternative A: Billing owns the reaction. Listing records the sale and makes a ListingSold event available across an agreed boundary. Billing receives that fact, interprets it using its own rules, and invokes the appropriate fee behavior. Listing does not need to name StopListingFee. Billing now depends on the meaning and structure of the event it consumes. The event publisher needs an explicit compatibility policy, although that does not require exposing Listing’s internal domain event unchanged. Billing keeps the relationship between sale facts and fee behavior within its own model. That is useful when the rule belongs to Billing and evolves with its charging policies. The design still needs to account for delivery failures, processing failures, and missed reactions. Suppose Listing records a sale but the corresponding fee remains active. Who detects that discrepancy, and who resolves it? Billing may own both the reaction and reconciliation, but that responsibility must be explicit. Successful publication alone does not establish that the business outcome is complete. Alternative B: Listing owns coordination. Listing requests StopListingFee from Billing. Listing therefore knows a capability Billing exposes. Billing remains responsible for validating the request and protecting its own rules. The request could be synchronous or queued. Either way, the coordinating component needs to distinguish acceptance from completion and decide how to handle rejection, an unknown outcome, or a response that does not arrive within the expected time. This option is justified only if Listing’s responsibility genuinely includes coordinating the fee change. Emitting the original event is not enough to establish that ownership. Nor does the statement “the sale is complete only when the fee stops” settle the question by itself. That requirement establishes a business outcome; deciding which component should be accountable for achieving it is still design work. Listing-owned coordination becomes more plausible when Listing already owns the broader completion process and can manage it without absorbing Billing’s fee rules. If it starts calculating charges or reproducing Billing’s eligibility logic, the coordination boundary is leaking. Alternative C: A process manager owns coordination. A separate component tracks the cross-component outcome and requests work from the responsible participants. Listing still owns the listing lifecycle. Billing still owns fee behavior. The process manager owns the progress of the interaction. This helps when a sale triggers several participants, steps have deadlines, or unresolved cases need escalation. The manager can record that the sale was observed, that a fee-stop request was issued, and whether completion was confirmed. That visibility has a cost. The manager needs durable state, recovery behavior, and a policy for its own failures. Its commands may use the same capabilities Billing already exposes; participants do not necessarily need a new interface simply because a process manager is calling them. For a single reaction, introducing this machinery may be harder to justify than making Billing’s existing responsibility explicit. DesignMessage crossing the boundaryContract knowledgeAccountability for completionBilling reactsListingSold eventBilling understands the published sale contractBilling can own reaction and reconciliation; this must be agreedListing coordinatesStopListingFee command and an appropriate completion signalListing understands Billing’s exposed capabilityListing tracks the outcome if coordination belongs within its responsibilityProcess manager coordinatesSale event, fee-stop command, and completion or failure signalsThe manager understands the contracts required by the processThe manager tracks progress and escalation; its own recovery must also be designed These alternatives allocate responsibility differently. They do not divide neatly into a coupled design and a decoupled one. Contract knowledge is one form of dependency. Billing understands the sale event in Alternative A. Listing understands Billing’s command contract in Alternative B. The process manager understands the contracts needed to coordinate Alternative C. Runtime availability is a separate concern. A synchronous call may require the responder to be available immediately. Buffered communication can allow the sender to proceed while the receiver is unavailable, provided the system can retain and eventually process the pending work. That changes the timing of the dependency rather than removing it. Release coordination is another concern. Consumers and publishers need compatible contracts, but they do not necessarily need simultaneous deployments. The relevant question is how those contracts can evolve without silently changing the business meaning. Moving the policy changes some of these dependencies. It does not automatically reverse all of them. Nor does its position prescribe a transport: an event does not require Kafka, and a command does not require HTTP. What would favor one option? If Billing independently owns the rule connecting a sale to fee accrual, Alternative A is a strong candidate. If Listing already owns a broader completion process that legitimately includes requesting the fee change, Alternative B may fit. If the interaction needs its own lifecycle, deadlines, and recovery across several participants, Alternative C becomes easier to justify. The business flow is an input to that decision. Ownership, contract obligations, and failure handling determine whether the chosen design can support it. The Missing Question Is Often About Time The sequence ListingSold → StopListingFee leaves an important question unresolved: when should accrual stop? The policy note alone does not specify that. It could be the effective sale time, when the sale is recognized by the business. It could be when Listing records the change, or when Billing processes it. Those moments can differ. If the contract carries timestamps, their meanings need to be explicit. For example, a published event might distinguish the effective time from the time the sale was recorded: JSON { "type": "ListingSold", "eventId": "...", "listingId": "...", "effectiveAt": "...", "recordedAt": "..." } These fields make the distinction visible. They do not establish which timestamp Billing should trust or what it should do with it. If Billing calculates accrual using an authoritative effective time, delayed delivery need not change the amount ultimately owed. But the system may already have issued an invoice, collected payment, or shown an outdated balance. It still needs rules for adjustment and for what the customer sees before the correction completes. A timestamp helps preserve the meaning of the fact. It does not remove the consequences of learning about it late. Suppose the business accepts a delay before the balance reflects a sale, provided the final calculation uses the correct effective date. An asynchronous reaction may satisfy that requirement. Suppose instead that no fee-related charge may be issued after the sale is accepted. That requires a stronger coordination or validation mechanism than receiving a sale event later. Merely replacing an event with a queued command does not provide that guarantee either. Corrections expose further decisions. If the sale was recorded incorrectly, should accrual resume? From which date? If an older sale notification arrives after a correction, how does Billing recognize that it no longer represents the applicable state? These questions concern the business meaning of the update as much as its delivery. This is where business outcomes beyond aggregate boundaries become relevant. A process spanning Listing and Billing does not automatically require one aggregate or one distributed transaction. Billing can protect a local invariant, such as refusing to calculate accrual beyond its recorded stop date, while the broader process still fails to establish that date correctly. The cross-component outcome needs explicit completion and recovery semantics. Compensation is another business operation, with its own rules and potential failures, rather than a database rollback. Eventual consistency does not by itself establish that the relevant rule was interpreted correctly or that unresolved work will be detected and repaired. Durable publication, retries, and idempotent handling support the design. They cannot supply a business rule the participants have not agreed on. A Boundary Still Needs a Reason An event on the board is not automatically an integration event. ListingSold may initially describe a fact inside the Listing model. Exposing that fact to another component creates a contract that needs deliberate semantics and an evolution policy. The published representation may differ from the internal event. Likewise, a cluster of activities does not establish an aggregate merely because the notes appear together. The grouping must be examined against the rules that must remain consistent. An aggregate does not prescribe a microservice, and a bounded context does not prescribe an independent deployment. These distinctions are part of why Domain-Driven Design does not select an architectural style. The ownership alternatives also exist inside a monolith. Listing and Billing could be modules in one application, with Billing reacting to an in-process event or receiving an explicit request through a module interface. The questions about rule ownership and contract knowledge remain. The execution guarantees may be quite different, however. A synchronous handler participating in the same database transaction can fail together with the originating operation. A background handler in the same application may run after that transaction commits and require durable delivery and recovery. Sharing a deployment does not settle those choices, just as separating deployments does not settle ownership. An independent deployment therefore needs its own justification. Billing might have a materially different scaling profile, need isolation from listing traffic, or require a separate security boundary. Its team might need genuine release autonomy. A runtime constraint might be difficult to satisfy within the existing application. These are reasons to evaluate separation, rather than automatic instructions to introduce it. The costs are concrete. An interaction that was previously local may now time out or complete without the caller receiving confirmation. Contracts need to evolve while different versions coexist. Observability must follow a business outcome across processes. Data previously updated within one transaction may require a different consistency strategy. Independent deployment can reduce some coordination costs while introducing others. I would not impose a fixed sequence of aggregates first, contexts second, services last. Domain learning and architectural constraints inform each other. A scaling or compliance requirement can change where responsibilities execute or force us to reconsider a proposed partition. It does not automatically change the meaning of the business model. When a boundary changes, the team should be able to explain which concern required the change. What I Would Record Beside the Wall I would capture a short decision record while discussing each significant policy, rather than leave all the reasoning for someone to reconstruct after the workshop. First, who owns the rule connecting the fact to the response? In this example, does Billing own the rule relating a sale to fee accrual, or does a broader business process require that outcome? Billing owns the fee mutation in either case. The separate question is who initiates the response and tracks its completion. Next, which contract crosses the boundary? Who maintains it, and what changes can its consumers tolerate? A note on the wall becomes an integration commitment only when the participants agree on the meaning and obligations it carries. The record should also identify what must remain consistent, the acceptable delay between steps, and the effective time used when information arrives late. It should say who detects an incomplete outcome, who resolves it, and what counts as evidence of completion. Acceptance scenarios make those decisions harder to leave vague. The official Extract Acceptance Tests pattern connects design sessions with more precise expectations about behavior. For this example, I would include a sale received after an invoice was issued, a correction received before an older sale event, and a fee-stop request whose completion cannot immediately be confirmed. The expected result in each case needs to come from the business rule. The record should remain short enough to maintain. Its purpose is to preserve the reasons behind the model, including decisions deliberately left open. Six months later, another engineer should be able to distinguish an agreed constraint from a convenient assumption. Closing The discussion began with ListingSold, a purple policy, and StopListingFee. Asking who owned the policy led us to contracts, coordination, consistency, and recovery. That was software design taking place through EventStorming. The workshop can help a team make those decisions, and the board can record them. What the event sequence cannot provide on its own is the justification for a particular allocation of responsibilities or a particular deployment boundary. Billing might own the reaction. Listing might coordinate it. A process manager might track a broader outcome. Choosing among them requires knowing the rule, the timing requirements, and who is accountable when the expected response does not happen. The wall can show the flow. Your design must explain who owns each decision and what happens when the flow breaks. References and Further Reading Introducing EventStormingEventstorming PatternsExtract Acceptance TestsWhere Does Software Architect Fit in Domain-Driven Design?Modern Software Architecture (Part 2): Coupling Is Where Architecture Actually HappensDomain-Driven Design Beyond Aggregate BoundariesDomain-Driven Design Is Not an Architecture

By Alireza Rahmani Khalili CORE

Executive Summary Offline-first mobile systems treat local storage as an active source of application state rather than a disposable cache. That design improves responsiveness and preserves core behavior during long network outages, but reconnection creates a distributed-systems problem as local mutations may now compete with server updates or changes from other devices. Android's official offline-first guidance recommends a local data source for repositories that use the network and describes queued synchronization with persistent work and retry behavior. SQLite write-ahead logging can provide efficient local transactional durability, but WAL itself does not resolve cross-device conflicts. A reliable synchronization design therefore requires explicit versioning, deterministic merge rules, idempotent delivery, and observable reconciliation rather than a simple "upload the latest row" strategy. Reconnection Is a Versioning Problem Consider a record edited locally while the device remains offline for 12 hours. During that interval, the server may accept another update to the same record. Both versions can be valid descendants of an earlier state, so network recovery does not establish which value is semantically correct. The central question is not which copy arrived last, but whether one update causally follows the other or whether both happened concurrently. A local mutation log makes that distinction manageable. Each mutation can carry an operation identifier, entity identifier, base version, logical metadata, and payload. The local state change and the queued operation should be committed atomically so a crash cannot preserve one without the other. A compact operation can resemble: Plain Text op = { id: "op-8f31", entityId: "task-42", baseVersion: 17, type: "SET_STATUS", value: "DONE", occurredAt: 1792449120000 } On synchronization, the server checks the base version before applying the operation. A mismatch indicates that another accepted change advanced the entity and that normal application should stop until a merge rule runs. Plain Text apply(op, current) { if (op.baseVersion != current.version) return conflict(op, current) current.status = op.value current.version++ return current } This resembles optimistic concurrency control, but the important mobile property is durability across process death, retries, and extended disconnection. Android guidance explicitly describes queued writes for offline-first applications, while SQLite WAL records committed transactions in an append-oriented log and permits readers to continue while writes are committed. WAL improves local persistence characteristics where the mutation log supplies synchronization semantics. Conflict Resolution Must Match Data Semantics Last-writer-wins is attractive because it is cheap and deterministic. Cloud Firestore documents use last-write-wins behavior for multiple offline changes to the same document. A comparable custom merge can select the value with the greater timestamp: Plain Text merge(local, remote) { if (local.updatedAt != remote.updatedAt) return local.updatedAt > remote.updatedAt ? local : remote return local.replicaId > remote.replicaId ? local : remote } The simplicity hides a correctness risk. Device clocks can drift, timestamps can tie, and a later timestamp does not necessarily represent a more important business action. LWW works best for fields where overwriting is acceptable, such as a display preference, and poorly for additive or irreversible business events. A production rule also needs a deterministic tie-breaker, such as a stable replica identifier, so every participant reaches the same result. Vector clocks solve a different problem by representing causality rather than physical time. Amazon's Dynamo paper associates versions with node-counter pairs and uses clock comparison to determine whether one version descends from another or whether branches are concurrent. A simplified comparison can flag concurrency when neither clock dominates: Plain Text dominates(a, b) { return everyKey(k -> a[k] >= b[k]) && anyKey(k -> a[k] > b[k]) } conflicts(a, b) { return !dominates(a.clock, b.clock) && !dominates(b.clock, a.clock) } This preserves information that LWW discards, but metadata grows with participating replicas unless compaction or bounded identity schemes are introduced. Dynamo explicitly treats concurrent branches as versions requiring reconciliation, illustrating an important principle stating causality detection and semantic merging are separate responsibilities. CRDTs move more reconciliation into the data type. Shapiro and coauthors formalized state-based and operation-based CRDTs that converge when their required algebraic and delivery conditions hold. A grow-only set is the simplest example because additions commute and state merge is set union: Plain Text applyAdd(state, op) { return union(state, setOf(op.value)) } mergeGrowOnlySet(a, b) { return union(a, b) } No accepted addition is lost, and merge is associative, commutative, and idempotent. Operation-based CRDTs can transmit smaller mutations instead of full state, but they require stronger delivery properties. Automerge demonstrates the broader local-first model by recording changes and merging concurrent edits without a central coordinator. CRDTs are powerful for collaborative or additive data, but metadata, tombstones, history retention, and domain constraints can increase storage and implementation cost. Server-authoritative reconciliation remains appropriate when the backend owns invariants that cannot be decided safely on a disconnected client. Inventory allocation, authorization state, settlement, and policy enforcement commonly require server validation. In that model, offline mutations express intent, while reconnection submits intent against the current authoritative state: Plain Text reconcile(op, current) { if (!policy.allows(op, current)) return rejected(current) return applyValidated(op, current) } The server may accept, transform, reject, or return a conflict representation that preserves enough context for deterministic recovery. Couchbase Lite exposes conflict-resolution mechanisms for replicated documents, reinforcing that replicated state needs an explicit policy rather than an accidental overwrite. Delivery Must Be Idempotent and Observable Retries are inevitable after ambiguous failures. A request may reach the server even when the response never reaches the device, so replaying the same mutation must not duplicate its effect. Stripe's API documentation describes idempotency keys specifically for safely retrying requests. A client-generated operation identifier can become that key: Plain Text op.idempotencyKey = uuid() save(op) sync(op) { existing = dedup.find(op.idempotencyKey) if (existing != null) return existing.response result = apply(op) dedup.save(op.idempotencyKey, result) return result } The deduplication record and business mutation should participate in the same transactional boundary where possible. On server systems that also publish downstream events, a transactional outbox can persist the state change and event record together, where Debezium documents the outbox pattern as a way to avoid inconsistencies between stored state and emitted events. Correctness still competes with product and operational costs. Automatic LWW minimizes interaction and metadata but may silently discard meaningful work. Vector clocks preserve concurrency information but add version metadata and reconciliation paths. CRDTs can improve convergence and offline collaboration but may increase stored history, bandwidth, and implementation complexity. Server-authoritative reconciliation protects invariants but can surface rejected offline work after a long delay, creating a difficult user experience. No single policy fits every field, so conflict behavior should be defined at the domain level rather than globally. Production controls should make synchronization behavior inspectable. Deterministic merge functions should produce the same output for the same inputs. Reconciliation endpoints should return authoritative state, accepted operation identifiers, and explicit conflict metadata. Unresolved conflicts should be observable through metrics and diagnostics rather than disappearing into retry loops. Automated tests should exercise duplicated operations, reordered delivery, clock skew, stale base versions, concurrent edits, deletion-versus-update races, process restarts, and repeated reconnects. Background synchronization should also use bounded retries and connectivity constraints where Android WorkManager supports network constraints and retry with backoff for persistent background work. Conclusion An application returning after 12 hours offline does not merely need to "sync the newest data." It needs to establish causality, preserve durable intent, prevent duplicate effects, and resolve concurrent state according to domain semantics. LWW is useful when overwrite is acceptable, vector clocks expose concurrency, CRDTs encode convergence into suitable data structures, and server-authoritative reconciliation protects centralized invariants. The strongest offline-first systems combine those techniques selectively, persist operations transactionally, attach stable idempotency keys, expose reconciliation outcomes, and test failure ordering as aggressively as ordinary business logic. Network recovery should trigger a deterministic protocol, not a race between copies. That distinction turns offline synchronization from a fragile retry mechanism into a controlled consistency model.

By Uthej Mopathi CORE

AI-generated pull requests are moving from experimental workflow to routine software delivery. Coding agents can now plan changes, edit multiple files, run tests, and submit pull requests with limited human intervention. A 2025 study of 567 Claude Code pull requests across 157 open-source projects found that 83.8% were eventually merged, and about 55% of merged changes required no further modification. A broader 2026 study analyzed more than 302,000 verified AI-authored commits from 6,299 repositories. The practical question is therefore no longer whether generated code can enter production. The harder question is what happens after the merge, when readability problems, architectural drift, latent defects, and future maintenance costs become part of the codebase. Faster Generation Changes the Review Equation AI assistance does not automatically imply lower quality code. A GitHub randomized controlled study involving 202 experienced developers found that Copilot-assisted implementations performed better on the study's functional tests and received modestly higher ratings for readability, reliability, maintainability, and conciseness. That evidence matters because technical debt should not be treated as an unavoidable property of machine-generated code. Production repositories, however, create conditions that controlled coding tasks cannot fully reproduce. The 2026 large-scale study of AI-authored commits identified 484,366 introduced issues where 89.3% were code smells, and more than 15% of commits attributed to every assistant studied introduced at least one detectable issue. Most importantly, 22.7% of tracked AI-introduced issues were still present at the repository's latest analyzed revision. The same research also found that AI-authored commits can remove existing smells, showing that generated code can both create and repay debt. That distinction changes the engineering problem. The central risk is not generation itself but generation speed combined with weak validation. DORA's 2025 research describes AI as an amplifier that magnifies existing organizational strengths and weaknesses. Earlier DORA analysis also associated increased AI adoption with larger change batches, slower review, and weaker delivery stability. Faster code production therefore raises the value of disciplined review rather than reducing it. Debt Begins at the Merge Boundary Technical debt becomes an organizational liability when a change is accepted into a maintained branch. The model that produced the code cannot own an incident, negotiate a migration, preserve a service level objective, or explain an architectural compromise six months later. Accountability must therefore follow the repository and the component, not the mechanism that generated the diff. This makes the pull request author field an increasingly weak representation of ownership. A generated change may carry an agent identity, a bot account, or a human account that delegated most implementation work. None of those identities answers the operational question of which engineering group is responsible for the behavior after deployment. Repository ownership rules provide a stronger boundary. GitHub CODEOWNERS can automatically request reviewers for designated paths, and protected branch settings can require approval from a code owner before merge. Required status checks can additionally prevent merging until defined CI checks pass. A policy can remain simple: Plain Text /src/main/java/com/acme/payments/ @payments-platform /src/main/java/com/acme/identity/ @identity-platform The important property is not the syntax. The important property is that an AI-generated payment change still reaches the team accountable for payment correctness, while an identity change still reaches the team accountable for authentication and authorization. Provenance may influence review depth, but provenance should never replace domain ownership. Human Review Becomes More Valuable, Not Less Higher generation throughput can create pressure to automate review with another model. AI review is useful for summarization, repetitive checks, suspicious patterns, and localized suggestions, but available evidence does not support treating it as equivalent to accountable human review in every context. A 2026 study covering 278,790 review conversations across 300 mature open source projects found that human reviewers exchanged 11.8% more review rounds when handling AI-generated code than human-written code. Human reviewer suggestions were adopted at a much higher rate than AI-agent suggestions, 56.5% versus 16.6%, and more than half of unadopted AI suggestions were either incorrect or addressed through alternative fixes. Another study analyzing more than 22,000 comments from AI-based GitHub review actions found substantial variation in effectiveness and reported that concise, contextual, hunk-level feedback was more likely to produce code changes. Current GitHub documentation explicitly states that Copilot code review does not replace human code review. GitHub also supports configurations in which Copilot approvals can count toward merge requirements, making repository policy an important governance decision rather than a tooling default. For critical components, AI review is better positioned as an additional signal before accountable approval, not as the entity carrying final ownership. Quality Gates Must Follow Risk A generated pull request should not receive weaker controls because tests pass. Passing tests establish only the behavior covered by those tests. Technical debt often appears elsewhere as duplicated logic, boundary violations, unnecessary abstractions, missing failure handling, insecure defaults, excessive coupling, or code that satisfies the immediate task while conflicting with local design conventions. CI can convert some of those conventions from reviewer memory into executable policy. An AI-assisted label, for example, can trigger additional checks without requiring a separate delivery pipeline: YAML ai-change-gate: if: contains(github.event.pull_request.labels.*.name, 'ai-assisted') steps: - run: ./gradlew test integrationTest - run: ./gradlew check - run: ./scripts/verify-architecture.sh The architecture check can encode repository-specific constraints such as forbidden dependencies, package boundaries, or service-layer access rules. The AI label is only a routing signal where the underlying controls should be based on change risk. A generated documentation correction does not require the same scrutiny as an authorization change, while a human-written authorization change should receive the same high-risk checks. GitHub rulesets support required checks and required reviews as merge conditions, allowing these policies to be enforced rather than documented as optional guidance. The Useful Metric Is Rework Counting generated lines, completed pull requests, or acceptance rate measures production volume, not maintenance quality. A more meaningful evaluation connects AI-assisted changes to downstream engineering cost. Review rounds, escaped defects, rollbacks, post-merge rework, issue survival, architecture violations, and time spent modifying recently merged code reveal whether higher implementation throughput is creating future work. This framing also avoids a false binary between "AI code" and "human code." The large-scale debt study found both issue introduction and issue removal in AI-authored commits. The relevant objective is therefore not minimizing AI involvement, but minimizing unmanaged debt while preserving useful productivity gains. Ownership should remain attached to the team that accepts and operates the change, while automated gates provide evidence that acceptance criteria were actually satisfied. Conclusion AI-generated pull requests change who produces code, but they do not change who must maintain software after deployment. Technical debt cannot be delegated to a model, hidden behind a bot identity, or treated as the future responsibility of an unspecified reviewer. The durable ownership boundary is the engineering team responsible for the affected component, supported by required human approval, executable quality gates, repository ownership rules, and measurements that expose downstream rework. AI can accelerate implementation and can even remove existing debt, but merge authority must remain tied to accountable engineering ownership. When generation becomes nearly instantaneous, disciplined acceptance becomes the scarce resource, and that acceptance is where responsibility for technical debt belongs.

By Uthej Mopathi CORE

Introduction: The Layer Nobody Designs AI agents are evolving from simple prompt-response systems into stateful, long-running applications that can reason, execute tasks, and interact with external systems. Yet, most agent architectures are driven by models, tools, memory, and orchestration — not runtime infrastructure that can run agents reliably. Unlike traditional applications, stateful AI agents need execution context to run (and recover from failures), concurrent tasks, and the ability to resume workflows without losing progress if they fail, etc. These capabilities cannot be provided by prompts or orchestration frameworks alone — there needs to be a runtime layer. This is where Agent Substrate becomes relevant. It provides a runtime foundation for stateful agent execution so that developers can separate the intelligence of the agent from the infrastructure that runs it (aka runtime). A closer look at stateful AI agents reveals why a dedicated runtime layer is essential, the architectural challenges it must address, and how Agent Substrate fits into the broader agentic AI technology stack. What Makes Agent Workloads Different AI agents might look like traditional workloads, but their execution patterns challenge the fundamental assumptions of application infrastructure. Request-driven services or batch jobs cannot run forever; agents can make random decisions, interact with other systems, and change their own state continuously. Five characteristics that make their runtime requirements fundamentally different. Long-lived, mostly waiting: An agent task can take minutes, hours, or days, and most of that time is spent waiting on a model, a tool, or a human. Holding a process open for the whole duration is wasteful and fragile. Requirement: Execution must be able to pause, release resources, and resume.Non-deterministic: The same input can take a different path each run. You can’t recover by re-running the code and expecting the same result, so what the agent did and decided has to be recorded as it happens. Requirement: Durable records of steps and outcomes, not just final state.Side effects on the real world: Agents send emails, move money, and modify systems. A naive retry after a crash can repeat an action that already happened. Requirement: Know exactly which steps completed, and make retries safe.State is the product: Memory, plans, and open tasks accumulate and are read and rewritten constantly. This isn’t a cache you can throw away; losing it means losing the agent. Requirement: State that is durable, versioned, and structured enough to tell current from stale.Many actors, shared state: Agents act for users, hand work to other agents, and collaborate with humans. Without boundaries, they overwrite each other, and nobody can say who decided what. Requirement: Isolation, access control, and an audit trail. Stateful AI agents need the durability of a database, the recoverability of a workflow engine, and the isolation of a multi-tenant platform, all integrated into a coherent execution environment. While traditional infrastructure offers these capabilities independently, agent workloads demand them together. This is the architectural gap that an agent runtime layer must address. The Lifecycle of a Stateful AI Agent Traditional applications run in predictable request-response cycles, but stateful AI agents run through dynamic long-running workflows. Their execution involves multiple reasoning steps, tool calls from the outside world, human approvals, and periods of inactivity. A typical stateful agent lifecycle involves six stages: Initialize: Receive a task, establish execution context, and create an initial state.Execute: Perform reasoning, make decisions, invoke tools, and update execution state.Persist: Save progress, intermediate results, and relevant context at durable checkpoints.Suspend: Pause execution while waiting for an external event, tool response, or human approval.Resume or Recover: Continue from the last valid checkpoint when the required event arrives or recoverable failures occur, avoiding duplicate side effects.Complete: Finalize the task, record the outcome, and retain relevant execution history for auditability. Importantly, this lifecycle is not strictly linear. An agent may repeatedly execute, persist, suspend, and resume before reaching completion. Failures may also trigger retries, recovery, or escalation. The runtime layer is responsible for managing these transitions reliably, ensuring that agent execution remains durable, observable, and recoverable without requiring the application developer to implement the lifecycle infrastructure from scratch. Why Traditional Infrastructure Is Not Enough Traditional infrastructure provides many of the building blocks needed to run AI agents, but no single piece of it covers the entire lifecycle of a stateful agent. The table below shows where traditional infrastructure falls short. Need Typical tool Why it is not enough for agents Preserve execution progress Stateless service Handles requests, but does not inherently retain agent state or progress across steps Isolated execution Containers Provide process isolation, but not durable execution or recovery Asynchronous processing Queue Moves work reliably, but does not manage full agent context, memory, or state transitions Durable persistence Database Stores data, but does not coordinate execution, retries, or recovery Orchestration and retries Workflow engine Provides strong primitives, but still needs integration with agent memory, policies, and model-driven behavior The real challenge is not the absence of infrastructure primitives. It is the need to combine them into a coherent runtime layer that can preserve state, resume execution, recover from failure, and enforce policy across long-running agent workflows. Designing the Runtime Layer for Stateful AI Agents A stateful AI agent needs more than compute resources and an execution environment. It also needs a runtime that preserves execution state, handles long-running processes, recovers from interruptions, and efficiently allocates resources. Unlike traditional application runtimes, which handle processes and requests only, an agent runtime has workloads that are frequently switching between active execution and waiting for external events. A well-designed agent runtime should address five core capabilities. These capabilities lay the foundation for running stateful agents reliably without requiring every application to build its own execution infrastructure. Runtime capability Responsibility State persistence Preserve execution context, memory, and intermediate progress across interruptions Lifecycle management Manage agent creation, execution, suspension, resumption, and termination Execution isolation Run agents in secure, isolated environments Resource efficiency Release compute resources when agents are idle and restore them when needed Recovery and continuity Restore saved execution state after interruptions and support reliable continuation Agent Substrate is an open-source runtime for running stateful workloads (e.g., AI agents) on Kubernetes. Instead of treating each agent as a container permanently running, it uses an actor-based execution model where instances of agents can be suspended when idle and resumed when needed. Its architecture combines: Actor lifecycle management: Tracks actor instances and manages their suspension and resumption.Checkpointing and restoration: Preserves process memory and writable filesystem state through snapshots.Worker pools: Maintains available compute capacity for scheduling and restoring actors.Execution isolation: Supports sandbox technologies such as gVisor and microVMs.Kubernetes integration: Uses Kubernetes for underlying worker provisioning and infrastructure management. Separating Reasoning, Orchestration, and Runtime Infrastructure A reliable agent architecture separates three responsibilities: what the agent decides; how its work is coordinated; and where its execution is maintained. They are tightly coupled, but they address different problems (though they interact). Layer Responsibility Examples Agent reasoning Determines what actions to take, selects tools, and interprets results LLMs, LangGraph, CrewAI Execution orchestration Coordinates tasks, dependencies, retries, approvals, and workflow transitions Workflow engines, agent orchestrators Runtime infrastructure Manages execution environments, state persistence, isolation, suspension, and recovery Agent Substrate, Kubernetes-backed runtime services The reasoning layer decides what to do next; the orchestration layer coordinates when and how that action is executed. The runtime infrastructure ensures that execution can continue reliably, even when agents become idle, encounter failures, or require additional resources. Where Agent Substrate Fits Traditional infrastructure provides the foundation for running applications, but stateful AI agents need a runtime layer to manage execution continuity, isolation, and resource utilization. Agent Substrate is an open-source Kubernetes-based runtime that addresses these requirements. Instead of treating each agent as a running container, it uses actor-based execution so stateful workloads can be suspended when idle and restored when execution needs to continue. Agent Substrate introduces several capabilities that align with the runtime architecture discussed earlier: Actor-based execution environments: Each agent workload is represented as an actor with its own execution context, enabling independent management of stateful instances.Actor lifecycle management: Coordinates actor creation, placement, execution, suspension, resumption, and termination.Suspension and resumption: Allows inactive actors to be suspended and restored when new requests arrive, reducing the need for continuously allocated compute.Memory and filesystem preservation: Uses snapshots to capture process memory and writable filesystem state, enabling execution to continue across suspension and restoration.Worker pools and resource multiplexing: Maps a large number of stateful actors onto a smaller pool of available workers, improving infrastructure utilization.Isolation and request routing: Uses sandbox technologies such as gVisor and microVMs to isolate execution, with ingress routing and controlled network access to manage communication securely. Mapping Agent Substrate to the Runtime Architecture Runtime capability How Agent Substrate addresses it State persistence Process memory and filesystem snapshots Lifecycle management Actor creation, suspension, resumption, and termination Execution isolation Sandboxed actors using gVisor or microVMs Resource efficiency Worker pooling and actor multiplexing Recovery and continuity Snapshot restoration and resumable execution Request management Actor-aware routing and network mediation Why This Matters Consider an AI agent that initiates a task, invokes external tools, and then waits for human approval. In a conventional deployment, its execution environment may remain allocated throughout the waiting period. With Agent Substrate, the actor can be suspended, its execution state preserved, and its compute resources released. When execution needs to continue, the actor can be restored onto an available worker. This separation allows many stateful workloads to be managed without maintaining a dedicated running environment for every idle agent. Agent Substrate does not replace agent frameworks or workflow orchestration engines. It provides the execution infrastructure; frameworks do reasoning and application logic, while runtime does isolation, state preservation, scheduling, and continuity of execution. Real-World Use Case: Stateful AI Incident Investigation Various use cases of stateful AI agents where Substrate can be useful are – Use case How Agent Substrate helps Implementation complexity AI incident investigation agent Preserves investigation context while waiting for telemetry or human approval Medium AI coding agent Maintains isolated development environments across coding sessions Medium AI customer support agent Preserves session state during long-running interactions Low–Medium AI infrastructure remediation agent Suspends while awaiting approval and resumes with its execution context Medium–High AI research agent Preserves intermediate work across extended research sessions Medium Implementation Details: AI Incident Investigation Consider an AI incident investigation agent that diagnoses infrastructure failures. When an alert fires, the agent collects telemetry, looks at logs, correlates recent deployments, and builds an investigation context. Unlike a request-response application, the investigation may take several minutes or hours. The agent might have to wait for monitoring data, an engineering response, or human approval before continuing. Keeping a dedicated execution environment running during this wait time is unnecessary. Restarting the agent, however, requires reconstructing its investigation context and intermediate results. An agent runtime (e.g., Agent Substrate) solves this problem by providing a stateful execution environment that can be suspended and restored. For example: Initialize: An infrastructure alert starts an investigation actor.Execute: The actor collects logs, metrics, and deployment information.Persist: Investigation context and intermediate findings are retained in the actor's execution state.Suspend: The actor is suspended while awaiting additional information or human approval.Resume: When new information becomes available, the actor is restored and continues using its preserved context.Complete: The investigation produces a diagnostic summary and recommended remediation actions. Step 1: Build a Minimal Incident Investigation Service Create a Python or Go application that exposes: endpointpurpose POST /investigate Start an investigation and save the incident context GET /status Return the current investigation state POST /evidence Add new telemetry or investigation findings POST /complete Finalize and summarize the investigation Step 2: Maintain Investigation State The actor could preserve a state object similar to: JSON { "incident_id": "INC-1042", "service": "payment-api", "status": "waiting_for_approval", "investigation": { "error_rate": "12%", "suspected_cause": "recent deployment", "deployment": "v2.4.1" }, "approval_required": true, "recommended_action": "review rollback" } This represents application-level investigation context. The actor must actually load and update this data for the demo to demonstrate preservation. Step 3: Deploy the Service as an Agent Substrate Actor Package the application as an OCI image and create an Agent Substrate actor template configured with an appropriate sandbox, worker pool, and full-state snapshot policy. The existing counter example provides a useful reference for configuring an actor template. Step 4: Demonstrate Suspension and Resumption Once the actor has stored its investigation context, manually suspend it: Shell kubectl ate suspend actor incident-1042 -a incident-demo When a new request arrives through the configured actor router, the suspended actor can be activated and its state restored. You can also use the explicit resume lifecycle operation. These commands assume you have created an actor named incident-1042 in an atespace named incident-demo; they are not commands for the existing counter demo. Step 5: Verify the Result After resumption, retrieve the investigation status and verify that the agent still has its original findings. Hands-on Tutorial: Running and Resuming a Stateful Workload with Agent Substrate The objective of this tutorial is to execute a stateful workload, suspend it, and resume it without losing its execution state. This experiment uses a local Kubernetes environment with Podman and Kind. Setting Up Agent Substrate First, clone the Agent Substrate repository Shell git clone https://github.com/agent-substrate/substrate.git cd substrate With Podman configured as the container provider for Kind, create the local Kubernetes cluster and install the Agent Substrate runtime: Shell export KIND_EXPERIMENTAL_PROVIDER=podman ./hack/create-kind-cluster.sh ./hack/install-ate-kind.sh \ --deploy-ate-system \ --credential-provider='{"name":"k8s.io"}' Note: The upstream installation scripts assume Docker. Our Podman-based setup requires local compatibility adjustments, so this should be treated as an experimental installation path rather than an officially supported deployment configuration. Once deployed, verify that the runtime components are running: kubectl get pods -n ate-system Deploying the Stateful Counter Agent Substrate provides a counter demo that maintains state across three locations: Process memory: An in-memory counter.Filesystem: A counter stored in a file.Root filesystem: A counter stored in the actor's writable root filesystem. Deploy the demo and create an actor instance: Shell ./hack/install-ate-kind.sh --deploy-demo-counter kubectl ate create actor my-counter-1 \ -a ate-demo-counter \ --template counter Forward the router service to the local machine: Shell kubectl port-forward -n ate-system \ svc/atenet-router 8000:80 From another terminal, invoke the actor: Shell curl -X POST \ -H "ate-target-actor: ate-demo-counter/my-counter-1" \ http://localhost:8000/ The initial execution returned: Each subsequent request increments all three counters, demonstrating stateful execution within the actor. Check the status of the actor: Suspending the Actor After invoking the actor multiple times, suspend it using Agent Substrate's lifecycle controls. During suspension, Agent Substrate checkpoints the actor's execution state and releases its active execution slot. Unlike terminating a conventional container, suspension is designed to preserve the state required to continue execution later. Resuming the Actor Resume the suspended actor and invoke the actor again. Instead of restarting the counters from their initial values, the actor continues incrementing from its previously preserved state. For example, if all counters reached 5 before suspension, the next successful invocation produces: This illustrates the key capability of the runtime: execution can be suspended and restored while preserving both in-memory and filesystem state. Observing the Results The experiment demonstrates the following behavior: Stage Memory Counter File Counter Rootfs Counter Initial execution 1 1 1 Before suspension 5 5 5 During suspension Checkpointed Preserved Preserved After resumption and next request 6 6 6 Illustrative values for five requests before suspension; actual counts depend on the number of requests executed. The important observation is not the counter value itself, but its continuity across the suspension and resumption boundary. For the complete Podman, Kind, and Agent Substrate setup instructions, along with all tutorial execution commands, refer to the GitHub repository. What Happens Behind the Scenes? Agent Substrate coordinates several runtime mechanisms to support this behavior: Actor lifecycle management: Tracks running, suspended, and resumed actors.Checkpointing: Captures execution state, including process memory and filesystem changes, according to the configured snapshot scope.Snapshot storage: Retains checkpoints so the actor can be restored.Worker management: Releases execution resources during suspension and allocates resources when the actor resumes.Request routing: Directs subsequent requests to the appropriate actor execution environment. The experiment shows that there is a fundamental difference between normal process execution and stateful actor runtime: an actor can preserve its logical execution state without continuously occupying an active worker execution slot. This is particularly useful for AI agents that spend large portions of their lifetime waiting for external tools, model responses, or human decisions. However, preserving execution state does not guarantee exactly-once execution of external side effects or transactional consistency. Those concerns require additional application-level controls. Architectural Trade-Offs and Production Considerations Agent Substrate provides capabilities for running stateful AI agents efficiently, but having a runtime layer of its own has architectural trade-offs too. Production environments need to balance execution continuity with resource use, performance, security, and operational complexity. The following considerations are important when designing a production-ready agent runtime. Architectural consideration Trade-off / Production implication State persistence vs. storage overhead Frequent snapshots improve recoverability but increase storage consumption, checkpoint latency, and I/O overhead. Dedicated environments vs. shared worker pools Dedicated environments provide predictable resource availability, while shared pools improve utilization but introduce scheduling and capacity-management challenges. Resource efficiency vs. resume latency Suspending idle agents reduces compute costs, but restoring execution may introduce latency when processing new requests. Isolation vs. operational complexity Sandboxing strengthens workload isolation but introduces additional runtime configuration, monitoring, and debugging requirements. Infrastructure vs. workflow recovery Restoring process state enables execution continuity but does not guarantee successful recovery of application-level workflows or external operations. Security, policy, and observability Production deployments require authentication, authorization, audit logging, secrets management, and end-to-end tracing beyond basic execution management. Beyond Infrastructure-Level Recovery One important point to note is that preserving an agent’s execution state is not the same as guaranteeing correctness of its actions. For example, an agent calls a payment API and fails immediately after the payment succeeds but before recording the response. Restoring the agent from its last checkpoint could cause the payment request to execute again. A runtime can restore execution context, but it cannot tell whether an external operation was done successfully or not. Production systems therefore need additional mechanisms (idempotency keys, durable operation records, transactional outbox patterns and reconciliation workflows) to reduce duplicate side effects and ensure consistency across system boundaries. Similarly, runtime-level checkpointing does not guarantee transactional consistency between agent state, databases, and external services. Security and Operational Readiness Beyond execution management, production deployments require integration with broader platform capabilities, including identity-based access controls, policy enforcement, secrets management, observability, and resource governance. These capabilities help ensure that agents operate within defined security boundaries and that their execution can be monitored, audited, and controlled. The key architectural principle is that an agent runtime provides execution continuity, while application-level orchestration and governance remain responsible for ensuring operational correctness. Agent Substrate addresses important infrastructure challenges, but building production-ready agentic systems still requires coordinated design across the reasoning, orchestration, runtime, and platform layers. Conclusion As AI agents evolve from simple request-response systems to autonomous long-running workloads, the infrastructure supporting them has to evolve too. Reasoning capabilities aren’t enough; agents need reliable execution, persistent state, isolation, and recovery to work in production. Traditional infrastructure provides the building blocks, but integrating them into a coherent runtime remains a challenge. Agent Substrate demonstrates how an actor-based execution model, combined with checkpointing, suspension, resumption, and resource multiplexing, can help bridge this gap. However, a runtime is only one part of the broader agentic architecture. Workflow orchestration, security, policy enforcement, and application-level consistency remain equally important for building trustworthy systems. Ultimately, the next step in agentic AI is not just making agents more intelligent, but making their execution more reliable, resilient, and operationally sustainable. A well-designed runtime foundation is what enables intelligent agents to move from experimentation to production-scale systems.

By Josephine Eskaline Joyce, Ph.D CORE

We run an LLM pipeline that summarizes support tickets for a large network operations team: it reads long, noisy ticket logs and produces short summaries, customer status notifications, technician ticket summaries, and regional executive rollups. It works. But once it was running steadily, a quieter problem showed up. On an ordinary day, summarizing every ticket with a top-tier model and a full log was more expensive and slower than it needed to be. Four levers — incremental summarization, trimming, right-sizing, and parallel execution — are compounding to cut blended LLM cost by roughly 60%. Keeping the pipeline alive during traffic spikes is a separate story. This one is about the bill and latency on a normal Tuesday, and the four changes that, so far, are cutting our LLM cost by roughly 60% and making summaries faster while protecting quality. Where the Cost and the Milliseconds Actually Go Before optimizing anything, it helps to look at the anatomy of a single summarization call. It costs you two things — tokens and time — and both come from the same handful of places: Input tokens: The ticket log, plus the context we retrieve from our vector store, plus the prompt. For a summarizer, this is the giant. Logs are long; the summary is a few sentences.Output tokens: The summary itself is small, but output tokens are the most expensive per token, and generation time scales with how many you produce.The model: Price per token varies enormously between a flagship model and a smaller one.Round-trips: Doing calls one at a time serializes latency that doesn't need to be serial. That anatomy hands you four levers, roughly in order of leverage: stop re-reading what you already summarized (incremental summarization), trim what's left (dedupe and truncate), use a cheaper model where quality holds (right-size), and stop waiting on calls one at a time (concurrency). Lever 1: Incremental Summarization — Stop Rereading the Whole Log A ticket log is not static. A busy incident ticket accumulates thousands of log lines over its life, and every status update needs a fresh summary. The naive approach re-sends the entire log to the model on every regeneration, so the same thousands of lines are paid for again and again, and the token bill grows with the ticket. Instead, we cache each ticket's latest summary in Redis and treat it as running state. When the ticket updates, we send the model the cached prior summary plus only the new log lines since we last ran, and ask it to fold that delta into an updated summary. The model reads a short summary and a small delta rather than the full history — and because logs can run to thousands of lines, this was where the biggest token savings came from. The guardrails: we periodically force a full re-summarization so small errors don't compound across many incremental updates, and the cached summary is always refreshed against new log lines rather than served stale. This is the classic rolling summarization pattern — cheap to run, and it turns an ever-growing log into a bounded, near-constant cost per update. Incremental summarization: instead of re-sending the entire growing log on every update, we send the cached summary plus only the new lines - so cost stays flat as the ticket grows. Lever 2: Token and Context Trimming — Make Each Call Smaller Incremental summarization stops us re-sending the whole history; trimming shrinks what's left on the calls we do make. Because input tokens dominate a summarizer's cost, we squeezed the rest: Deduplicate the log: Machine-generated logs repeat the same alarm and heartbeat lines over and over. Collapsing those duplicates removed large redundant blocks before they ever hit the model.Truncate to what matters: Instead of shipping a ticket's entire history, we cap the window to the most recent and most relevant events.Cap the retrieved context: More retrieved chunks from the vector store is not automatically better - each one is pure input cost. We limited how many we pull.Cap max output tokens: Summaries are short by design, so a hard output ceiling prevents runaway generations and, because generation time tracks output length, directly trims latency. Together, these cut a large share of the remaining input on every call. Fewer tokens in and out is the rare optimization that reduces cost and latency at the same time. Lever 3: Model Right-Sizing — Pay for Quality Only Where It Matters Not every ticket deserves the flagship model. We select the model per ticket from business signals — severity, number of customers impacted, and revenue impact — so premium-model tokens go to the tickets that actually warrant them (major outages, high-value customers) while routine tickets get a smaller, cheaper, and faster model. The lowest-severity tickets skip the LLM altogether. A non-LLM extractive summary (key fields, error counts, first and last events) costs zero tokens. It's the same severity tiering we use to keep the pipeline resilient — here it doubles as the cheapest tier of all. The guardrail is a quality bar: before trusting a cheaper tier for a class of tickets, we validate its summaries against the flagship's, and when the signal is ambiguous, we route up, not down. Right-sizing is a double win: smaller models cost less and respond faster, so it reduces both spend and typical response time. Lever 4: Parallel Execution — Stop Waiting in Line Summaries are independent of one another, so processing them one at a time was leaving latency on the table. Running calls concurrently (async) collapses the wall-clock time to clear a burst of tickets and sharply improves perceived latency when several land at once. The one caveat is that unbounded concurrency will happily blow past your provider's tokens-per-minute limit. So concurrency runs under a shared token budget: as many calls in flight as the budget allows, and no more. You get the throughput without tripping the ceiling. Results No single trick did it; the levers compounded. Together they're bringing our blended LLM cost down by roughly 60% so far, while making summaries faster and steadier under load: leverwhat changed Incremental summarization Full log on every update → prior summary + new lines only Trimming Deduplicated and truncated input, capped output Right-sizing Flagship-for-everything → cheaper model by severity; lowest tier skips the LLM (extractive) Concurrency Serial calls → parallel, under the token budget Blended LLM cost ~60% lower (early results) The biggest surprise wasn't an exotic trick — it was simply not paying to re-read tokens we had already summarized. The other three levers compounded on top. What We Watch For Summary drift: Because each update builds on the previous summary rather than the raw log, a small error can propagate across updates. A periodic full re-summarization from the original log resets any accumulated drift.Quality regressions: Over-truncation can drop the one log line that mattered, and a too-cheap model can miss nuance. We keep a periodic eval/spot-check, so cost cuts don't quietly erode quality.Don't optimize noise: Measure first; a low-volume path rarely repays the complexity. Takeaways Measure where the tokens go – For a summarizer, input dominates - optimize that first.Biggest wins first – Reuse prior summaries instead of re-reading the whole log, trim what you do send, and right-size the model. Exotic tricks come later.Concurrency is a latency lever, not a cost lever – and it must respect your rate limits.Guard quality whenever you trim or downshift – pair every cost cut with an eval.

Last month, a nightly regression came back with 60 failures out of roughly a thousand tests. I spent the first half of the next morning fixing none of them. I sorted them: some were the same failure under different names, some were environment noise, and some were the product actually breaking. By lunch it was three things. A dependent service had returned errors for about a two-hour window, a data seeding job hadn't finished before the suite started, and one genuine defect. Sixty red rows, three problems. That sorting is most of my week. It's also the part nobody automates, because it isn't a fixed sequence of steps. It's judgment applied to evidence scattered around a CI dashboard, a log group, a ticket history, and whatever got merged Friday afternoon. So I built a triage step that does the mechanical sorting for me. Not an agent, not self-healing tests. A post-suite job that clusters failures, pulls surrounding evidence, and hands me a summary I can verify. This is what's in it, what it cost, and the two ways it went wrong before it went right. What Automation Is Good At, and What It Isn't Automation answers one question extremely well: did this pass or did it fail? It answers it faster than I can and the same way every time, which is the whole point. What it doesn't tell me: Why did this fail?Are these eight failures actually one failure?Which of them do I care about before the release call at 11?Have we seen this before? (We usually have.)What should I look at next? Every one of those is a reading problem, not an execution problem. The data already exists. It's just scattered around, and reassembling it by hand is slow and boring. The split I settled on: automation executes, a model reads, and I decide. The third part isn't a formality, and most of the engineering below exists to protect it. Step One: Cluster Without the Model My first mistake was handing raw failures to a model and asking what happened. Sixty stack traces is a lot of tokens; the answer took thirty seconds and was vague in a way that's hard to argue with. The fix was to do the deterministic part deterministically. Most failures in a big suite are duplicates. You can find that out with string normalization and a hash — no inference involved. Python # triage/cluster.py import hashlib import re from collections import defaultdict from xml.etree import ElementTree as ET # Order matters: strip the most specific patterns first. NOISE = [ (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-" r"[0-9a-f]{4}-[0-9a-f]{12}\b"), "<UUID>"), (re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}\S*"), "<TS>"), (re.compile(r"0x[0-9a-fA-F]+"), "<ADDR>"), (re.compile(r"https?://[^\s'\"]+"), "<URL>"), (re.compile(r"'[^']{0,80}'"), "<STR>"), (re.compile(r"\b\d+\b"), "<N>"), ] def signature(message: str) -> str: """Collapse a failure message to a stable shape.""" if not message.strip(): return "" first_line = message.strip().splitlines()[0][:300] for pattern, replacement in NOISE: first_line = pattern.sub(replacement, first_line) return first_line def cluster(junit_path: str) -> dict: groups = defaultdict(list) for case in ET.parse(junit_path).getroot().iter("testcase"): problem = case.find("failure") if problem is None: problem = case.find("error") if problem is None: continue shape = signature(problem.get("message", "")) key = hashlib.sha1(shape.encode()).hexdigest()[:10] groups[key].append({ "test": f"{case.get('classname')}.{case.get('name')}", "shape": shape, "time": float(case.get("time", 0)), }) return groups On that 60-failure run, this collapsed to six clusters. Two of them held 30 and 15 tests. That single number — 60 down to 6 — is most of the value, and no AI was involved in producing it. It also changes the economics of everything downstream. The model never sees 60 stack traces. It sees six normalized shapes plus a slice of evidence for each shape, which is roughly 4K tokens instead of 60K. Cheaper, faster, and a much smaller surface for the thing to make something up on. If you build one piece of this, build this piece. Step Two: Attach Evidence, With Hard Limits Each cluster then gets enriched from sources that are already there: the log group for the window the suite ran in, the deploys that landed since the last green run, and any tickets whose text matches the normalized shape. Python # triage/evidence.py import boto3 logs = boto3.client("logs") PLACEHOLDERS = ("<UUID>", "<TS>", "<ADDR>", "<URL>", "<STR>", "<N>") def log_hits(shape: str, start_ms: int, end_ms: int, group: str, limit: int = 5): """Pull a bounded sample of matching log lines for one cluster.""" # The normalized shape has placeholders in it; search on the # longest literal run instead. stripped = shape for token in PLACEHOLDERS: stripped = stripped.replace(token, "|") literal = max(stripped.split("|"), key=len).strip() if len(literal) < 12: return [] # too generic to search on; don't guess response = logs.filter_log_events( logGroupName=group, startTime=start_ms, endTime=end_ms, filterPattern=f'"{literal}"', limit=limit, ) return [ {"ts": e["timestamp"], "stream": e["logStreamName"], "message": e["message"][:400]} for e in response.get("events", []) ] Two constraints in there are deliberate, and both came from things going wrong. The limit=5 is because the second version of this pulled every matching line and blew past the context window on a noisy cluster. Five representative lines with timestamps answer "is this the same error, and when did it start" just as well as five hundred. The len(literal) < 12 guard is because a cluster whose shape normalizes down to something like Expected but was will match half the log group. Returning nothing is correct there. An empty evidence list is a fact, and the prompt below is written so the model has to say so rather than fill the gap. Step Three: Ask for a Reading, Not a Verdict The prompt is the part I rewrote the most. What finally worked was forcing every claim to point at a piece of evidence by ID, and constraining the service field to a list of things that actually exist. Python SYSTEM = """You are triaging failures from one CI run. Rules: - Every claim must cite evidence by its id. No id, no claim. - If the evidence does not support a cause, set cause to null. Do not guess. "Insufficient evidence" is a correct answer. - suspected_service must be one of KNOWN_SERVICES or null. - Do not restate the stack trace back to me. Return only JSON matching this schema: {"clusters":[{"cluster_id":str,"test_count":int, "cause":str|null,"suspected_service":str|null, "confidence":"low"|"medium"|"high", "evidence_ids":[str],"suggested_next_step":str}]} """ Before that enum existed, the model named a service we don't run. It sounded completely reasonable — right naming convention, plausible responsibility, adjacent to the real one. That's the failure mode that actually costs you: not obvious nonsense, but a confident answer that reads as if it came from someone who knows your architecture. An engineer who didn't know the service list would have gone looking for it. KNOWN_SERVICES gets injected from the same dependency manifest the deployment uses, so it can't drift from reality. Step Four: Check Its Work Before a Human Sees It This is the piece I'd argue is non-optional, and it's about forty lines. Python # triage/validate.py def validate(model_output: dict, clusters: dict, evidence: dict, window: tuple) -> dict: start_ms, end_ms = window kept, dropped = [], [] for item in model_output.get("clusters", []): cid = item.get("cluster_id") if cid not in clusters: dropped.append((cid, "unknown cluster id")) continue if item.get("test_count") != len(clusters[cid]): item["test_count"] = len(clusters[cid]) # we own this number cited = set(item.get("evidence_ids", [])) real = {e["id"] for e in evidence.get(cid, [])} if item.get("cause") and not cited: dropped.append((cid, "cause with no evidence")) continue if not cited <= real: dropped.append((cid, f"cited missing evidence: {cited - real}")) continue for e in evidence[cid]: if e["id"] in cited and not (start_ms <= e["ts"] <= end_ms): dropped.append((cid, "evidence outside run window")) break else: kept.append(item) return {"clusters": kept, "dropped": dropped} Counts come from the parser, never from the model — that's the one class of error I refuse to ship, because a wrong test count makes every other number in the summary suspect. Anything citing evidence that doesn't exist gets dropped rather than shown with a caveat, since a caveat still puts the idea in the reader's head. The dropped list is not thrown away. It's the closest thing I have to a quality signal on the triage step itself, and I look at it weekly. When one cluster type keeps landing there, the problem is usually my evidence collection, not the model. What It Actually Produces Before, for one cluster: Plain Text 30 tests failed. After: Plain Text cluster 4a1c9f2e — 30 tests cause: dependent service returned 503 on order lookups service: order-api confidence: high evidence: log lines at 02:14:06, 02:31:44, 03:02:10 (same message, same stream); no matching lines in the prior night's run next: check order-api health for the 02:10–03:20 window The second one I can disprove in about ninety seconds. That's the actual bar. Not "is it right" — "can I check it quickly." Every design decision above serves that. Numbers Triage time on a large regression run: roughly 90 minutes to about 20.Clusters per run: 5–8, from 40–70 raw failures.Model cost: a few cents per run. Not a factor.Wall clock for the triage step: under a minute, running after the suite.Claims dropped by the validator: around 1 in 8 in the first month, closer to 1 in 20 once the service enum and evidence limits were in. That last number is the one I'd watch if you build this. A rising drop rate means something upstream changed. Where I Stopped, On Purpose The obvious next step is letting it act. Reopen the ticket, rerun the cluster, patch the test. I did the first two. I won't do the third. The pitch for self-healing tests goes: the agent notices a failing test, decides the locator is stale, updates it, the test goes green, nobody gets paged. Run that on a test that was failing because the product broke. The agent didn't fix anything. It deleted the evidence, faster than a human would have noticed. The suite is green, the dashboard is calm, the defect ships. Green isn't the goal. Green with the check still intact is the goal, and nothing I've built or used reliably tells the difference between "this assertion is stale" and "this assertion just caught something." So I tier by blast radius: Tier Examples Gate Read-only clustering, summarizing, log search, explaining a trace none — worst case I ignore it Suggest root cause, test selection, defect priority, proposed edits engineer reviews before it moves Change config, closing defects, disabling tests, anything release-related explicit approval, always Most of the anxiety about AI in QA comes from treating it as one permission level. It isn't one. Testing the Thing That Helps You Test There's a part of this the industry hasn't sorted out, and I'm including it because pretending otherwise would be dishonest. The output isn't deterministic, so pass/fail assertions don't map onto it cleanly. What I do instead is keep a fixture set of about twenty past runs where I already know the real root cause, and score new prompt or model versions against them on two things: did it name the right cause, and did it cite evidence that actually supports it. It's regression testing with a rubric instead of an assert, and it's the only reason I noticed when a model version change made the summaries longer and vaguer without making them wrong. That's the other thing to plan for. Behavior shifts underneath you on someone else's release schedule. A prompt-based workflow that's been stable for months can quietly degrade after an upgrade nobody on your team asked for. Pin your model version, and treat a version bump as a change that needs a test run. And you are feeding logs — possibly source, possibly customer data — into a service. Whatever your org's answer is on that, get it in writing before you build this, not after. It's a security conversation more than a QA one, but it lands on our desk anyway. The Point The most valuable thing here isn't generating tests. It's reading everything we already collect and can't get through fast enough. Speed comes from the model. Repeatability comes from automation. Evidence comes from systems we already run. The call stays with me, mostly because I have to answer for the release.

Beyond Manual Mapping: AI-Orchestrated Governance for Hybrid Cloud It's Tuesday morning, and Slack is already on fire. A Salesforce admin renamed a custom field over the weekend, and your Spring Boot microservice has been throwing 400 errors since 6 a.m. You spend the next four hours tracing logs, patching a DTO, and redeploying. By Wednesday, you're in an "alignment meeting," manually cross-referencing field mappings between two systems that are supposed to talk to each other without your help. Every architect running a hybrid Salesforce and Spring Boot stack has lived this week. It's not a one-time incident; it's a recurring tax on the org. It grows with every schema change, every new field, every integration you bolt on. Most teams accept it as the cost of doing business. It doesn't have to be. Where the Wheels Come Off The friction comes from two architectures that were built with opposite priorities. Salesforce is optimized for speed. A business user creates a custom field in Setup, and it's live in minutes, no code review, no pipeline, no waiting. Spring Boot is optimized for rigor. Fields live in strongly typed DTOs. Changing one means a commit, a PR review, a test run, and a deployment through CI/CD. That's a feature, not a bug; it's what keeps your backend from falling over. When the two connect over REST, that mismatch turns into outages. Here's the kind of DTO that breaks the moment Phone__c becomes PhoneNumber__c on the Salesforce side: Java public class CustomerDTO { private String customerId; private String phone; // maps to Salesforce field Phone__c // getters/setters omitted } Nothing in this class tells you the field it depends on just got renamed. The Salesforce admin has no idea a Java class is bound to Phone__c. The developer has no idea the picklist they're parsing was just restructured with two new values. Information doesn't flow; it fragments. Teams patch around it, and the patches pile into a Frankenstein integration made of XSLT transforms, one-off converters, and tribal knowledge that lives in one person's head. That tax is measurable. Developers routinely lose 30 to 40 percent of their time to keeping brittle connections alive instead of shipping features. Every schema change becomes a small bet against production. A Different Approach: Zero-Touch Governance What if the system watched itself and fixed what it could, with a human still holding the approval button? That's the idea behind AI-orchestrated API governance: an orchestration layer that watches both Salesforce and Spring Boot, understands what changed and why it matters, and proposes (or, once trusted, implements) the fix. The important word is "understands." A naive diff tool sees Cust_Phone__c disappear and CustomerPhone__c appear, and flags it as a breaking deletion plus a breaking addition. An AI layer doing semantic analysis recognizes it as a rename, maps it to the existing customerPhoneNumber field in your DTO, and drafts the OpenAPI and Java changes before anyone gets paged. Building the Intelligence Layer Three components have to work together for this to be trustworthy instead of terrifying. 1. The Observation Engine This layer taps Salesforce Change Data Capture (or Platform Events) on one side and your Spring Boot Actuator endpoints on the other, then asks an LLM to classify what actually happened. A simplified CDC listener might look like this: Java @Component public class SalesforceChangeListener { @EmpConnectorSubscription(channel = "/data/Contact_ChangeEvent") public void onFieldChange(ChangeEvent event) { FieldChange change = FieldChange.builder() .objectName(event.getEntityName()) .oldFieldApiName(event.getChangedFieldOldName()) .newFieldApiName(event.getChangedFieldNewName()) .changeType(event.getChangeType()) .build(); semanticClassifier.classify(change); } } The classifier itself is a targeted prompt, not a general chatbot call: Java String prompt = """ A Salesforce field changed on object %s. Old API name: %s New API name: %s Change type: %s Classify this as one of: RENAME, TYPE_CHANGE, DEPRECATION, NEW_FIELD, BREAKING_REMOVAL. If RENAME, return the confidence score and the likely corresponding Spring Boot DTO field name. Respond only in JSON. """.formatted(objectName, oldName, newName, changeType); That last constraint matters more than it looks. Constraining the model's output to a fixed schema is what makes the downstream Policy Engine able to act on the result programmatically instead of parsing prose. 2. The Policy Engine Detection without guardrails is just a faster way to break things. The Policy Engine enforces the rules your architecture team already agreed on, things like "no breaking changes to public APIs without a major version bump" or "PII fields must stay encrypted in transit." Java public class GovernancePolicy { public PolicyResult evaluate(ProposedChange change) { if (change.isBreaking() && !change.hasMajorVersionBump()) { return PolicyResult.blocked( "Breaking change requires API major version bump"); } if (change.touchesPiiField() && !change.isEncrypted()) { return PolicyResult.blocked( "PII fields must remain encrypted end to end"); } if (change.isCustomerFacing() && change.deprecationWindowHours() < 48) { return PolicyResult.blocked( "Customer-facing changes need a 48 hour deprecation window"); } return PolicyResult.approved(); } } If a proposed change trips one of these rules, automation stops and an architect gets looped in. That's the whole point: the system earns trust by refusing to act outside its lane, not by acting fast. 3. The Execution Layer Once a change clears policy, this layer turns the proposal into an actual pull request. It can regenerate the OpenAPI contract: YAML CustomerDTO: type: object properties: customerId: type: string customerPhoneNumber: # renamed from "phone" type: string description: "Renamed from Salesforce Cust_Phone__c to CustomerPhone__c" and the matching Java class: Java public class CustomerDTO { private String customerId; private String customerPhoneNumber; // was: phone, mapped from CustomerPhone__c // getters/setters omitted } then open a PR, run the existing test suite in staging, and post the results back to the team. The API contract becomes the single source of truth both systems reconcile against, instead of a document that drifts out of date the week after it's written. A Second Example: Picklist Drift Renames aren't the only failure mode. Picklists shift constantly, and Spring Boot enums don't forgive silently. Say a Salesforce admin adds "Pending Review" to an Order_Status__c picklist that a Java enum is validating against: Java public enum OrderStatus { NEW, IN_PROGRESS, SHIPPED, CANCELLED // "Pending Review" doesn't exist here yet } Any order that lands in that new status now fails deserialization. The Observation Engine catches the new picklist value through the Metadata API, the Policy Engine checks whether enum changes count as breaking under your versioning rules, and the Execution Layer proposes the enum update alongside a migration note for any records currently in that state. That's a five-minute PR instead of a production incident discovered by a customer. What This Means for Your Team The obvious worry is that this replaces developers. It doesn't; it changes what they spend their day on. Right now, a good chunk of your time goes to translation work: mapping fields, writing converters, chasing down what changed and why. That work is necessary, but it isn't strategic. Once an orchestration layer is handling the translation, your role shifts from translator to curator. You define the data architecture, you write the policies the system enforces, and you spend your attention on the problems that actually need a human's judgment. Making the Transition Nobody should flip this on in production on day one. Trust gets earned in stages. Start with AI-Suggested Governance. The system proposes changes and opens PRs; a human reviews and merges. You're delegating the tedious analysis, not the decision. Once accuracy holds up across enough changes, typically approaching 99 percent, move to AI-Automated Testing. The system applies changes in staging, runs your regression suite, and reports results without a human triggering each step. Only after it's proven itself across hundreds of real changes should you consider Zero-Touch in Production, where the system can act autonomously within policy boundaries you've already defined and tested. Wrapping Up The hybrid enterprise, Salesforce for the customer relationship, Spring Boot for the backend logic, isn't going away. But manually bridging the two with mapping layers and tribal knowledge is a relic you can retire. AI-orchestrated API governance doesn't remove developers from the loop. It removes the busywork from their day and gives back the four hours you used to spend in a war room every time a field got renamed.

This is Part 3 of an 8-part series about PWAs on iOS. We've covered the history of the technology and the three ingredients every PWA needs: HTTPS, a manifest, and a service worker. Now it's time to look at the engine all of that runs on, and why, on the iPhone, Apple chose it for you long ago. A typical bug report from a tester goes something like this: "Works fine on Android. On iPhone we tried Safari and Chrome, and it breaks the same way in both." The tester dutifully opened two browsers and believes two engines are covered, when in fact they tested the same WebKit twice in different wrappers. Chrome on iPhone renders pages with the same system engine as Safari, and so do Firefox, Edge, Brave, and the rest. They differ in toolbars, bookmark sync, and looks, but in every one of them the page is built and run by WebKit, in whatever version shipped with iOS. It's easy to jump to the opposite conclusion: if there's only one engine, testing Safari must be enough. That's a mistake too, and just as common. When someone says "it works in Safari for us" on a call, it usually means they opened the site in a tab and stopped there. But one engine doesn't mean one environment. On an iPhone, it runs in at least four, and each has its own cookies, its own storage, and its own rules for how long a process lives. Checking a Safari tab and declaring victory is like looking into one room out of four and concluding that the whole house is tidy. We'll go through the rooms below. A Fork That Outlived Its Parent WebKit wasn't written from scratch in Cupertino. Its ancestor was KHTML, the engine of KDE's Konqueror browser, and next to it lived a JavaScript interpreter called KJS. Both projects were open source and distributed under the LGPL, a license that lets you take the code, modify it, and even build it into a closed-source product, but requires you to publish your changes to the library itself. KDE is an international free software community. It began in 1996, when Matthias Ettrich, a student at the University of Tübingen, announced on Usenet a project for a graphical desktop environment for Unix-like systems, and KDE 2.0 in October 2000 introduced Konqueror, which replaced the file manager and doubled as a browser. Apple started work on its fork in the summer of 2001. The reasons given for the choice were very practical: KHTML and KJS were small (under 140,000 lines of code), cleanly designed, standards-compliant, and easier to work with than the other options available. In January 2003, Steve Jobs unveiled Safari at Macworld, and the same day an Apple engineer, Don Melton, wrote to the KDE developers: Safari was built on their code, the modified versions had been made public, and a list of the changes would follow in a separate email. You often read that "Apple open-sourced WebKit in 2003," and that's not quite accurate. In 2003, two libraries were open: WebCore (HTML and CSS parsing, layout and rendering) and JavaScriptCore, both under the LGPL because they had grown out of KDE code released under that license. The WebKit framework itself, the glue code that Safari and other Mac programs used to talk to the engine, was only opened up on June 7, 2005, together with a public repository and bug tracker. Dave Hyatt of the Safari team announced it on his blog. The first iPhone in 2007 shipped with the same engine, which had to be squeezed onto a slow processor and a small amount of memory. In 2010, the project announced WebKit2, an architecture where the interface and the page content live in separate processes. The UI Process handles the window and the buttons, while the Web Content Process parses HTML, runs JavaScript, and can crash without taking the whole app down with it. On a phone this design matters more than on a laptop, and we'll come back to it. By then Google had been shipping Chrome on WebKit for a year and a half, but Chromium's multiprocess model had been built its own way from the start, and maintaining two architectures in one project was getting more and more expensive for both sides. On April 3, 2013, Google announced a fork called Blink and named exactly this reason in the announcement itself. Chrome on Android and Safari on iPhone have been cousins ever since, sharing a common ancestor. But after thirteen years of separate lives, the engines have drifted so far apart that how one behaves tells you almost nothing about how the other will. As for KHTML, where it all began, KDE eventually stopped developing it: work on KHTML ended in 2016, and in 2023 it was officially retired. One Line in the App Store Rules The WebKit requirement fits into a single line of the App Store rules. Guideline 2.5.6 says: "Apps that browse the web must use the appropriate WebKit framework and WebKit JavaScript." Today the guideline adds that you may apply for an entitlement to use an alternative browser engine, with links to the rules for the EU and Japan, but everywhere else the requirement is unchanged. For Google and Mozilla, this meant their iPhone browsers stayed shells for years: their own interface, account sign-in, tab and password sync, and inside it the same WKWebView that any app in the store can use. Chrome on iOS can't enable a feature WebKit hasn't shipped yet, and it can't work around a bug WebKit hasn't fixed yet. If something gets fixed in Chromium on a Tuesday, iPhones get the fix only when Apple ships an iOS update with a newer WebKit, and a fresh version of Chrome from the App Store changes nothing. Apple justifies the rule on security grounds, and I don't think that's a mere excuse. In its developer documentation, Apple calls browser engines one of the most common attack vectors, and that's hard to argue with: an engine constantly processes untrusted code from all over the internet. When there is one engine, and it's part of the system, a single iOS update closes the hole for every browser at once. On top of that, fast JavaScript needs JIT compilation, meaning the ability to create executable code in memory at runtime, and ordinary iOS apps aren't given that permission. I find the decision reasonable for the platform and costly for the web. The security is paid for by making the web on iPhone depend on a single supplier, and that supplier's release schedule automatically becomes everyone else's. Attempts to get out of this arrangement began before Apple allowed anything. In December 2022, press reports appeared saying that Apple was preparing for the EU's Digital Markets Act and was discussing dropping the mandatory WebKit rule. About six weeks later, on January 31, 2023, the Chromium team opened an issue in its public tracker for an experimental port of Blink to iOS. The issue stated up front that they were building a stripped-down test shell, content_shell, rather than a browser for the store, because such a browser would violate the App Store rules. Around the same time there were reports that Mozilla was also trying to get its Gecko engine running on iPhone. I'll say how that turned out below, when we get to the regulators. Four Rooms in One House Even with a single engine, the environment it runs in can differ, and for a web app that difference matters more than the browser's brand. I keep four such environments in mind, and data doesn't flow from one to another on its own. Room 1 The most familiar one is an ordinary Safari tab. Cookies, localStorage, IndexedDB and camera and notification permissions are shared across the tabs of a profile, Web Inspector on a Mac can attach to the tab, and almost every tutorial on the internet was tested right here. Room 2 SFSafariViewController looks almost the same. It's a Safari window with Reader and password autofill that a native app opens on top of itself when you tap a link in a feed or an email. The app itself can't see what's happening inside. Before iOS 11, that window shared cookies with Safari, and a lot of single sign-on setups relied on it. In iOS 11 Apple separated the storage, and each app got its own cookie jar for such windows. Today Apple's documentation explicitly sends anyone who needs data shared with Safari to a different API, ASWebAuthenticationSession. The old behavior still circulates in forums and articles. If someone confidently says that the user is already logged in inside SFSafariViewController, ask which iOS version they tested that on. Room 3 WKWebView is a different story. This embeddable web view is what hybrid apps, the in-app browsers of messengers, and Chrome and Firefox themselves are built on. Its storage belongs to the host app: Safari's cookies don't carry over, but the app itself can read and set cookies through WKHTTPCookieStore, so your session won't be a secret from the host. Even the quota is different. Since iOS 17, WebKit lets a single site in a browser app, meaning one that can be set as the default browser, use up to 60% of the disk, but inside the web view of an ordinary app only up to 15%. Room 4 An icon on the Home Screen, opened as a web app, lives apart from all the others. It has its own container with its own cookies, localStorage, IndexedDB, Cache Storage and its own permissions, and that container can't see Safari or any web view embedded in other apps. Someone who's logged in inside a tab will install the app, tap its icon and find a login form, and camera access granted in the tab has to be granted again. Since iOS 26, a site added to the Home Screen opens as a web app by default. You can turn off the "Open as Web App" switch when adding it, and then the icon becomes an ordinary bookmark that takes you back to Room 1. When a bug report comes from a user, the first thing we need to work out is which room they were in. For that, I usually hide a diagnostics screen in the app, opened by long-pressing the version number, that shows an object like this: JavaScript async function whereAmI() { const standalone = matchMedia("(display-mode: standalone)").matches || navigator.standalone === true; return { standalone, path: location.pathname, sw: navigator.serviceWorker?.controller?.scriptURL ?? "none", caches: "caches" in self ? await caches.keys() : [], build: document.documentElement.dataset.build, ua: navigator.userAgent, }; } navigator.standalone is a nonstandard Apple property, and inside an embedded WKWebView both checks come back false, so they can't tell a Safari tab from an in-app browser inside a messenger. The user agent sometimes helps, since it may contain the name of the host app, but that's a hint for a human reading the report, and I wouldn't build logic on it. I take the address from pathname because the query string sometimes holds one-time tokens that have no business being in logs. From here on, when I say a web app opened from the Home Screen, I mean an actual PWA, not a plain bookmark to a site page that someone has put on the Home Screen. Why Users Log In Three Times The most common story I've run into across different teams goes like this. A corporate single sign-on login works fine in a Safari tab, because the identity provider's cookie is already in the profile. Then a native wrapper appears, and sign-in there is opened through SFSafariViewController, expecting the same cookie to be there. It isn't, so the person types their password a second time. Then the site is put on the Home Screen, and it all happens a third time, because the icon has its own container. Support gets a complaint: "the app keeps logging me out," even though the app never did anything of the kind. Each sign-in the user went through simply happened in a new, separate container. You can't read Safari's cookies from another environment. There's no public API for it, and that's by design. The workable options are more boring. A native app that needs a session from Safari should use ASWebAuthenticationSession, available since iOS 12. It shares persistent cookies with Safari and asks the user for permission before opening the sign-in page, and if you specifically don't want a shared session, the prefersEphemeralWebBrowserSession flag makes the sign-in private. For a PWA, I assume that sign-in starts from scratch after installation, and I try to keep it short: OAuth with PKCE, a redirect back to an address inside the manifest's scope, and a long-lived session on your own domain, so that the user signs in once after installing rather than every week. I also always check on a real device where the provider's page opens and where the redirect brings the user back, because navigating outside scope in an installed app behaves differently than it does in a browser tab. Twenty Billion Dollars for a Search Bar If the WebKit-only requirement rested on security alone, it would be easier to argue about. But Safari has a commercial side too: the right to be its default search engine is worth a great deal. In the US Department of Justice's antitrust case against Google, court documents disclosed in spring 2024 put the amount Google paid Apple in 2022 to be Safari's default search engine at about $20 billion. That doesn't prove WebKit was made mandatory on every iPhone for the sake of money, but it shows what's at stake: Safari is a significant source of revenue for Apple. So anything that could take users away from it on iPhone, whether a browser choice screen or other engines, touches not only architecture but revenue. Critics of the rule that every browser must use WebKit, above all Open Web Advocacy, have argued for years that without competition between engines Apple has little incentive to move quickly on the features that make web apps resemble native ones. As an example, they point to Web Push, which web apps on iPhone didn't get until spring 2023. Apple, for its part, points to Safari's speed, to privacy, and to its view that some APIs from Chromium are dangerous for users. I'm not firmly on either side of this argument. Some of the features missing on the iPhone are ones I wouldn't hand to a random website on my own phone either. Others were delayed for years without a clear explanation, and from the outside it's very hard for me to tell where concern for the user ends and concern for the App Store begins. The Month PWAs Disappeared in Europe The first to break the WebKit-only rule was the European Union. The Digital Markets Act (DMA) required Apple to allow other browser engines on iPhone, and iOS 17.4, released in March 2024, introduced BrowserEngineKit, a set of system interfaces for browsers that run on their own engine. It applies in the 27 EU member states, but the European Economic Area as a whole isn't covered, so a user in Norway, for example, is left with plain WebKit. Apple spelled out the conditions in great detail, and they read more like a list of reasons not to bother. A browser with its own engine must be distributed only in the EU, must be eligible to be set as the default browser, and must pass 90% of the Web Platform Tests and 80% of Test262 even with JIT turned off (for example, when the user has Lockdown Mode enabled). It also has to publish a vulnerability disclosure policy and quickly patch actively exploited vulnerabilities, with Apple naming a 30-day window for straightforward cases. Mozilla reacted right after the announcement, in January 2024. A company spokesperson, Damiano DeMonte, told The Verge that Mozilla was "extremely disappointed" and explained why: "The effect of this would be to force an independent browser like Firefox to build and maintain two separate browser implementations — a burden Apple itself will not have to bear." Formally, Apple had complied with European law. In practice, it had made it prohibitively expensive for any company that develops its own browser engine to bring it to Apple's devices in the EU. For PWAs, the most interesting part happened in the beta. In February 2024, developers in the EU noticed that in the iOS 17.4 beta, PWAs no longer opened in their own window and had turned into bookmarks that launch an ordinary browser tab. Gone with the window were the separate data container and Web Push, which had arrived only a year earlier. Apple blamed the same DMA: since browsers on other engines were now allowed, they would have to be let into the web app mechanism too, and that would require a new integration architecture. Building it, Apple said, was impractical: on top of the law's other requirements, very few people use Home Screen web apps, and the company judged the cost unjustified. Open Web Advocacy, developers and the press kicked up a fuss, the European Commission got involved, and on March 1, 2024, Apple reversed the decision. The final iOS 17.4 brought web apps back in the EU, still running on WebKit. That autumn Apple took the next step: starting with iOS 18.2, a browser with its own engine in the EU can create Home Screen web apps that run on that engine. On paper, that's exactly the freedom people were fighting for. But only a browser with its own engine that has met all of Apple's conditions can use it. London and Tokyo Followed Brussels The UK's Competition and Markets Authority (CMA) wrapped up its study of mobile ecosystems in June 2022 by proposing an investigation into how the mobile browser market works. On November 22 of that year, it launched an in-depth market investigation into browsers and cloud gaming. Apple challenged that decision in court, and after a ruling by the Competition Appeal Tribunal, the investigation was suspended from March 31, 2023. The CMA appealed the ruling; the Court of Appeal upheld the CMA's decision on November 30, 2023, and on January 24, 2024, the investigation resumed. The final report came out on March 12, 2025, and its conclusion is blunt: competition in the UK mobile browser market isn't working well, users and businesses both suffer as a result, and the main concerns relate to Apple's rules for browsers on its devices. The CMA decided to change those rules not through the investigation but under the UK's new digital markets law, the Digital Markets, Competition and Consumers Act, which came into force on January 1, 2025. On October 22, 2025, it designated Apple and Google as having Strategic Market Status in mobile platforms, and it explicitly included browsers and browser engines in its definition of a mobile platform. The regulator now has a tool to demand changes. But as of autumn 2026, there is still no requirement to allow other engines on iPhone: Apple's first commitments to the CMA concern the rules for reviewing and ranking apps in the App Store. Japan went further. The Mobile Software Competition Act (MSCA) was passed in June 2024 and came into full force in December 2025. In the guidelines to it, which got a lot of coverage in summer 2025, the Japanese regulator specifically barred Apple from placing unjustified technical restrictions on alternative engines, from imposing high financial costs on developers, and from steering users away from browsers built on such engines. In effect, Japan ruled out the European scenario in advance, where permission technically exists but using it doesn't pay. On December 17, 2025, Apple announced its Japan changes in iOS 26.2. Users in Japan got a choice screen for browsers and search engines, and browser developers could move to engines other than WebKit, with access to JIT and multiprocessing and, as Apple puts it, under strict security and privacy requirements. As for the prototypes mentioned earlier, they remained prototypes, and ordinary Chrome on iPhone, in Europe and Japan alike, still runs on WebKit. In June 2026, Kyle Pflug of the Microsoft Edge team published benchmark results for a research prototype of a Blink-based browser for iOS, built with BrowserEngineKit. In Speedometer 3.1, the prototype beat Safari on iOS 26.5.1 by about 29%, and Pflug himself stressed that he was showing a research build, not announcing a product. As of autumn 2026, there are almost no full-fledged iPhone browsers running on their own engine. I wouldn't build product plans around their arrival any time soon, but "never" isn't the right word either. So for now, what the web can do on iPhone is determined by WebKit, and you have to plan a product around what WebKit lacks for PWAs on iOS. Some of the gaps are visible right away, while others surface at the most inconvenient moment. The one that stands out first is the lack of an install button: a site can't offer to install itself as an app. There Will Be No "Install" Button Chromium has a beforeinstallprompt event: a site uses it to show its own button and trigger the system install dialog. iOS has no such event, and the only way to put a site on the Home Screen is by hand, through Share and then Add to Home Screen. Because of this, onboarding on iPhone always ends up a little clunky. In a Safari tab, you can show the instructions in words and a picture; in an already installed app, you have to hide them; and in a messenger's in-app browser, they're often useless because the elements they refer to, such as the Share button, simply aren't there. There are no background mechanisms either. Background Sync, Periodic Background Sync and Background Fetch aren't implemented on iOS, so you can't send out a queue of pending requests after the user has closed the app. On the hardware side, Web Bluetooth, Web NFC, and WebUSB are unavailable, and navigator.vibrate is missing even for a short bit of haptic feedback. Take an app for a warehouse worker who scans barcodes with the phone's camera: on Android, a failed scan can be signaled with a vibration, but on iPhone there's no such signal, so the error has to be shown with a screen color or a sound. If the product needs a Bluetooth printer, or has to work with the screen locked, it's more honest to discuss a native app right away than to promise that the web will catch up soon. With no background sync, the whole model for saving data changes. I put operations into a queue in IndexedDB and send them at moments when the app is definitely running: when it comes back to the foreground and when the network reappears. Each operation gets an idempotency key, so that resending after a dropped connection doesn't create a duplicate: JavaScript let flushing = false; async function flushOutbox() { if (flushing) return; flushing = true; try { for (const job of await outbox.pending()) { await api.send(job.payload, { idempotencyKey: job.id }); await outbox.markSent(job.id); } } catch { // no network or the server returned an error: we'll try again on the next event } finally { flushing = false; } } The online event tells you the device has connected to a network, but it doesn't guarantee that a request will reach the server. For example, the phone picks up Wi-Fi on the subway, but there's no internet behind it yet, and the very first request fails. That's why the loop stops at the first error and waits for the next trigger: another online event, or the app returning to the foreground. The second weak spot is in the interface: if you don't show the user that some operations haven't been sent yet, they'll close the app believing everything went through. How a service worker fits into this scheme, and why it doesn't save you when the app is closed, is covered in Part 5 of this series. The Safari Version Is the iOS Version Everything above applies to WebKit in general, but on a specific phone the set of available features also depends on the iOS version. On a Mac, Safari updates separately from the system. On an iPhone, it doesn't work that way: the engine is built into iOS and updates only together with it. Safari 16.4 arrived with iOS 16.4, Safari 26 with iOS 26, and a user can't install a newer browser on an older system. Look at the device report in your web analytics (Matomo, Google Analytics, or your server logs): if a noticeable share of visitors come from iPhones running iOS 16.2, those people have no Web Push. Telling them to "update Chrome" won't help, because Chrome on such a phone runs on the same WebKit from 16.2. Only updating iOS can help. That's why I tie the test plan to iOS versions: on an iPhone, the browser brand says almost nothing about the engine version. I keep two reference points: the lowest version the product still promises to support, and the current one. Detecting the engine version from the user agent (UA) isn't a good idea either. The user agent is a string that a browser sends with every request to introduce itself to the site: its name, its engine, and the operating system. On iPhone it's almost useless: every browser has the word Safari in it, and the AppleWebKit number hasn't changed in years. If the condition that enables push depends on parsing such a string, I ask for it to be removed in code review. In code, check for capabilities, not versions: is there a PushManager (the object a site uses to subscribe to push) and is the app running in standalone mode, meaning launched from an icon on the Home Screen. If there's no PushManager and the app is open in a tab, show the install instructions. If the app is already installed and there's still no PushManager, the iOS version is too old, and you should offer the user a system update. If PushManager is there, show the notification bell. Safari's release cycle has a rhythm worth knowing. The major autumn releases bring what was shown at WWDC (Apple's developer conference in June), but for web apps on iPhone, a lot of the important changes arrive in spring, most often in the x.4 releases. Safari 15.4 in March 2022 brought more than 70 additions, among them <dialog>, :has() and support for icons from the manifest (though apple-touch-icon still takes priority in Safari). Safari 16.4 in late March 2023 added 135 features, including the Badging API and, of course, Web Push for Home Screen web apps. Safari 18.4 in spring 2025 brought Declarative Web Push, and 26.4 in March 2026 added WebTransport. The WebKit team acknowledges the pattern itself in its post about 26.4: "March has a way of bringing a lot of new things to WebKit — and this year is no exception." There are exceptions, too: 17.4 in March 2024 was modest in features, but it contained all the changes that Europe's DMA required. You can find out what the next release will bring in advance, and without any insider contacts. Can I Use and MDN show what Safari already supports, and the Standards Positions page shows where WebKit stands on new specifications. Safari Technology Preview on a Mac lets you try features months before release, though for iPhone it's only a rough guide: it doesn't fully reproduce iOS behavior. The Safari release notes and posts on the WebKit blog list the changes in each release point by point, and the blog also carries notes for every Technology Preview build. On the iPhone itself, in Safari's settings under Advanced, there are switches for experimental features: in iOS 16.4 the item was called Experimental Features, and in newer versions it was renamed Feature Flags. They're handy for trying out an API ahead of time, but before you investigate a bug a user has reported, reset them to their defaults. Otherwise you'll be reproducing the problem in a browser with settings that the user doesn't have. A Link from a Messenger Opens in the Wrong Place Many people reach a site without going through Safari at all: through a link in a messenger, in Instagram, in an email client or in a corporate app. Such apps open the site either in SFSafariViewController or in their own browser built on WKWebView, and in the second case you land in someone else's container, where the host decides what's allowed. There's usually no Add to Home Screen item there, the Share button may not exist at all, and a hint saying "tap Share" leads the person nowhere. For these cases, it's better to show a short text asking the user to open the link in Safari, along with a button that copies the address. Many messengers already offer their own "open in browser" control, but it's safer not to rely on it and to build your own. Debugging all this is a special kind of fun. Web Inspector on a Mac can attach to Safari tabs and to installed web apps, but it will only see someone else's WKWebView if the app's developer has explicitly allowed inspection: since iOS 16.4 there's an isInspectable property for that, and it's off by default. You most likely won't be able to attach your inspector to the browser inside a popular messenger. If a bug reproduces only "from the messenger," I first ask the person to open the same link in Safari, and a good share of such bugs disappear there, because they lived in the host's embedded window, which your service worker might never have reached. JavaScriptCore Does Things Its Own Way Suppose the page opened in the right place: in a Safari tab or in an installed app. Next, the code starts running, and here too things differ from what you know from Chrome. Because Chrome on iPhone runs on WebKit, it executes JavaScript not with the familiar V8 but with WebKit's own engine, JavaScriptCore. V8, the JavaScript engine in Chrome, has little in common with it beyond the language specification. JavaScriptCore has four execution tiers. First, the code is run by an interpreter called LLInt (low-level interpreter). When a function gets hot, meaning it's called or loops often enough, the engine compiles it on the fly, which is known as JIT (just-in-time compilation): first with the simple Baseline JIT tier, then with the optimizing DFG (data flow graph) tier, and finally with the most aggressive one, FTL (faster than light). The engine decides when to move up a tier based on an execution counter and on statistics about value types that it collects as it goes. In an ordinary interface, the difference between JavaScriptCore and V8 is barely noticeable. On heavy computation, the picture varies: one engine is faster on some tasks, the other on others. So it makes sense to profile for iPhone on an iPhone itself: measurements from Chrome DevTools tell you about V8's performance, not JavaScriptCore's. Lockdown Mode changes this picture more than any optimization does. It's an optional mode of extreme protection, introduced in iOS 16: the person turns it on themselves in Settings, under Privacy & Security, and the phone restarts. Apple built it for the very few people who may be personally targeted because of who they are or what they do. Apple says that in this mode some complex web technologies are blocked, so websites may load more slowly, web fonts may not display, and images may be replaced with a missing-image icon. JIT is unavailable in it as well: Apple says so directly in its conditions for browsers on third-party engines, discussed earlier. That means all JavaScript runs on the slowest tier only, the interpreter, and a page that flies for everyone else can be noticeably sluggish for such a user. So if a bug reproduces for only one user while everyone else on the same iOS version is fine, ask them whether Lockdown Mode is on (Safari shows a banner when it is). A site can't detect this itself, and Apple doesn't describe any way to. Ask before you spend a week hunting for a race condition in React: with slow JavaScript, the order of events changes, and an ordinary page starts to behave erratically. You don't need to work around the mode. What you need is an interface that stays usable with slow JavaScript, without web fonts, and with images that may fail to load. JavaScript syntax has its own release schedule, too. Lookbehind in regular expressions (a check of what comes before a match, a construct like (?<=...)) arrived in Safari only in 16.4, in spring 2023, together with Import Maps. Before that, a single such regex in a bundle caused a SyntaxError at parse time, and the user saw a white screen instead of one broken function. As far as I know, transpilers don't rewrite lookbehind for you. You either rewrite the expression by hand or raise your minimum supported iOS version to 16.4. structuredClone, on the other hand, has been available since 15.4, and a polyfill for it (code that imitates a missing function) only adds weight on modern iPhones. The Process Will Die, but the Data Must Survive So far we've looked at how JavaScript executes on iPhone. Now let's look at how long it lives: a page can vanish at any moment, without warning, and for your data that matters more than any difference between engines. It's time to return to WebKit2, which came up in the history of the engine. There, the interface and the page content live in separate processes, and your page is run by the Web Content Process. When the system runs low on memory, jetsam steps in, the iOS mechanism that frees memory by terminating processes, and it simply kills that process. This is especially likely when the user switches away from the app: they're filling in a form, get distracted by a phone call or switch to the camera to photograph a document, and while they're busy, the system removes the page left in the background from memory. The page gets no "about to close" event: neither beforeunload nor pagehide arrives, and everything it held in memory (variables, objects, the not-yet-submitted contents of a form) disappears at once. When the person comes back, they find the app starting from scratch, or an empty form on the screen. For any screen beyond a single short form, my takeaway is a simple rule: write the draft to IndexedDB as the user types, with no "Save draft" button, which people won't press anyway, because they don't know their app can be killed out from under them. JavaScript // The form the user is filling in const form = document.querySelector("#invoice"); // ID of the pending save. While the person is typing, we keep resetting it let timer; // The input event fires on any change to a form field, // that is, on every keystroke form.addEventListener("input", () => { // The person is still typing: cancel the save scheduled by the previous keystroke clearTimeout(timer); // Schedule a new save in 400 ms. If the field changes again before then, // the timer is reset, so the write happens once, after a pause in typing, // and not on every letter timer = setTimeout(() => { // Collect all the form's field values into a plain object like { fieldName: value } const data = Object.fromEntries(new FormData(form)); // Write to IndexedDB under the key "invoice", overwriting the previous draft. // drafts here is a small wrapper over IndexedDB (its code isn't shown in this example). // savedAt lets us show "there's an unsaved draft from 2:02 PM" when restoring drafts.put("invoice", { data, savedAt: Date.now() }); }, 400); }); This pattern is called debouncing: the write is postponed until there's a pause in typing, so the database isn't hit on every letter. The 400 ms delay means the last few characters typed before the process dies can be lost, and for an invoice I'm fine with that. Restoring matters more. You must never silently fill in a half-completed form. The person should see a line like "There's an unsaved draft from 2:02 PM" and choose for themselves whether to continue it or throw it away. Otherwise, sooner or later an old draft will land on top of a new invoice, and sync will take the blame. Two stubborn myths surround storage for PWAs on iOS: the "hard 50 MB limit" and "everything gets wiped after seven days." The first is long outdated: as I mentioned earlier, since iOS 17 WebKit calculates quotas as a percentage of disk size, an installed web app gets the same quota as a site in the browser, and Apple doesn't publish a fixed figure in megabytes. The second grew out of a real change, but one that affected only Safari. In March 2020, WebKit introduced a seven-day limit in ITP (Intelligent Tracking Prevention, Safari's protection against tracking): if someone uses Safari for seven days and never interacts with a site, everything the site wrote with scripts, from localStorage to IndexedDB and service worker registrations, gets deleted. But that rule was written for sites opened in Safari. An installed web app runs outside Safari, and in that same announcement WebKit wrote that it doesn't expect data in such an app to be deleted: that isn't what ITP is meant to do, and it would treat a case of deletion as a serious bug. So the "seven days" story shouldn't be applied to installed PWAs. None of this guarantees that data lives forever, because it can disappear for other reasons. When the device is low on storage, the system can evict a PWA's data regardless of ITP, and if the user deletes the app's icon from the Home Screen, the data goes away with the app. navigator.storage.estimate() gives only an approximate figure. WebKit grants navigator.storage.persist() requests based on heuristics, and one of them, according to WebKit's own description, is that the site is launched as a Home Screen web app. Anything you can't afford to lose should live on the server, and the local database should serve as a cache and a queue. Everything above applies to an app that's already on the Home Screen. But a PWA doesn't get its own storage in advance: it appears at the moment of installation, and every installation gets its own. So in Part 4 of this series, we'll tap Share, choose Add to Home Screen, and see what iOS creates at that moment: where the icon comes from, exactly when that fourth room we talked about above appears, and why the same site can be installed as a PWA twice, leaving you with two apps that know nothing about each other.

Every Abstraction Leaks. Generative AI Is No Exception. Joel Spolsky gave us the Law of Leaky Abstractions years ago, and it lands hard on the newest layer in software development: generative AI. When you prompt a model to produce code, it builds an abstraction over the implementation. You get working functionality without touching the underlying logic. Great, until the day it breaks. So what do you do when the abstraction leaks? The conventional answer is simple: go back and read the code. Tudor Girba, creator of Glamorous Toolkit and the voice behind moldable development, argues that answer is a trap. When AI-generated code fails or needs to be understood, the right move is not heroic manual reading. It is reaching for deterministic, contextual micro-tools that make the system explainable on demand. Here is the uncomfortable part. AI code can follow unfamiliar patterns, skip the mental model a human author would have embedded, and ignore your team's conventions. That often makes it harder to read than human-written code, not easier. And the evidence that the abstraction is already leaking is piling up. The Numbers Are Not Subtle GitClear analyzed millions of changed lines of code from 2020 through 2023. Their finding: code churn, the share of lines reverted or rewritten within two weeks of being authored, is projected to double in 2024 against its 2021 pre-AI baseline. The mix of changes shifted too. "Added" and "copy and pasted" code grew relative to "updated," "deleted," and "moved" code. In plain terms, AI-assisted code tends to break the DRY principle and reads like the output of a short-term contractor rather than a senior engineer. Then there is plausible incorrectness. A peer-reviewed study by Kabir and colleagues looked at 517 ChatGPT answers to Stack Overflow questions. Fifty-two percent contained incorrect information, and users missed that misinformation 39 percent of the time. Why? Because 77 percent of the answers were verbose, and the polished prose masked the errors underneath. People still preferred the ChatGPT answers 35 percent of the time, high error rate and all. Capability is climbing, to be fair. On SWE-bench, Princeton's benchmark of 2,294 real GitHub issues, the best model at launch (Claude 2) solved just 1.96 percent. By mid-2025, purpose-built agent scaffolding reached about 65 percent on a curated 500-instance subset. That is roughly a 33x jump. It is also still one-third of real-world issues left unsolved, because real engineering means coordinating changes across many functions, classes, and files at once. The Verification Bottleneck Is the Leak in Its Purest Form Here is the asymmetry that should keep engineering leaders up at night. Generating 500 lines of code takes seconds. Verifying those lines for security holes or logic errors can take an hour. The volume of AI output is becoming too large for humans to review. That creates pressure to hand review over to the AI itself, which introduces a recursive trust problem: you are now trusting one non-deterministic system to check another. The very speed that makes AI attractive is exactly what makes its failures dangerous. A Taxonomy of How AI Code Fails Failure Mode What Happens Evidence Code churn amplification Lines reverted within two weeks at double the pre-AI rate GitClear, 153M lines DRY violation More copy-paste at the expense of refactoring GitClear Plausible incorrectness 52 percent error rate masked by confident language Kabir et al. Verification asymmetry Generation in seconds, verification in hours GitClear community Compositional failure Models stumble on multi-file coordination SWE-bench Review overload AI output exceeds human review capacity GitClear The Quiet Risk: Skill Atrophy There is a second failure happening in slow motion, and it is about people, not code. Automation bias is the human tendency to favor automated suggestions and overlook contradicting information, even when we are right. In coding, it shows up in two ways. Errors of commission: you accept an AI suggestion without checking it. Errors of omission: you miss a bug because you trusted the output. When the AI eventually fails, and you have lost the skill to diagnose it, you either blindly trust the next suggestion or abandon the tool entirely. Neither is good. GitHub says developers write code 55 percent faster with Copilot. GitClear's data suggests that speed arrives with a maintainability tax, a productivity-versus-quality paradox. The Nielsen Norman Group found generative AI lifts employee performance by an average of 66 percent, with the biggest gains going to the least-skilled workers. Helpful for output, worrying for growth: junior developers may lean on AI before building foundational skills, while seniors use it as a genuine multiplier. This is not a new story. Lisanne Bainbridge named it in 1983 with the "ironies of automation." Automation removes the routine practice that keeps skills sharp, yet the human is needed most precisely when the automation fails. Air France Flight 447 is the hard lesson: when the autopilot disconnected over iced sensors, pilots with degraded manual flying skills stalled the aircraft. Research on GPS navigation shows the same erosion of spatial memory. Passively accepting AI code risks the same slow loss of algorithmic reasoning. Why "Read the Code" Cannot Be the Fallback Anymore Girba's challenge rests on four points, and each one holds up. Scale defeats comprehension. Modern systems are too large and interconnected to understand by reading. AI generates code faster than any team can review it. The reading fallback simply cannot keep pace. AI code lacks authorial context. An LLM may use unexpected idioms and carry none of the design intent a human would embed. Two developers reading the same generated function can reach different conclusions about what it is for. Reading is ad hoc and unrepeatable. It does not scale across a team or across time. Traditional debugging assumes determinism. Breakpoints, step-through, and stack traces all assume the same input yields the same output. LLM output is non-deterministic. The same prompt can produce structurally different code each time, so reproducing a bug becomes unreliable. Multi-step agent workflows branch in ways a conventional stack trace never captures. Semantic equivalence hides errors. Two outputs can be semantically identical but syntactically different, which defeats diff-based and line-by-line debugging. The real question has shifted from "what does this line do?" to "what did this system actually do at runtime?" Reading alone cannot answer that. So Girba reframes the core skill. It moves from "the ability to read and write code" to "the ability to make systems explainable." Moldable Development and the Deterministic Toolkit Moldable development starts from one observation: developers spend most of their time understanding systems, not writing code. Every system is unique, so the tools used to understand it should be moldable, meaning customizable to the system in front of you. Instead of leaning only on generic IDE features, you build many small custom tools: views, inspectors, and queries tuned to your domain. The idea is that any question about a system can be answered through narratives made of contextual tools. Think data science, but pointed at your software. Glamorous Toolkit is the concrete implementation, an open-source environment for building custom inspectors and views for every object in your system. A query result renders as a table, a graph structure renders visually, a business rule shows its decision tree. All deterministic, all contextual. It now ships a built-in moldable agent harness with programmable agents and explainable agent actions, so you can understand what the AI did deterministically rather than taking its word for it. The Broader Ecosystem Is Already Here Time-travel debuggers. Tools like rr record an execution once and replay it deterministically, with identical address spaces, registers, and syscall data every run. Pernosco, built on rr, delivers omniscient debugging with instant navigation and dataflow analysis, and users report roughly 5x faster bug resolution, sometimes 100x. Replay QA explores builds, reproduces failures, and provides deterministic recordings with root cause analysis.Causal tracing. OpenTelemetry lets you understand a system from the outside by tracing the full path of a request. Jaeger implements compatible trace collection and visualization for distributed systems.AI-era observability. Langfuse offers open-source trace logging, cost tracking, and prompt management. LangSmith gives full trace visibility from a single trace up to production-wide metrics. Braintrust surfaces critical patterns in agent traces automatically. OpenTelemetry has standardized GenAI span conventions across OpenAI, Anthropic, and AWS Bedrock.Compiled AI frameworks. Stanford's DSPy compiles programs into optimized, deterministic pipelines, pulling the LLM out of runtime control flow entirely and replacing prompt-response guesswork with reproducible execution. The Behavioral Harness: Deterministic Tools in Production The most mature pattern for wiring deterministic tools into AI coding is the behavioral harness, a two-loop design. The inner loop gives coding agents deterministic feedback during development. The outer loop runs independent automated acceptance tests against the generated service. The inner loop combines guides (instructions that shape the agent) with sensors (deterministic checks that validate output). Contract-driven mocks wire OpenAPI and AsyncAPI specs into the app so downstream interactions are executable, letting the agent verify its work against defined responses, failures, and recovery paths. Thoughtworks draws a useful line between computational controls, run by deterministic tools, and inferential controls, which rely on model judgment. The strongest harnesses combine both, with behavioral checks producing feedback without an LLM judging correctness. Schema tightening is the first line of defense. An "examples validate" step checks payloads against the API spec before any test runs. Sensors then confirm contract conformance, verify HTTP values, and assert observable side effects. Keep guides and sensors aligned, and you get allowlist-style gating: the agent passes only if its output matches the spec. The Result That Changes the Economics In October 2026, Naresh Jain ran a controlled experiment with Specmatic and Thoughtworks: coding agents with and without a behavioral harness, 10 runs each, against 178 automated acceptance tests. Metric Sol (No Harness) Sol (Harness) Luna (No Harness) Luna (Harness) Accepted runs 10/10 10/10 3/10 10/10 Est. cost per run ~$0.41 ~$2.68 N/A ~$0.27 Avg time 6 min 51 sec 19 min 53 sec N/A N/A Read that Luna column again. The frontier model, GPT-6 Sol, passed everything with or without the harness. The cost-optimized GPT-6 Luna succeeded in only 3 of 10 unharnessed runs, then hit 10 of 10 with the harness, at about 33 percent lower cost per run than the frontier model ($0.273 versus $0.406). The takeaway is sharp. Deterministic validation layers can make a cheaper model match frontier reliability. Note the flip side too: the harness added roughly 7x cost to Sol with no quality gain, so harness value depends on the underlying model. Use it where it pays. Humans on the Loop, Not in It Kief Morris of Thoughtworks frames three postures: Humans outside the loop: vibe coding, no oversight.Humans in the loop: line-by-line inspection of every change.Humans on the loop: engineering the harness that validates the work. The "on the loop" model fixes the debugging bottleneck. When output is unsatisfactory, an in-the-loop human patches the artifact directly. An on-the-loop human changes the harness so future runs come out correct. The harness becomes a form of "executable attention," pointing you at the exact contract, response value, or interaction that broke, instead of asking you to read everything. Separate the "why loop" (outcomes) from the "how loop" (code artifacts), and human focus moves to outcome validation rather than implementation detail. The New Job: Harness Engineer The chain from abstraction to skill atrophy to leakage is not a prediction. It is measurable today in churn rates, error frequencies, and benchmark gaps. Girba's reframing reorients the whole problem. The question is not how to preserve our ability to read code. It is how to build systems that stay explainable without anyone having to read the code at all. There is a convergence happening. Moldable development says every question about a system should be answerable through contextual tools. The behavioral harness says deterministic sensors can provide targeted feedback without human inspection. The Specmatic and Thoughtworks experiment shows this is real, not theoretical. So the developer's role is shifting. Less code reader and writer, more harness engineer and system explainer: the person who designs the deterministic tools, specifications, and validation layers that keep AI-generated systems trustworthy. That is not a smaller job. It demands deep system understanding, just expressed through tooling instead of line-by-line reading. The investment priority follows directly. Teams adopting GenAI coding tools should invest proportionally in deterministic contextual tooling: time-travel debuggers, causal tracing, moldable inspectors, behavioral harnesses, and AI-era observability. When the abstraction leaks, and it will, the fallback should be a tool that answers the question, not a developer staring at unfamiliar code. References Glamorous ToolkitGitClear: Coding on CopilotKabir et al., ChatGPT Stack Overflow studyAre Harnesses Worth the Cost? (martinfowler.com)rr projectPernoscoOpenTelemetry observability primerLangfuse docsLangSmith docsDSPyGitClear: Durable Change Velocity 2026SWE-bench paperSWE-benchAutomation biasNielsen Norman Group: AI tools productivity gainsDeskillingAir France Flight 447Glamorous Toolkit siteReplay.ioLangfuse tracingSemantic equivalence studyMoldable DevelopmentJaeger docsBraintrust docsOpenTelemetry GenAI semantic conventionsOpenTelemetry GenAI conventions repoHumans and Agents (martinfowler.com)

By Viquar Khan CORE

In this blog post, we will see how I turned our family's speed test ritual into a family internet speed guessing game that you can play on your own Wi-Fi. Every family has that one argument. "The internet is slow again." "No, it is fine; you are just streaming in 4K." Nobody really knows who is right, because nobody knows the actual speed. So we started guessing before every speed test. It became a game, and the game became a website. Let's see how it works and how I built it. The author describes developing Guess the Speed, a multiplayer game that transforms family internet speed tests into a competitive activity. Players submit secret download and upload guesses before running a real speed test against Cloudflare's network, then earn points based on accuracy using place rankings, bonuses for close guesses, and a cutoff rule for large misses. The post details features including room codes for multiplayer phone participation, a TV-optimized display mode, live result gauges, and shareable results. It explains the technical implementation using Astro, TypeScript, Cloudflare Workers, and Durable Objects for managing game rooms, alongside privacy measures like automatic data deletion and cookieless analytics. The author reflects on lessons learned about focused game design, the appeal of real data, and privacy as a feature. Why I Built It Speed test night on the living-room TV became our family game night. Everyone shouts a guess before I press go. Here is the funny part. My kids get it right almost every time, and Dad loses on purpose. I started my career in performance testing, then moved into development, so I think I am allowed to be bad at guessing speeds. That is my excuse, anyway. At some point I realized this was a real game. It needed scores, secret guesses, and a proper winner. So I built it. How a Round Works A round takes about a minute. Here are the steps: Add players and pick a face for each one.Everyone locks in a secret download guess and a secret upload guess.Run a real speed test against Cloudflare's network.Compare the guesses with the result and score the closest players. That's it. No accounts, no sign-up, nothing to install. As shown below, the home screen keeps all of this on one page, with the live gauge in the middle and the player cards around it. A Sample Round (With Real Scoring) Let's ground this with one simple example. Say the speed test returns 240 Mbps download and 20 Mbps upload. Three players locked in their guesses: PlayerDownload guessUpload guessMom25022D&J20030Dad50050 Now the scoring rules kick in: Place points: first place earns 3 points, second earns 2, and third earns 1.Spot-on bonus: +1 point for each guess (download or upload) within 5% of the real value.50% cutoff: if your average miss is over 50%, you get no place points. Here is how our round shakes out: Mom is closest overall. Her download guess is only about 4% off, so she also gets the spot-on bonus. Total: 3 + 1 = 4 points.Mia lands second with 2 points.Dad guessed way too high. His average miss is far beyond 50%, so he gets 0 points. Dad loses on purpose, remember? In this round, the cutoff rule made sure he lost on merit too. The Features I am Most Happy With Once the basic game worked, I kept adding things that make it feel good in a real living room. Room codes for everyone's phones. Start a room, share a six-letter code or a QR code, and up to 12 players can join. Everyone guesses from their own phone. The server keeps all guesses private until the results are revealed, so nobody can peek and copy. The host also chooses which device runs the test. TV mode. This is a bigger gauge and bigger text, built for a couch ten feet from the screen. It supports fullscreen and uses a screen wake lock where the browser allows it, so the TV does not fall asleep mid-round. A live gauge with clear stages. You can follow the test through ping, download, and upload with live values and result cards. Watching the needle move is half the fun. Share the winner. The champion dialog shows the winner and the scores next to a Share result button. It uses the device share sheet or copies a link. Festive themes. Pick light, dark, or a seasonal theme, or leave it on Auto (by date) and let the calendar decide. Dev diagnostics. This one is for the curious. It shows the summary, latency samples, individual test requests, a bandwidth percentile, the test plan, and browser details. You can copy or download the latest run as JSON. Add ?dev=1 to the URL to open it. How It Is Built I kept the stack small and boring on purpose: Astro and TypeScript build the static site, which is served from Cloudflare Workers.@cloudflare/speedtest runs the real speed test in the browser.A GameRoom Durable Object manages the multiplayer rooms, using WebSocket hibernation so idle rooms cost almost nothing.A PlayStats Durable Object keeps the aggregate play counter.Game and scoring rules live in src/lib and are covered by Vitest tests. A few things I want to call out. Scoring is plain, testable code. Rules like the 50% cutoff and the tie-break are exactly the kind of thing that breaks quietly. Keeping them in src/lib with unit tests meant I could change the rules without fear. Durable Objects fit this problem well. One room is one object, and it holds the secret guesses until reveal. That is a natural fit for "a small piece of shared state that many phones talk to." Testing without waiting. Real speed tests take time, so I added a simulated mode. Append ?mock=1 for a short fake test. Use ?tv=1 to jump straight into the TV layout. To run it locally: Shell npm install npm run dev And to exercise rooms and the play counter, run the Worker locally: Shell npm run dev:worker Deployment is handled by Cloudflare Workers Builds. wrangler.jsonc runs npm run build, then npx wrangler deploy uploads dist as static assets. The Durable Object migrations run as part of that deploy. Everything below comes from the open-source repo at https://github.com/QAInsights/guessthespeed, so you can read the real code. Architecture at a Glance Astro and TypeScript build the static site, served by a Cloudflare Worker.worker/index.ts is the HTTP entry point. It creates rooms and upgrades WebSocket connections.GameRoom (worker/game-room.ts) is a Durable Object. There is one per room, and it owns the sockets and the stored state.src/lib/room.ts and src/lib/scoring.ts hold the rules as plain TypeScript functions with no Cloudflare APIs, covered by Vitest. The design rule I followed: the Durable Object is a thin shell. It loads state, calls a pure function, persists the result, and broadcasts. All the game decisions live elsewhere. One Durable Object Per Room A room code maps straight to an object with idFromName. Every phone that knows the code reaches the same instance, and every message for that room is handled in one place. That means there is no distributed locking to think about. Room codes are six letters from a 19-letter consonant alphabet, which gives roughly 47 million combinations. Creating a room means picking a code and asking its object to initialize itself. If the object already holds a room, it answers 409, and the Worker tries another code. TypeScript // worker/index.ts (simplified) for (let attempt = 0; attempt < 5; attempt += 1) { const code = generateRoomCode(/* crypto-backed random */); const hostToken = crypto.randomUUID(); const stub = env.ROOMS.get(env.ROOMS.idFromName(code)); const response = await stub.fetch("https://room.internal/init", { method: "POST", body: JSON.stringify({ code, hostToken, now: Date.now(), kind }), }); if (response.status === 201) return jsonResponse({ code, hostToken }, 201); if (response.status !== 409) return jsonResponse({ error: "room_creation_failed" }, 503); } return jsonResponse({ error: "room_code_unavailable" }, 503); The collision check needs no separate database. The object's own storage is the source of truth for whether a code is taken. The hostToken is a secret returned only to the creator. Presenting it on connect is what makes a socket the host. WebSockets That Survive Hibernation A room can sit idle for minutes while a family argues about the result, and I did not want to pay for an always-on object. The WebSocket Hibernation API lets the object be evicted from memory while the sockets stay connected. The catch is that anything held in memory is gone when it wakes up. Two things keep a room correct across hibernation: Game state lives in storage, not in fields. Each message handler starts with loadRoom().Per-socket identity lives in the socket attachment. After the first hello message, the object stores who this socket is with serializeAttachment. TypeScript // worker/game-room.ts (simplified) private async connectWebSocket(): Promise<Response> { if (!(await this.loadRoom())) return Response.json({ error: "room_not_found" }, { status: 404 }); if (!roomHasCapacity(this.ctx.getWebSockets().length)) return Response.json({ error: "room_full" }, { status: 503 }); const pair = new WebSocketPair(); this.ctx.acceptWebSocket(pair[1], ["room"]); return new Response(null, { status: 101, webSocket: pair[0] }); } private async handleHello(socket: WebSocket, message: Record<string, unknown>) { // ...validate clientId against a UUID pattern... const room = await this.loadRoom(); const attachment = { clientId: message.clientId, isHost: typeof message.hostToken === "string" && message.hostToken === room.hostToken, }; socket.serializeAttachment(attachment); this.sendState(socket, room, attachment); } The clientId is a random UUID the browser generates once and keeps in local storage. A returning phone sends the same ID, so the server can match it to its player after a dropped connection or a screen sleep. The reply to hello is always a full state snapshot, not a replay of missed events, so a reconnect cannot drift. One tradeoff worth naming: my per-socket rate limiter is an in-memory WeakMap, so it resets when the object hibernates. I treat the limits as a safety net, not accounting, so losing them on wake is acceptable. The Server Is the Referee Clients never change state directly. They send an action, and the server decides. TypeScript // worker/game-room.ts (simplified) const room = await this.loadRoom(); const result = applyAction( room, attachment.clientId, attachment.isHost, parsed, Date.now(), { onlineOwners }, ); if ("error" in result) { sendError(socket, result.error); this.sendState(socket, room, attachment); // resync the sender return; } await this.ctx.storage.put(ROOM_KEY, result.room); this.broadcastState(result.room); applyAction(room, clientId, isHost, action, now) returns either a new room or an error code. It takes now as a parameter instead of reading the clock, which makes it trivial to test. The room tests run actions in sequence and assert on the result, with no sockets involved. When an action is rejected, the sender gets the error plus the current state. That small habit fixed a whole class of stale-UI problems, because the client always snaps back to the truth. Keeping Guesses Secret If the server broadcast one shared state object, anyone could open dev tools and read everyone's guess. So the server never sends the raw room. It builds a view per recipient. TypeScript // src/lib/room.ts (simplified) const reveal = room.game.phase === "results" || room.game.phase === "champion"; players: room.game.players.map((roomPlayer) => { const { owner, ...player } = roomPlayer; delete player.guessedAt; return { ...player, guess: reveal || owner === clientId ? { ...player.guess } : { down: null, up: null }, mine: owner === clientId, }; }), TypeScript // worker/game-room.ts private broadcastState(room: Room): void { for (const socket of this.ctx.getWebSockets()) { const attachment = getAttachment(socket); if (attachment) this.sendState(socket, room, attachment); } } Each socket gets its own projection: your own guess, nulls for everyone else's, until the results phase. The same function strips internal fields like the owner and the guess timestamp. A room test asserts that other players' guesses stay hidden from both players and the host while guessing and testing, then appear at reveal. Cleaning Up Rooms With a Sliding Alarm Rooms should disappear after six hours of inactivity. My first version called setAlarm after every accepted action, which meant a storage write for every guess and every tap. A later commit replaced that with one rule: only schedule an alarm if none exists, and let the alarm re-arm itself. TypeScript // worker/alarm.ts export function alarmToSchedule( existingAlarm: number | null, deadline: number, ): number | null { return existingAlarm === null ? deadline : null; } TypeScript // worker/game-room.ts (simplified) async alarm(): Promise<void> { const room = await this.loadRoom(); if (room && Date.now() < room.updatedAt + ROOM_TTL_MS) { await this.ctx.storage.setAlarm(room.updatedAt + ROOM_TTL_MS); // still active return; } await this.ctx.storage.deleteAll(); for (const socket of this.ctx.getWebSockets()) { try { socket.close(1000, "Room ended"); } catch { continue; } } } The room stores updatedAt. When the alarm fires, it checks whether the room was active since it was scheduled. If yes, it sets a new alarm for the new deadline. If not, it deletes everything and closes the sockets. The result is a sliding time-to-live with at most one alarm in flight. Scoring as Pure Functions Scoring rules are exactly the kind of code that breaks quietly, so they live in src/lib/scoring.ts with no I/O. The core of it is a relative miss for each metric. TypeScript function metricMiss(guess: number | null, actual: number): number { if (!(actual > 0) || !Number.isFinite(actual)) return 1; return guess === null ? 1 : Math.abs(guess - actual) / actual; } // miss = (downMiss + upMiss) / 2 // bonus = Number(rounded(downMiss) <= spotOnPct) + Number(rounded(upMiss) <= spotOnPct) Looking back at our round: Mom's download miss is |250 - 240| / 240, about 4.2%, which is under the 5% spot-on limit. Her upload miss is 10%, so no second bonus. Place assignment skips players over the cutoff and handles ties the way a race does: tied players share a place, and the next place is skipped. JavaScript let place = 0; for (let index = 0; index < eligible.length; index += 1) { if (index === 0 || compare(eligible[index - 1], eligible[index]) !== 0) { place = index + 1; } eligible[index].place = place; eligible[index].placePoints = settings.placePoints[place - 1] ?? 0; } Two small details earned their own tests: Rounding before comparing. Misses are rounded to four decimals before ties are detected. Without that, floating-point noise decides who is tied.A zero or missing real speed. metricMiss returns a full miss instead of dividing by zero, so scores stay finite. The test file has cases for each rule: 3, 2, 1 points, an exactly 50% miss staying eligible, shared ties, the download-then-upload tiebreak, a player with no guess, and fewer than three players. TypeScript it("awards 3, 2, and 1 for first, second, and third", () => { const scores = scoreRound( [player("one", 100, 20), player("two", 90, 18), player("three", 70, 16)], actual, defaults, ); expect(scores.map(({ place, placePoints }) => [place, placePoints])).toEqual([ [1, 3], [2, 2], [3, 1], ]); }); Race Mode and the Server Clock Race mode changes the rule: the host starts the test, and players can guess while it runs. A guess locked in later earns a bigger share of the points, but a guess that misses the result earns nothing. The interesting design question is who owns the clock. Phone clocks drift and phones lag, so every timestamp comes from the Durable Object. Starting the test stores raceStartedAt: now, each guess stores guessedAt: now, and the result computes the timing bonus from the same server clock. TypeScript // src/lib/scoring.ts (simplified) const duration = endedAt - raceStartedAt; const fraction = duration === 0 ? 1 : Math.max(0, Math.min(1, (guessedAt - raceStartedAt) / duration)); const timingBonus = 0.5 + 0.5 * fraction; const total = Math.round(basePoints * timingBonus); The bonus ranges from 0.5 for a guess at the very start to 1.0 for a guess at the last moment. A table-driven test pins the edges and a rounding case. TypeScript it.each([ [0, 0.5, 25], [50, 0.75, 38], // 37.5 rounds up [100, 1, 50], ])("applies timing at %s%% of the test", (at, timingBonus, total) => { const [score] = applyRaceTiming( [baseScore], [{ id: "racer", guessedAt: at }], 0, 100, ); expect(score).toMatchObject({ basePoints: 50, timingBonus, total }); }); Applied to our round, with place points scaled to 30, 20, 10 in race mode: Mom's base is 30 plus a 10-point spot-on bonus, which is 40. If she locks in at 80% of the test, the bonus is 0.9, and she scores 36. Mia's base is 20, and locking in at 20% gives 0.6, so she scores 12. Waiting is rewarded, as long as you do not wait too long. There is one more leak to close. If players could watch the live speed climb, they would copy it in the last second. In race mode, the progress relay forwards only the step counters and drops the measurements. JavaScript const keys = race ? ["step", "steps"] : ["mbps", "pingMs", "step", "steps", "bytes"]; The relay also accepts progress only from the device assigned to run the test, and it is throttled to ten updates per second. Challenges and Lessons A few things that needed more than one attempt: Redundant alarm writes. Resetting the alarm on every action was correct but wasteful. The sliding alarm in section 7 fixed it.Host handover edge cases. Handing the host role to another phone sounds easy. The rules are: only the host can do it, not while a test is happening, not to an offline player, and not to the same device. The offline case shipped as a follow-up fix after a handover to a disconnected player left a room with no live host.Interrupted tests. If the device running the test reloads mid-test, the room can get stuck in the testing phase. A small pure function, interruptedTestStep, decides what the client does next: resend a result it already holds, or abort the round once.Abuse limits on a public endpoint. Messages are capped at 2 KB, each socket at 20 messages per second, rooms at 12 players, and WebSocket upgrades check the origin. Worker request bodies are size-bounded too.Testing real-time behavior without waiting. Because the rules are pure, most tests never open a socket. For the browser, a ?mock=1 flag runs a short fake speed test, so a full round takes seconds. What I would tell someone starting a similar project: Keep the Durable Object thin and move every decision into functions that take now as an argument.Never broadcast shared state when players should not see each other's data. Project a view per socket.Put the identity in the state in storage, and assume the object can wake up empty at any time.Prefer one self-rescheduling alarm over resetting a timer on every message. Privacy Decisions A game for families should not feel like a tracking machine, so I made a few choices early: In single-screen mode, settings, player names, and scores stay in your browser's local storage.Room names, faces, guesses, and scores are deleted after six hours without activity.The play counter stores totals only, and Cloudflare Web Analytics is cookieless.The speed test itself runs in your browser against Cloudflare's network. You can read the full details on the privacy page. Try It On Your Own Wi-Fi Here is how to play your first round: Head to https://guessthespeed.com/.Add everyone playing and pick a face for each person.Lock in your secret guesses (or have people join from their phones with the room code).Press go and watch the gauge.Check the scores and share the champion. If you want ideas for a full evening, there is a family game-night guide on the site. There is also a guide that explains how the real speed test works, and a 30-minute classroom activity for kids. What I Learned A few takeaways from this build: Small, focused games beat big ideas. The whole concept fits in one sentence, and that makes every decision easier.Real network data makes people care. A made-up number is boring. A real measurement on your own Wi-Fi starts arguments, which is exactly what a game needs.Privacy is a feature. Deleting rooms after six hours and keeping analytics cookieless cost me very little, and they make the game easier to recommend to other families. So that is Guess the Speed. A speed test, a few secret guesses, and a family that cannot stop arguing about the Wi-Fi. Which of your guesses would be furthest off: your download speed or your upload speed? Let me know in the comments below. Happy gaming!

By NaveenKumar Namachivayam CORE