DZone Spotlight

Wednesday, October 28 View All Articles »

Searching Kafka Without an Index: Lessons From Building Message Search for Kafdrop

The message left the first service at 2:14 PM and never reached the last one. Between those facts: four pipeline stages, six topics, a pub/sub API moving every record. The engineer on call had Kafdrop open, the offsets looked fine, and nobody could say which stage had eaten the record. That gap is why we built message search for Kafdrop at a hackathon. The setup was familiar from my day job: a platform moving data between pipeline stages over a pub/sub API, with no way to ask where a message went when something breaks. Kafdrop showed partitions and offsets. It couldn't search inside the messages. Kafka has no index. That is the whole problem in one sentence. A topic is an append-only log split across partitions, and the consumer API reads by offset: hand it 41,207, and it returns record 41,207. There is no "find the record containing order-88412." To search, you pick a starting point and read forward until you find what you want or stop. So that is what our feature does, with guardrails that keep the search from hurting the cluster. The interesting part was the caps, the consumer setup, and the SRE agreement. Why a Full Scan Is Not an Option The obvious approach is to read the whole topic and grep. Topics hold days of data, so the whole thing doesn't fit in memory. Reading every record on every search hammers the brokers, and during an incident that is the worst time to add load. Most searches end with a few matches, so nearly all of that work is wasted. The other approach is to mirror every topic into a search engine. That works, and plenty of teams do it. It also means running a second distributed system and explaining index lag during every incident. For a hackathon debugging tool, that was too much machinery. The Design: Seek, Scan, and Stop The feature takes four inputs: a topic, a query string, a start time, and two caps. The start time bounds how far back the search goes. Kafka's offsetsForTimes maps a timestamp to the first offset at or after it per partition, or null when the partition holds nothing that new. The user picks "the last hour" or "since the deploy." The result cap bounds memory: return the first N matches and stop. Debugging usually needs one record, not ten thousand. The scan cap bounds broker load: stop after M records even with no match. Every search becomes a fixed amount of work, whatever the user types. A typo costs M reads. The scan can never grow beyond that. Polls use bounded batches, so the tool never holds more than one page of records at a time. Nothing loads a whole topic. Safety: Assign, Don't Subscribe The consumer setup is where a search tool can quietly cause damage. Three decisions were non-negotiable. First, the search consumer uses assign, not subscribe. Subscribe joins a consumer group, bringing coordination and rebalances. A debugging tool has no business in any of that. Assign hands the consumer a fixed partition list and stays out of group coordination entirely. Second, the consumer never commits offsets. Auto-commit is off, and the code never calls commit. The default config commits automatically, so this is the one that bites: a search tool that advances a production group's offsets is a horror story, and the fix is one config line. Third, the scan cap doubles as a safety control: every search reads at most M records, a worst case SRE can read before approving. The Search Loop Here is the core in Java. The three constants are the whole design: matches to return, records to read in the worst case, records per poll. The values are illustrative; tune them to your topics. Java private static final int RESULT_CAP = 25; private static final int SCAN_CAP = 50_000; private static final int POLL_BATCH = 500; public List<FoundMessage> search(String topic, String query, long startTimeMs) { Properties props = new Properties(); props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, brokers); props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, String.valueOf(POLL_BATCH)); List<FoundMessage> matches = new ArrayList<>(); try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) { List<TopicPartition> partitions = new ArrayList<>(); for (PartitionInfo p : consumer.partitionsFor(topic)) { partitions.add(new TopicPartition(topic, p.partition())); } consumer.assign(partitions); Map<TopicPartition, Long> times = new HashMap<>(); for (TopicPartition tp : partitions) times.put(tp, startTimeMs); for (Map.Entry<TopicPartition, OffsetAndTimestamp> e : consumer.offsetsForTimes(times).entrySet()) { if (e.getValue() != null) consumer.seek(e.getKey(), e.getValue().offset()); else consumer.seekToEnd(Collections.singletonList(e.getKey())); } long scanned = 0; while (scanned < SCAN_CAP && matches.size() < RESULT_CAP) { ConsumerRecords<String, String> batch = consumer.poll(Duration.ofMillis(500)); if (batch.isEmpty()) break; for (ConsumerRecord<String, String> r : batch) { if (++scanned > SCAN_CAP) break; if (r.value() != null && r.value().contains(query)) { matches.add(new FoundMessage(r.topic(), r.partition(), r.offset(), r.timestamp(), r.value())); if (matches.size() >= RESULT_CAP) break; } } } } return matches; } From the top: the config disables auto-commit and fixes the poll batch size. Assign takes a fixed partition list. offsetsForTimes turns the start time into a starting offset per partition; the null branch seeks to the end for partitions with nothing that new. The loop reads forward and stops at whichever cap hits first. Nothing commits, and try-with-resources closes the consumer. Matching is a substring check on the record value. Tombstones carry nulls, so the null check is load-bearing. Shipping It: A Container, Kubernetes, and the SRE Agreement The app ships as a container image: the search service and its config, nothing else. It runs on Kubernetes as a small deployment with tight resource limits, since the result cap already bounds memory. One replica is plenty. YAML apiVersion: apps/v1 kind: Deployment metadata: name: kafdrop-search spec: replicas: 1 selector: matchLabels: app: kafdrop-search template: metadata: labels: app: kafdrop-search spec: containers: - name: search image: registry.example.com/kafdrop-search:1.0.0 resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" env: - name: KAFKA_BROKERS value: "kafka:9092" The manifest is deliberately boring: one replica, tight memory limits, brokers passed as config. The SRE conversation was short, because the design answered their questions first. The tool only reads: assign instead of subscribe, no offset commits, read-only Kafka credentials. Every search is bounded by the scan cap, so the worst case was a number everyone could read. That agreement mattered more than the code. The caps were the trust. A Debugging Routine That Works First, search the topic for the message using something unique: an order ID, a request ID. Found it? Note the offset and timestamp. Second, check consumer group lag for the downstream stage. If the group consumed past your message's offset, the stage read it, and the problem is in the consumer's logic. If lag is stuck or growing, the consumer is behind, crashed, or wedged on a poison record. Third, look at the consumer: logs around the message's timestamp, error handling, the dead-letter path. The order matters. It separates "the message never arrived" from "the message arrived and something went wrong." Before, step one was guesswork. Now it takes minutes. What We Kept Kafka gives you ordering and durability, not search. Bolting search onto an append-only log means accepting the scan. Make it bounded and boring: seek to a timestamp, read forward in small batches, stop at the first cap. Keep the three decisions that made it safe. Assign instead of subscribe, so the tool never joins a consumer group. Never commit offsets, so reading changes nothing. A scan cap that turns every search into a known quantity of broker work. Get those right, and SRE will let you run it against production. Get them wrong, and you have built a very efficient way to cause your next incident. After the hackathon, the team kept the tool for one job: finding where a message stopped. A search, a lag check, and a look at the consumer, done in minutes. Diagnosis time during incidents fell. More

Designing Memory Architecture for Stateful AI Agents: Short-Term Context and Long-Term Persistence

By Kapil Shirish Bidikar

A language model holds nothing between calls. Each request arrives with no memory of the previous one, and the model responds from the prompt in front of it and nothing more. For a one-shot question, that is fine. For an agent a person returns to across days, threads, and separate tasks, it becomes the defining design constraint. The behavior users expect from an agent, recalling what they said last week, resuming a task that stalled, applying their stated preferences without being asked again, is precisely the behavior a stateless model cannot produce on its own. A large language model (LLM) does not forget your context because it chose to. It never had it. The reflex fix is to paste the history back in. Keep a buffer of prior turns and prepend it to every prompt, so the agent looks like it remembers. That illusion survives a short demo and breaks in production. Building stateful AI agents that hold continuity across sessions takes a memory architecture, not a bigger buffer. Why Stateless Models Break Multi-Session Agents The conversation-buffer approach fails in three measurable ways, and they compound. The first is cost. Every prior turn you replay is billed again on the next call, so a buffer that grows linearly turns a fifty-turn session into fifty replays of the same opening tokens. The second is latency. Larger prompts take longer to transmit and to process, and that overhead lands on every turn, not only the long ones. The third is the one that surprises teams: recall does not hold up as the prompt grows. Research on long-context behavior, notably Liu and colleagues in 2023, found that models attend most reliably to information at the start and end of a prompt and least reliably to whatever sits in the middle. Bury a critical instruction in turn nine of a forty-turn transcript and the model may act as though it is not there. There is a harder limit underneath all of this. Every model has a fixed context window, commonly between 8,000 and 128,000 tokens depending on the model. Cross it, and the provider either rejects the request with a context-length error or silently truncates the oldest turns, which means the agent loses exactly the early context the buffer was meant to preserve. None of this touches the cross-session problem. A buffer lives inside one session. Open a new thread tomorrow, and the buffer is empty again. The preferences and facts the user established yesterday are gone, because nothing extracted them into a place that outlives the session. So the standard approach does not scale, and the reason is that it conflates two different jobs: holding the immediate conversation and remembering across conversations. Those are separate problems with separate lifetimes, and they need separate machinery. Measuring Context Pressure in a Live Agent Before designing anything, instrument what you already have. The failure modes above stay invisible until you measure token consumption per turn against the model's window, and most teams find the ceiling only when a request errors out in front of a customer. A small instrument that tracks tokens as the session grows turns a production incident into a logged warning. Python # context_budget.py # Instruments an agent's session log so you can see context pressure build # up turn by turn, instead of discovering it when a request silently truncates # or the provider returns a context-length error mid-conversation. from dataclasses import dataclass, field # In production, count with the model's real tokenizer (for example tiktoken's # cl100k_base or the o200k_base encoding). We wrap it so a missing encoding or # offline build degrades to a conservative heuristic instead of crashing the # request path, which is what you actually want in a hot loop. def count_tokens(text: str) -> int: try: import tiktoken return len(tiktoken.get_encoding("o200k_base").encode(text)) except Exception: # ~4 characters per token is a safe upper-bound heuristic for English. return (len(text) + 3) // 4 @dataclass class SessionLog: budget: int # hard ceiling for prompt tokens events: list = field(default_factory=list) def add(self, role: str, text: str) -> None: self.events.append((role, text, count_tokens(text))) def used(self) -> int: return sum(t for _, _, t in self.events) def pressure(self) -> float: # Fraction of the budget consumed. Alert teams above ~0.75 so there is # headroom for the model's own output tokens, not just the prompt. return self.used() / self.budget log = SessionLog(budget=2000) for turn in range(1, 13): log.add("user", "Walk me through the failure we saw in the nightly run. " * turn) log.add("assistant", "Here is the breakdown of the root cause and the fix. " * turn) if log.pressure() > 0.75: print(f"turn {turn:>2}: used={log.used():>5} tokens " f"pressure={log.pressure():.0%} ACTION: evict or summarize") else: print(f"turn {turn:>2}: used={log.used():>5} tokens pressure={log.pressure():.0%}") The threshold matters. Alerting at seventy-five percent of the window rather than at one hundred leaves headroom for the model's own output tokens, which are drawn from the same budget. Count tokens with the model's actual tokenizer, since the tiktoken library exposes the o200k_base and cl100k_base encodings used by current models, because a whitespace estimate can be off by a wide margin on code, punctuation, or non-English text. The heuristic fallback in the wrapper exists so the request path degrades to an estimate rather than throwing when an encoding is unavailable. Once you can watch pressure climb, the case for a second memory layer stops being theoretical. A Two-Layer Memory Architecture for Stateful AI Agents The architecture that holds up splits memory by lifetime. Short-term memory is the raw event log of the current session: every user turn and agent response, appended cheaply, scoped to one interaction, archived or discarded when the session ends. Long-term memory is the distilled residue, the facts, preferences, and summaries worth keeping, extracted from raw events and persisted across every future session. Short-term memory answers what just happened in this conversation. Long-term memory answers what the system knows about this user. Keep both in one store and you will either bloat the prompt with raw transcript or lose durable facts in the churn. Long-term memory is shared infrastructure, often across many tenants, so the first structural decision is addressing. Records are keyed by a hierarchical namespace rather than a flat identifier, and the hierarchy itself enforces isolation. A retrieval scoped to one tenant cannot resolve a key that belongs to another, because the tenant identifier is a structural prefix of every key rather than a filter applied after the fetch. Python # namespaces.py # Memory records are addressed by a hierarchical namespace, not a flat key. # The hierarchy is what enforces multi-tenant isolation: a retrieval scoped to # one tenant can never resolve a key that belongs to another, because the # tenant id is a structural prefix of every key rather than a filter applied # after the fetch. import re # Components are validated so a caller cannot smuggle a separator into an id and # escape its own subtree. This is the single most common isolation bug in # home-grown memory stores: an unescaped "/" in a user id reads as a new level. _SAFE = re.compile(r"^[A-Za-z0-9_-]+$") def make_namespace(tenant: str, actor: str, scope: str) -> str: for part in (tenant, actor, scope): if not _SAFE.match(part): raise ValueError(f"unsafe namespace component: {part!r}") return f"/{tenant}/{actor}/{scope}" class MemoryStore: def __init__(self): self._records: dict[str, list[str]] = {} def put(self, namespace: str, value: str) -> None: self._records.setdefault(namespace, []).append(value) def query(self, namespace_prefix: str) -> list[str]: # Only keys under the caller's prefix are visible. The prefix check is # the isolation boundary, so it runs before any ranking or scoring. hits = [] for key, values in self._records.items(): if key == namespace_prefix or key.startswith(namespace_prefix + "/"): hits.extend(values) return hits store = MemoryStore() store.put(make_namespace("tenant-a", "user-42", "preferences"), "prefers metric units") store.put(make_namespace("tenant-a", "user-42", "facts"), "deploys to two regions") store.put(make_namespace("tenant-b", "user-42", "preferences"), "prefers imperial units") print("tenant-a/user-42 sees:", store.query("/tenant-a/user-42")) print("tenant-b/user-42 sees:", store.query("/tenant-b/user-42")) # A caller trying to break out of its subtree by embedding a separator is # rejected at construction time, before any record is written or read. try: make_namespace("tenant-a", "user-42/../tenant-b", "preferences") except ValueError as exc: print("isolation guard:", exc) The validation is not decoration. The most common isolation defect in a home-grown memory store is an unescaped separator in a user-supplied identifier, where a value shaped like a path fragment reads as a new namespace level and lets a caller climb out of its own subtree. Rejecting unsafe components at construction time closes that hole before a single record is written or read. Raw events are cheap to append and nearly useless to search. Turning them into retrievable knowledge is the job of an extraction pipeline, and that pipeline belongs off the request path. Distillation calls a model and costs hundreds of milliseconds, so running it inline would tax every turn with work the user never sees. An asynchronous worker reads batches of raw events and writes back typed long-term records. Python # extraction_worker.py # Raw session events are cheap to append and useless to search. The extraction # worker runs off the request path, reads batches of raw events and distills # them into durable long-term records: facts, preferences and rolling # summaries. Keeping this asynchronous is the point. Distillation calls a model # and takes hundreds of milliseconds, so doing it inline would tax every turn. import queue import threading import time raw_events: "queue.Queue[dict]" = queue.Queue() long_term: list[dict] = [] def distill(batch: list[dict]) -> list[dict]: # PLACEHOLDER for the model call. In production this is a structured-output # request to an extraction model that returns typed records. The rule-based # stub below keeps the worker deterministic and runnable for tests; swap it # for the real call and keep the same return contract. records = [] for ev in batch: text = ev["text"].lower() if "prefer" in text: records.append({"type": "preference", "value": ev["text"], "src": ev["id"]}) elif any(k in text for k in ("region", "deploy", "cluster", "schema")): records.append({"type": "fact", "value": ev["text"], "src": ev["id"]}) return records def worker(batch_size: int = 4, idle_flush: float = 0.2) -> None: batch: list[dict] = [] last = time.monotonic() while True: try: ev = raw_events.get(timeout=idle_flush) if ev is None: # poison pill: drain then stop if batch: long_term.extend(distill(batch)) break batch.append(ev) except queue.Empty: pass # Flush on a full batch or after an idle gap, so a quiet session still # gets its memory persisted instead of waiting for more traffic. if len(batch) >= batch_size or (batch and time.monotonic() - last > idle_flush): long_term.extend(distill(batch)) batch, last = [], time.monotonic() t = threading.Thread(target=worker, daemon=True) t.start() for i, msg in enumerate([ "I prefer metric units in reports", "we deploy to two regions", "the order schema changed last week", "thanks, that helps", "I prefer dark themed dashboards", ]): raw_events.put({"id": f"e{i}", "text": msg}) raw_events.put(None) t.join(timeout=2) for rec in long_term: print(rec) Two details in that worker earn their place. The batch flushes on a size trigger or an idle timeout, so a quiet session still gets its memory persisted instead of waiting for traffic that never arrives. The drain-on-shutdown path matters even more: an earlier version that broke on the poison pill without flushing the partial batch silently dropped the last few events whenever the worker stopped, the kind of lost-write bug that surfaces as missing memories weeks later. The model call is stubbed here with deterministic rules so the worker stays testable. The contract is the point. Swap the stub for a structured-output extraction call, and the rest holds. With both layers in place, a single agent turn reads from each, assembles a prompt within the token budget, and persists the new event for later extraction. Python # agent_turn.py # One agent turn that reads from both memory layers and respects a token # budget. Long-term records come first because they are dense and stable; # recent raw turns fill whatever budget remains. This ordering matters: models # attend less reliably to the middle of a long prompt, so the highest-value # memory belongs near the edges, not buried in the transcript. def count_tokens(text: str) -> int: return (len(text) + 3) // 4 # heuristic; production swaps in the real tokenizer def assemble_context(long_term: list[str], recent: list[str], budget: int) -> list[str]: context, used = [], 0 # Long-term memory is admitted first, up to half the budget, so a long # transcript can never crowd out durable facts the agent depends on. for rec in long_term: cost = count_tokens(rec) if used + cost > budget // 2: break context.append(rec) used += cost # Recent turns fill the remainder, newest first, dropping the oldest. for turn in reversed(recent): cost = count_tokens(turn) if used + cost > budget: break context.append(turn) used += cost return context def run_turn(user_msg, long_term, recent, budget=400): # Reserve room for the user message before admitting any memory, otherwise # the assembled prompt overshoots the budget once the message is appended. reserve = count_tokens(f"user: {user_msg}") + 1 context = assemble_context(long_term, recent, budget - reserve) prompt = "\n".join(context + [f"user: {user_msg}"]) # PLACEHOLDER for the model call. The returned context window is what the # agent actually reasons over; assert it before spending tokens on inference. assert count_tokens(prompt) <= budget, "context exceeds budget" recent.append(f"user: {user_msg}") # short-term log keeps growing return prompt long_term = ["fact: deploys to two regions", "preference: prefers metric units"] recent = [f"user: earlier message {i}" for i in range(20)] prompt = run_turn("why did the canary roll back?", long_term, recent, budget=120) print(prompt) print("---") print("prompt tokens:", count_tokens(prompt), "of budget 120") Ordering is a deliberate choice. Long-term records go in first and are capped at half the budget so a long transcript can never crowd out the durable facts the agent depends on; then recent turns fill the remainder, newest first. Because models attend least reliably to the middle of a prompt, the highest-value memory belongs near the edges. Reserving room for the user message before admitting any memory keeps the assembled prompt from overshooting once the message is appended, a bug the budget assertion catches the moment it slips. Trade-Offs in Extraction Timing, Retrieval and Consistency Every decision in this design buys one property at the cost of another, and pretending otherwise is how teams get surprised later. Asynchronous extraction keeps turn latency flat, but it introduces an eventual-consistency window: for the few seconds between an event and its distillation, the long-term layer does not yet know what the user just said. For most assistants, that lag is invisible. For a workflow where the user states a fact and immediately expects the agent to act on it across a session boundary, you either shorten the window or read through to the short-term log for the current session. Synchronous extraction removes the lag and adds those hundreds of milliseconds to every turn instead. Pick the trade deliberately. Retrieval carries its own tension. Score candidates purely on semantic similarity and a stale record can win, because a preference the user set months ago and has since reversed still matches the query embedding closely. Blend a recency decay into the score and freshness gets a vote. Python # ranking.py # Retrieval that scores only on semantic similarity will happily surface a # year-old preference the user has since reversed. Blending a recency decay # into the score fixes that, but the decay rate is a real tuning knob: too # aggressive and the agent forgets stable facts, too gentle and stale memory # wins. This compares pure similarity against a recency-weighted score on the # same candidate set so the trade-off is visible. import numpy as np # query vector and candidate vectors stand in for embeddings; in production # these come from an embedding model and a vector index, but the ranking math # is identical regardless of where the vectors originate. query = np.array([0.9, 0.1, 0.0]) candidates = [ {"text": "prefers metric units", "vec": np.array([0.88, 0.12, 0.0]), "age_days": 210}, {"text": "prefers imperial units", "vec": np.array([0.86, 0.14, 0.0]), "age_days": 3}, {"text": "works on payments", "vec": np.array([0.20, 0.80, 0.1]), "age_days": 12}, ] def cosine(a, b): return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b))) def rank(decay_per_day: float): scored = [] for c in candidates: sim = cosine(query, c["vec"]) # Exponential recency decay. A fresher record with slightly lower # similarity can now outrank a near-duplicate that is months old. recency = np.exp(-decay_per_day * c["age_days"]) blended = sim if decay_per_day == 0 else 0.6 * sim + 0.4 * recency scored.append((round(float(blended), 3), c["text"])) return sorted(scored, reverse=True) print("pure semantic: ", rank(decay_per_day=0.0)[0]) print("recency-weighted 0.01:", rank(decay_per_day=0.01)[0]) print("recency-weighted 0.05:", rank(decay_per_day=0.05)[0]) The decay rate is a real knob, not a default to ignore. Too aggressive and the agent forgets stable facts that should persist for months; too gentle and outdated memory keeps surfacing. Notice in the output how a record that is three days old overtakes a months-old record once recency is weighted at all, even though the older record scores marginally higher on similarity alone. The right constant depends on how fast your domain's facts actually change, which is something only the people running the system can calibrate against real traffic. There is also a quieter cost to the whole approach. Extraction and embedding are not free, since every event you choose to distill and index consumes compute and storage, so indexing everything is rarely the right call. Selective extraction, keeping facts, preferences, and periodic summaries rather than raw chatter, keeps the long-term store small enough to search quickly and cheap enough to justify. Teams that skip the two-layer split do not avoid this cost. They pay it as a degraded user experience: repetitive agents that re-ask for the same information, prompts that balloon until they error, and memory that evaporates the moment a session closes. Separating short-term context from long-term persistence is what turns a stateless model into an agent that behaves as though it remembers, and it stays an ordinary software engineering problem once you stop expecting the model to solve it for you. More

Part 4: Multi-Agent Orchestration — Pairing Goose With Quarkus Flow State Machines

By Daniel Oh

CORE

Designing Frontend Applications for Failure, Not Just Success

By Somya Biswari

Testing Java and Spring Applications With Kotlin: Low-Risk to Adoption

By Aniket Bhattacharyea

Running a Self-Hosted WAF in Production: Practical Lessons From SafeLine

By Carrie Luo

Microsoft Fabric Throttling: Diagnose Before You Scale

If Your Fabric Capacity Is Throttling, Do Not Start With the SKU A report takes twenty seconds to open. A pipeline that normally finishes before breakfast is still running. Someone sees the capacity above 100% and proposes the obvious fix: move from F64 to F128. That might be the right decision. It should not be the first one. Microsoft Fabric brings data engineering, warehousing, real-time analytics, data science, and Power BI onto shared capacity. That is one of its strengths. It also means one inefficient item can consume compute that another workload needs. A poorly shaped warehouse query, an oversized Spark session, an hourly refresh of data that changes once a day, or a report page with too many expensive visuals can all surface as the same complaint: “Fabric is slow.” The complaint does not tell you the cause. The useful operating model is: Confirm the capacity is throttling. Isolate the operation creating pressure. Optimize that workload. Scale only when the remaining demand is legitimate. This distinction matters because a larger SKU gives inefficient work more room to run. It does not make the work efficient. First, Separate Slow Performance From Throttling Slow performance and capacity throttling can look similar to a user, but they are different problems. A report visual can be slow because its DAX is expensive. A warehouse query can be slow because it scans far more data than it needs. A Spark job can wait because it requested more executors than are available. Network latency, gateway pressure, cold cache, locks, and source-system performance can also add delay. Throttling is more specific. It happens when billable operations consume more Capacity Unit seconds than the capacity can supply, including the future capacity Fabric uses to absorb smoothed usage. When that budget is exhausted, Fabric progressively delays or rejects new work. This leads to the first rule: Do not diagnose throttling from a slow screen or a utilization spike. Confirm it in the Capacity Metrics app. Microsoft explicitly notes that utilization above 100% does not immediately mean throttling. Fabric allows bursting, smooths usage, and provides limited overage protection before user operations are affected. Use the Throttling view and System events on the Compute page to confirm whether the capacity entered interactive delay, interactive rejection, or background rejection. Rejected requests can also surface errors such as CapacityLimitExceeded. If the app shows no throttling during the incident, investigate the item itself. Buying capacity to fix a bad query is an expensive way to avoid reading its execution plan. What Fabric Is Protecting You From Fabric does not force every operation to fit inside the CU budget of the current second. It uses two mechanisms to absorb variability. Bursting lets an operation temporarily use more compute than the provisioned SKU provides so it can finish quickly. Smoothing spreads the CU accounting for that operation across future 30-second timepoints. Interactive operations are typically smoothed over five to 64 minutes, depending on consumption. Background operations are typically smoothed over 24 hours. Smoothing is not a performance optimization. It does not make a query cheaper or slow a job down to fit the capacity. It changes how consumed compute is applied against the capacity limits. When smoothed usage continues to accumulate faster than idle capacity can pay it down, Fabric moves through progressive protection stages: Future capacity consumedProtection stageUser impactUp to 10 minutesOverage protectionNew operations continue without throttlingMore than 10 minutes, up to 60 minutesInteractive delayNew interactive operations receive a 20-second submission delayMore than 60 minutes, up to 24 hoursInteractive rejectionNew interactive operations are rejectedMore than 24 hoursBackground rejectionNew interactive and background operations are rejected Operations already running are allowed to complete. The capacity recovers when unused CUs burn down the accumulated carryforward. That delayed accounting explains a common mystery. The item users see failing at 10:00 AM might not be the item that created the pressure at 10:00 AM. A large background operation completed earlier can still be consuming future capacity through smoothing. Diagnose From the Capacity Down, Not From the complaint up When a capacity incident occurs, use the same investigation path every time. 1. Pin Down the Affected Window Record the time, capacity, workspace, reported symptom, and whether the failure affected interactive work, background work, or both. Do not begin with a full month of data. Start with the narrowest window that contains the incident. The Capacity Metrics app shows the previous 14 days of compute usage and displays hourly usage in the multimetric ribbon. Select the affected day and hour. When filters are applied, the utilization and throttling visuals can show 30-second timepoints rather than only periodic peaks. 2. Confirm the Protection Stage Open the Throttling tab and check which threshold crossed 100%: Interactive delay means the capacity consumed more than 10 minutes of future capacity.Interactive rejection means it consumed more than 60 minutes.Background rejection means it consumed more than 24 hours. Then review Overage (Carryforward). The add, burndown, and cumulative values tell you whether the capacity experienced a short surge that quickly recovered or sustained demand that kept adding debt. Minutes to burndown is especially useful. It estimates how long recovery would take if no new operations entered the capacity. 3. Find the Items That Consumed the CUs In the matrix by item and operation, sort by CU(s) for the selected window. Expand the highest consumers to see the operation type. Duration alone is not enough. A long-running operation can use modest compute, while a short burst can consume many CUs. Add the optional columns that help explain the incident: Rejected countFailed countSuccessful countIn-progress countOverloaded minutesItem sizePerformance delta Hovering over matrix values exposes the operation-level breakdown. Capture the workspace, item, operation, user or service principal, CU seconds, operation count, and time. 4. Classify the Shape of the Problem Most incidents fall into one of four patterns: PatternWhat the evidence looks likeLikely responseOne expensive operationA single run dominates CU secondsOptimize code, query, model, or data layoutExcessive frequencyA modest operation runs far more often than its data changesReduce refresh or execution frequencyHigh concurrencyMany acceptable operations overlapControl concurrency, isolate workloads, or scaleSustained business demandOptimized work remains consistently highScale up or scale out This classification prevents random tuning. It tells you whether to reduce cost per operation, reduce operation count, change where the work runs, or provide more compute. Reducing frequency and rescheduling are not the same optimization. Changing an hourly refresh to a daily refresh removes unnecessary executions and lowers total consumption. Moving the same 24 hourly refreshes to a different part of the day does not make them cheaper. It only changes when they compete with other work. Optimize the Workload That Actually Caused the Pressure Fabric workloads share capacity, but they do not share one tuning playbook. Once you know which operation consumed the CUs, focus on that workload instead of trying to optimize everything. Warehouse and SQL Analytics Endpoint Start with Query Insights to identify completed queries using the most CPU. Microsoft's Query Insights documentation includes this query: SQL SELECT TOP 100 distributed_statement_id, query_hash, allocated_cpu_time_ms, label, command FROM queryinsights.exec_requests_history ORDER BY allocated_cpu_time_ms DESC; If a query stands out, check how much data it scans, whether filters are applied early, and whether joins or statistics need attention. Compare repeated executions, not just the first run, because cold starts can distort the picture. The goal is fewer resources consumed for the same result, not simply a faster first response. Spark, Notebooks, and Lakehouse Processing Use Spark monitoring and the Tune tab to look for heavy shuffles, skewed stages, spills, or executors that sit underused. Filter data early, avoid repeated actions, and review Delta file layout when small files become a recurring problem. Also check session lifetime. Leaving compute running after the work is done is easy to overlook. If bursty Spark workloads regularly compete with reporting, consider whether separately billed on-demand Spark is a better fit. Measure the cost and isolation trade-off before making the switch. Data Factory Pipelines and Dataflow Gen2 A pipeline can orchestrate several expensive operations, so the pipeline itself may not be where the CUs are going. Check the notebook, copy, refresh, or transformation activity behind it. The simplest savings often come from removing unnecessary runs, using incremental loads, and preserving query folding in Dataflow Gen2 where supported. Moving the same work to another hour might reduce contention, but it does not reduce total CU consumption. Removing work you do not need does. Power BI Semantic Models and Reports Separate refresh pressure from interactive report pressure. For refreshes, focus on incremental processing, folding, and schedules that match how often the source changes. For reports, use Performance Analyzer to find expensive visuals and DAX measures, then compare those findings with the semantic model's operations in Capacity Metrics. A page with too many costly visuals can feel like a capacity problem even when the better fix is in the report design. Real-Time Intelligence For KQL databases and Eventhouses, inspect wide queries, repeated aggregations, retention and caching choices, and ingestion patterns. Limit scans to the data actually needed and consider pre-aggregation when queries repeatedly calculate the same result. Keep the freshness requirement in view. Lower CU consumption is not a win if the data now arrives too late to be useful. Real-Time Intelligence also has different throttling behavior, so do not assume every workload uses the same interactive delay stage. Reduce Blast Radius When Optimization Is Not Enough Some workloads are efficient and still should not share a failure boundary. An executive reporting estate, a self-service experimentation workspace, and a large overnight engineering platform can have different service expectations. If one workload is allowed to consume the headroom needed by another, the architecture is coupling them operationally even if their data products are separate. Fabric gives you several options: Scale up when optimized demand legitimately exceeds the current SKU.Scale out by moving workspaces to another capacity when isolation, administration, or service levels differ. This adds capacity cost and operational ownership, so the service-level benefit must justify it.Use on-demand billing for Spark when dynamic Spark demand should not consume the shared capacity.Configure surge protection to limit the background compute that can consume capacity and make interactive throttling less likely.Enable capacity overage billing when continuity is worth the additional cost and the feature fits your operating model. Microsoft documents overage billing at three times the normal capacity rate, so estimate the cost first. Surge protection is a guardrail, not free capacity. When it activates, background jobs can be rejected. Microsoft recommends correctly sizing and isolating critical solutions rather than treating surge protection as a replacement for optimization or capacity design. Prove That the Optimization Worked A workload is not optimized because its code changed or one test completed faster. Compare the same business operation before and after the change. Track: MeasureWhy it mattersCU seconds per operationShows whether the work became cheaperOperation countExposes unnecessary frequency and retriesTotal CU secondsShows the net capacity effectDurationMeasures user or batch completion timeData scannedReveals pruning, cache, and query-shape improvementsRejected and failed countConfirms whether capacity pressure still disrupts workCarryforward and minutes to burndownShows whether the capacity recovers fasterInteractive latencyConfirms that users experience the improvement Run the comparison over a representative period. A change that improves one query but shifts expensive work into refresh, maintenance, or another engine is not an end-to-end optimization. For recurring operations, calculate both cost per run and daily cost: Plain Text Daily CU-seconds = CU-seconds per operation × operations per day Reducing a query from 1,000 to 800 CU seconds saves 20% per run. If it runs ten times more often after the redesign, total consumption still increases substantially. Efficiency and frequency belong in the same calculation. A Practical Throttling Response Worksheet Use this during the next capacity incident: Plain Text Incident window: Capacity and SKU: User-visible symptom: Confirmed throttling stage: Carryforward and minutes to burndown: Top workspace: Top item: Top operation: Interactive or background: CU-seconds: Operation count: Rejected count: Problem shape: [ ] One expensive operation [ ] Excessive frequency [ ] High concurrency [ ] Sustained business demand Optimization applied: Before CU-seconds per operation: After CU-seconds per operation: Before daily CU-seconds: After daily CU-seconds: Problem shifted to refresh, maintenance, or another engine: Decision: [ ] Keep current SKU [ ] Scale up [ ] Isolate on another capacity [ ] Move Spark to on-demand billing [ ] Add a protective capacity control This turns a vague performance complaint into a traceable engineering decision. The Goal Is Not Zero Utilization A well-used Fabric capacity should do work. The goal is not to keep every chart comfortably low or eliminate every temporary excursion above 100%. The goal is to ensure that each operation earns the compute it consumes, critical workloads retain the service level they need, and sustained business demand fits inside a deliberate capacity envelope. When throttling appears, ask these questions in order: Did the capacity actually throttle, or is one item simply slow?Which item and operation consumed the CUs?Is the problem cost per operation, frequency, concurrency, or real growth?Can the workload be made cheaper without reducing the business outcome?If not, should the answer be more capacity or better isolation? That is the difference between reacting to throttling and operating Fabric well. Sources This article is grounded in current Microsoft documentation. Product behavior and preview features can change, so verify the latest guidance before making production capacity decisions. Understand capacity throttling and smoothing (Microsoft Learn)Capacity Metrics app compute page (Microsoft Learn)Evaluate and optimize your Microsoft Fabric capacity (Microsoft Learn)Performance guidelines for Fabric Data Warehouse (Microsoft Learn)Query Insights in Fabric Data Warehouse (Microsoft Learn)On-demand billing for Apache Spark in Microsoft Fabric (Microsoft Learn)Optimization guide for Power BI (Microsoft Learn)Manage surge protection for Fabric capacities (Microsoft Learn)Data clustering in Fabric Data Warehouse (preview) (Microsoft Learn)Partitioning for Delta tables (Microsoft Learn)

By Amrutha Satishkumar

Why Kotlin or Java Is the Wrong Question For Enterprises

If you've sat through more than one architecture review, you've probably sat through a protracted Kotlin-versus-Java debate that goes nowhere. The team spends weeks comparing Kotlin's concise syntax to Java's maturity, runs a poll on developer preference, and produces a decision matrix nobody fully trusts. Then the meeting ends, and nothing changes, because nobody agreed on what problem they were actually trying to solve. Choosing between infrastructure instruments, such as Kotlin and Java, is an investment decision, and like any investment decision, it needs a thesis. Four business pressures should drive the discussion: time-to-market, operational risk, talent economics, and cross-platform consistency. The Problem With Asking Kotlin or Java When the question of Kotlin or Java arises, syntax and feature comparisons dominate the discussion, because they're concrete and measurable, and a natural place to start. You can count lines of code, benchmark compile times, and run surveys on developer satisfaction, but none of that will tell you if those metrics connect to any business outcome you actually care about. A language can be a good technical fit and still be a poor investment for a particular system. A stable backend handling 50,000 transactions a day with a three-person maintenance team faces a different set of risks than a payments service changing weekly under regulatory pressure, and both look different again from a monorepo running the same business logic across Android, backend, and edge. The JVM language strategy question comes down to where complexity actually lives in the system. Ask where Kotlin meaningfully supports your modernization strategy, and where staying with Java makes better sense right now. Modern Java May Be the Right Answer For a stable, low-change system, modernizing the existing Java stack can address the underlying problem without introducing the costs of another language, toolchain, or learning curve. If the system's main constraints don't align with Kotlin's strengths, staying with Java may offer a better return. If you're considering Kotlin, first identify where its capabilities address a problem worth solving. Start With the Business Problem Drawing on Yuri Geronimus's experience as Senior Staff Engineer and Chief Architect for EMEA Enterprise Payments Platforms at Verifone, the Kotlin Enterprise Playbook frames Kotlin adoption as an enterprise investment decision rather than a language preference. It organizes the business case around four key levers: delivery throughput and time-to-market, defect-cost avoidance, legacy modernization, and talent economics. Time-to-Market If your delivery lead time affects revenue, product launches, or certification cycles, ask whether the team can deliver faster. Kotlin's concise syntax means less boilerplate per feature, and fewer null pointer exceptions reaching production translates to less time spent on hotfixes and incident response. Whether that translates to faster delivery depends on where your bottlenecks sit. If the constraint is product clarity or test coverage rather than boilerplate or type errors, language choice won't fix it. Operational Risk Where are your expensive defects concentrated? Kotlin's type system encodes nullability explicitly, allowing some null-related errors to be caught at compile time rather than in production. For systems where a single production bug triggers a compliance review or a customer-facing outage, that compile-time safety has measurable financial value. For a read-heavy reporting service where reliability is already strong, the business case is thinner. Talent Economics The cost of a JVM talent problem can show up in longer time-to-hire, higher onboarding overhead, attrition, or difficulty moving engineers between teams. Java developers can build on their existing JVM experience when learning Kotlin, making internal mobility a potential part of the business case. The strategic partnership between JetBrains and Spring, announced in May 2025, also means that 27% of Spring developers who've already used Kotlin are a potential internal resource pool you may not have fully mapped. Cross-Platform Consistency If your organization maintains duplicated business logic across an Android app, a backend service, and an edge deployment, that duplication is an operational risk disguised as a technical decision. Kotlin Multiplatform (KMP) can let teams share domain logic, models, and validation across platforms, potentially reducing duplicated implementation and the risk of logic drift. That also reduces the compliance and maintenance burden of finding and fixing the same bug in multiple places. Keep in mind that you should evaluate the case for KMP separately, with the scope of shared code kept deliberately narrow. Kotlin doesn't automatically improve time-to-market any more than a new CI tool does. The question is whether you've identified the constraint it's positioned to address. Measure the Outcome You Actually Care About To make a solid business case for Kotlin, you need a measurable hypothesis. Each business lever maps to an observable metric. Time-to-market connects to delivery lead time and deployment frequency (both DORA metrics your platform team likely already tracks). Operational risk connects to change failure rate, mean time to recovery, and incident cost. Talent economics connects to time-to-hire, time to first commit after onboarding, and attrition rate. Cross-platform consistency connects to duplicated-logic defect rate and the volume of bugs that required fixes in more than one codebase. Those metrics support the business case, but you still have to translate them into financial terms. For an enterprise investment decision, connect the operational improvement to a financial outcome. Faster delivery can reduce the cost of delay, fewer incidents can reduce remediation and customer-impact costs, and shorter onboarding can reduce hiring and training costs. Bear in mind that productivity gains don't translate directly into headcount savings. Faster delivery can instead create capacity for new product work or reduce the cost of delay on a revenue-generating feature. Conflating the two overstates the financial case and creates a credibility problem when someone on the finance team models it out. Also, don't count productivity gains as both lower operating costs and increased revenue in the same proposal. A hypothesis that says "we expect Kotlin's null safety to reduce null-related production incidents by X% over 12 months, based on our current incident log showing Y% of incidents trace to NPEs" is defensible. A hypothesis that says "Kotlin will improve developer productivity" gives you nothing to measure against. Interoperability Makes This a Scoped Decision The Kotlin-or-Java framing misleads because it implies a binary. Kotlin compiles to JVM bytecode and is fully interoperable with Java, so Kotlin and Java files can coexist in the same project without a rewrite. Your existing Spring Boot or Hibernate configurations stay in place. Your Java libraries continue working. A single new service, a shared utilities module, or even the test layer of an existing Java codebase are all valid starting points for introducing Kotlin. IntelliJ IDEA, JetBrains' own IDE, includes a Java-to-Kotlin converter and first-class mixed-project support, so the tooling overhead of running both is lower than teams typically assume. That shifts the question from "should we adopt Kotlin?" to "where should we introduce Kotlin first?" A new, high-change service with a measurable business outcome may be a reasonable candidate. A stable order-management system with a small team and clean Java code can probably wait on Kotlin until a clearer business case emerges. Account for the Iceberg and the J-Curve An honest transition plan accounts for costs that typically don't appear in the optimistic spreadsheet. The J-curve is real. Before a team reaches full Kotlin proficiency, velocity drops. Training takes time, mixed-language code reviews add overhead (reviewers need to understand both languages to catch idiomatic issues), and CI pipelines may need reconfiguration for multi-language builds. Compliance documentation may need updating if your audit trail references language versions or toolchain components. The eventual decommissioning of legacy Java code, if that's the direction, carries its own cost that's easy to leave out of a five-year model. Scope the initial experiment carefully, and build these costs into the transition plan upfront. A proposal that acknowledges the J-curve and plans for it presents a more credible case than one that doesn't. Three Questions for Your Next Architecture Review If your team is heading into an architecture review where Kotlin is on the agenda, these three questions cut through the syntax debate and turn it into a scoped, testable proposal. Where are the expensive defects or operational problems concentrated? Pull the last 12 months of production incidents and categorize the ones with the highest cost (in engineer time, customer impact, or compliance exposure). If a significant share traces to null-related errors or type-safety failures, Kotlin's type system is directly relevant. If the distribution looks different, the investment case needs a different justification. What does your JVM talent pipeline actually look like? Map your current team's Kotlin exposure, your open Java roles, and the average time-to-hire over the past year. If you're competing for senior Java engineers in a tight market and losing candidates to organizations with more modern stacks, that's a talent economics argument worth quantifying. If your team is stable and hiring is closed, the argument doesn't apply. Which service, module, or domain is the lowest-risk place to test whether Kotlin improves the target outcome? Look for something actively changing (so you'll have delivery-cycle data to measure), with a well-defined scope, and without deep compliance dependencies that would complicate a mid-stream language shift. New functionality is often the best candidate. Adding Kotlin tests to an existing Java service can also provide a relatively low-production-risk way to gain experience with the language and toolchain. These questions convert an abstract language debate into a scoped experiment with a falsifiable hypothesis. Run the experiment, measure the result, and expand where it works. Make the Strategic Decision Your CFO and CTO won't ever see a line item for Java versus Kotlin, but they will see the costs of delivery lead time, incidents, onboarding overhead, and duplicated-logic defects. The strategic decision is where to invest to improve those outcomes, domain by domain, with measurement in place. Evaluate Kotlin where you have a number to test against, start with the lowest-risk entry point, and leave stable systems alone when the case is weak. The goal is a better outcome where the investment is justified. If you want to see how other enterprise teams have structured this transition, JetBrains has a practical case for it at kotlinlang.org/lp/enterprise-playbook. How has your team navigated this decision? Share your experience in the comments.

By Aniket Bhattacharyea

How to Build Your Own AI SRE From Scratch

If you are responsible for keeping a production system alive, you have probably been pitched an "AI SRE" in the last few months. Maybe more than one. Maybe more than one this week. When we sat down at Bronto to map that space, we ended up with 64 tools across seven camps: pure-play investigation agents, observability platforms with AI built in, incident management tools with AI on top, AIOps and alert correlation, hyperscaler agents like AWS DevOps Agent and Azure SRE Agent, Kubernetes copilots and open source projects, and a long tail of adjacent tools. All of them use the same two words on their landing page, and they solve very different problems. So the first question most teams ask is the classic one: do we buy, or do we build? In our landscape article, we argue that the choice isn't binary. You can go full DIY, build on open source foundations like HolmesGPT or K8sGPT, buy a product and extend it, or buy it as-is. Most teams land somewhere in the middle. In this article, I want to make a different argument for building one, at least once: build to understand what you buy. Even if you never run your own DIY agent in production, putting one together by hand shows you where its answers come from, what it can't see, and why it goes wrong when it does. After building one, you know which questions to ask a vendor, and you know how to get better results from whatever you end up running. That's the profit you should take away from this post. The fun is the building and seeing your agent try to investigate an incident: it might get it wrong, it might get it right, but you'll see how it interacts with your tools and your data. This post is based on a hands-on lab we ran at an AWS and Bronto evening, and everything is available through bronto-community/ai-sre-lab. By the end, you will have a small agent with the following capabilities: reads logs, traces, and metrics through an MCP server,reads the application's source code and commit history through GitHub's MCP server,writes its hypothesis, evidence, and confidence into a GitHub issue. Note that in the lab we use Bronto as the observability backend. I work there, so I am biased to think it is your best option, but you can of course use any other solution. What an AI SRE Is AI SRE is mostly a marketing term. In short, it's an agentic loop pointed at the production system that can answer questions about it and help you troubleshoot. Like any agentic loop, it has inputs on one side (an alert, a question from a human), a model in the middle, tools underneath (mostly MCP servers these days, for telemetry, code, cloud APIs, runbooks), and outputs on the other side (a Slack message, a ticket, a pull request). So, it's an agent like any other. It's what you connect to it that makes it an agent for SRE. The Patient: A Small Online Shop Our AI SRE needs something to investigate. The lab comes with a tiny storefront: four Python services instrumented with OpenTelemetry, plus a payment processor mock standing in for a third party. You can, of course, point the agent directly at your own data. It's read-only, so it is not able to harm your system. What You Need To run this lab, you need the following: A container engine, like Docker DesktopAn AWS account with access to a model on Amazon Bedrock.A Bronto API key (there is a free trial), or another observability backend with an MCP server.A GitHub account and a fine-grained personal access token. It needs to read the contents of your copy of the storefront repo and write issues to it.Optional, for nicer output: jq and glow. You can watch this video to follow along as we walk through each step: Start the Shop Clone the lab and generate the storefront repository. The script builds a small Git history with a handful of commits and tags two releases, v3.1.0 (healthy) and v3.2.0: Shell git clone https://github.com/bronto-community/ai-sre-lab cd ai-sre-lab ./build-storefront.sh Your agent will later read this code through GitHub, so push it to a repository of your own, tags included: Shell cd storefront gh repo create storefront --public --source . --push git push --tags cd .. Then point the Collector at your telemetry backend and bring everything up: Shell cd demo echo "BRONTO_API_KEY=<your key>" > .env make up Note, that demo/collector.yaml ships to Bronto's EU ingestion endpoint. If your account lives in another region, or you want a different backend, that file is the only thing you need to change. Next, run make up, which deploys v3.1.0 and starts the traffic. It prints a message that you should leave it about ten minutes before you do anything else. An investigator with no baseline has nothing to compare against. So, take your time going through the first few next steps so it can build up! Step 1: The Loop All agents start with a basic file that receives prompts, sends them to the model, and returns the results. It's a simple first agent, so here is more or less the code in full: Python import os from bedrock_agentcore import BedrockAgentCoreApp from strands import Agent from strands.models import BedrockModel model = BedrockModel( model_id=os.environ.get("BEDROCK_MODEL_ID", "us.anthropic.claude-haiku-4-5-20251001-v1:0"), region_name=os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION", "us-west-2"), ) app = BedrockAgentCoreApp() @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model) return {"result": str(agent(payload.get("prompt", "Hello!")))} if __name__ == "__main__": app.run() # POST /invocations, GET /ping on :8080 Two libraries do the work. Strands Agents runs the agent loop: send the prompt to the model, run any tool the model asks for, feed the result back, repeat until the model has an answer. The Bedrock AgentCore SDK wraps that in an HTTP server with the /invocations and /ping contract AgentCore Runtime expects. Today that runs on your laptop. Later, the same container can run on AgentCore Runtime without changes. Every step of the lab is a prebuilt image, so you don't need to build anything. Create a .env next to where you run the commands: Shell AWS_DEFAULT_REGION=us-west-2 AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... BEDROCK_MODEL_ID=us.anthropic.claude-haiku-4-5-20251001-v1:0 With this setup, you can start the agent for the first time: Shell docker run --rm -p 8080:8080 --env-file .env ghcr.io/bronto-community/ai-sre-agent:step-1 Open a second terminal and ask it something: Shell curl -s localhost:8080/invocations -d '{"prompt": "Who are you?"}' | jq -r .result It will say, "Claude, an AI assistant made by Anthropic." Correct, and completely useless for incident response. Right now this is a model behind an endpoint, with no identity and no tools. Step 2: Identity The second step changes only one line of code. Every Markdown file in prompts/ becomes part of the system prompt: Python SYSTEM_PROMPT = "\n\n".join(p.read_text() for p in sorted(Path("prompts").glob("*.md"))) @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model, system_prompt=SYSTEM_PROMPT) ... The image ships with two files. role.md: Markdown You are the on-call SRE for Storefront, an online shop. You investigate production incidents: what broke, where, and why. And rules.md: Markdown - Only state what you can back with evidence from your tools. - If you have no tool that can answer, say so and name what you would need. - Be brief. You also get to add your own. Create a my.md: Markdown - Answer in at most three bullet points. - End every answer with your confidence: low, medium or high. - Sign off as "your friendly on-call dino". Start step 2 as follows. The command contains your custom instructions as an additional volume: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-2 Shell curl -s localhost:8080/invocations -d '{"prompt": "Checkout is slow. What is wrong?"}' | jq -r .result You should get a short answer that says it has no access to telemetry, rates its own confidence as low, and signs off as a dino. That's a good result. The rule "say so and name what you would need" did its job: an agent that admits it can't see anything is far more useful than one that guesses. Step 3: Eyes Now the agent gets something to look at. Step 3 adds one tool source, Bronto's MCP server: Python bronto = MCPClient(lambda: streamable_http_client( os.environ.get("BRONTO_MCP_URL", "https://mcp.eu.bronto.io/mcp"), http_client=httpx2.AsyncClient( headers={"X-BRONTO-API-KEY": os.environ["BRONTO_API_KEY"]}, timeout=60, ), )) @app.entrypoint def invoke(payload: dict) -> dict: agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto]) ... That's the whole change. The MCP server's tools (search logs, list datasets, and so on) become tools the agent can call, and Strands handles the plumbing. This is also the point where MCP pays off: if your telemetry lives somewhere else, you swap this client for that vendor's MCP server and keep everything else. Add BRONTO_API_KEY to your .env (and BRONTO_MCP_URL if your account isn't in the EU region), then: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-3 Start with a question that tells you what it can see: Shell curl -s localhost:8080/invocations -d '{"prompt": "What services do you know about?"}' | jq -r .result It should list the storefront services. Go ahead and ask it some more questions about the state of those services: what HTTP endpoints are available? What is the average error rate right now? With this, you have a first working version. Next we want to investigate an incident. To do so, ship the next release of the storefront: Shell make release According to the Makefile, symptoms take about 60 seconds to show up in p95. So wait a minute or two and then ask it: Shell curl -s localhost:8080/invocations -d '{"prompt": "Anything wrong with checkout?"}' | jq -r .result It will come back with a hypothesis about what is wrong and what you might be able to do about it. Take a look at the telemetry yourself and check: is the analysis correct, or did the agent pick up some false information? Step 4: Mouth An answer in your terminal is nice for a demo, but in production you would want to have it sent to a Slack channel, a pager app, or an issue tracker. In our example, we send the answer as a GitHub issue. This would let you tag your coding agent to review the issue and open a pull request that fixes it. You now have two options for how this gets filed. Either you give your agent another tool, and it can decide for itself where to send the hypothesis, or you make the code file the issue, so every analysis gets pushed into a dedicated issue: Shell def file_issue(question: str, hypothesis: str) -> str: r = httpx2.post( f"https://api.github.com/repos/{os.environ.get('GITHUB_REPO', 'ai-sre-lab/storefront')}/issues", headers={"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}"}, json={"title": f"[investigation] {question}"[:120], "body": hypothesis}, timeout=30, ) r.raise_for_status() return r.json()["html_url"] @app.entrypoint def invoke(payload: dict) -> dict: question = payload.get("prompt", "Hello!") agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto]) hypothesis = agent(question) return {"result": file_issue(question, hypothesis)} To make sure that what the agent provides fits the output, an additional new prompt file tells the agent how to write it: Markdown Your answer is filed as a GitHub issue for the on-call team. Write it as one: a one-line hypothesis, the evidence for it, your confidence, and what you could not check. Before running the next version of your agent, add your GitHub token and your repository to .env: Shell GITHUB_TOKEN=github_pat_... GITHUB_REPO=<your-user>/storefront Run step 4: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" ghcr.io/bronto-community/ai-sre-agent:step-4 Ask it again to investigate the issue: Shell curl -s localhost:8080/invocations -d '{"prompt": "Anything wrong with checkout?"}' | jq -r .result The response is now a link. Open it and read the hypothesis. Is it right? How confident is it? There's a good chance it's confidently wrong. While testing the lab, one run concluded that checkout was down because a query returned nothing: an empty result is not proof of an outage, but nothing told the agent that. Step 5: Better Instructions What you do next doesn't involve code. You do what you'd do with a new human on the rotation: you tell them how you investigate. Add these lines to your my.md: Markdown - Slow is broken too: compare latency (p50, p95) before and after, not only errors. - Compare against a quiet window earlier the same day. - Look for `deploy` spans near the onset: they name the commit that went live. Each line encodes a habit experienced SREs have, and models don't bring by default. Checkout can be very broken without throwing a single error. A number means little without a baseline. And the first question in most incidents is "what changed?" The prompt files are read when the container starts, so restart step 4 with the same command and ask a more realistic question: Shell curl -s localhost:8080/invocations \ -d '{"prompt": "Customers say checkout got slow in the last 30 minutes. What is going on?"}' | jq -r .result Compare the new issue with the old one. Is it better? Worse? Did it fail outright? Step 6: Better Model As you might now, a different model can create a different output, so if you are not satisfied with the results yet, change one line in .env: Shell BEDROCK_MODEL_ID=us.amazon.nova-lite-v1:0 Restart your agent, ask the question again, and compare the outputs again: better or worse? Step 7: Code So far the agent can see that something changed and maybe even when. If your instructions worked, it found the deploy. What it can't do is look at what was deployed. Step 7 adds the second MCP server, GitHub's, restricted to read-only repository tools: Python github = MCPClient(lambda: streamable_http_client( os.environ.get("GITHUB_MCP_URL", "https://api.githubcopilot.com/mcp/x/repos/readonly"), http_client=httpx2.AsyncClient( headers={"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}"}, timeout=60, ), )) @app.entrypoint def invoke(payload: dict) -> dict: question = payload.get("prompt", "Hello!") agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=[bronto, github]) ... To make it useful to the agent, add one more prompt file, code.md. It connects the telemetry to the code: Markdown The shop's source code is in the GitHub repository `ai-sre-lab/storefront`. Every service reports the commit it runs as `deployment.commit`. Replace the repository path with your own, then mount it as an additional volume: Shell docker run --rm -p 8080:8080 --env-file .env \ -v "$PWD/my.md:/app/prompts/my.md" \ -v "$PWD/code.md:/app/prompts/code.md" \ ghcr.io/bronto-community/ai-sre-agent:step-7 Ask the same question again: Shell curl -s localhost:8080/invocations \ -d '{"prompt": "Customers say checkout got slow in the last 30 minutes. What is going on?"}' | jq -r .result Does it find the deploy? Does it open the release and pick the right commit? In our testing, it found the deploy marker and the commit, and then stopped. The release contains several commits, and the last hop, reading every one of them and figuring out which change explains the symptom, didn't happen on its own. If you experience the same, try to fix it yourself by refining your system prompts or by trying yet another model. When you're done, make rollback puts the healthy version back, and make down cleans everything up. If you want to give your agent a harder case, make psp-slow degrades the third-party payment processor. So, Buy or Build? Now you know what an AI SRE is and how it works, from the inside. That's a much better position from which to answer the question this post started with. If you buy, a lot of what hurt along the way becomes someone else's problem: confidently wrong answers that need endless prompt tuning, context windows overflowing with search results, wiring up every integration yourself. AWS DevOps Agent, for example, starts investigating when an alert fires and connects to the tools you likely already run, and you can still extend it with custom skills and your own tools. Take the four questions from the landscape article with you into every demo: Where does it get its data? Does it act, or only advise? Can you see its reasoning? What happens when it's wrong? After this lab, you'll know what good and bad answers to those look like. If you build, what you have now is a skeleton, and you can build more features towards a complete picture on top of it: Let alerts trigger it instead of curl.Run it on AgentCore Runtime instead of your laptop (agentcore deploy).Give it more tools: your cluster, your cloud APIs, your past incidents and postmortems.Observe the agent itself. Its own traces tell you which tools it called, how many tokens it burned, and where it went wrong, and that's the input you need to improve it. Either way, the data underneath matters more than the agent on top. Every improvement in this lab depended on the telemetry being there: a baseline from ten quiet minutes, latency and not just errors, a deployment.commit attribute that connects a span to a line of code. An agent can only reason about what your system tells it. That was true for humans long before agents showed up. So: what would your agent get wrong? The full lab, including the slides, the agent images for every step, and the storefront harness, is at github.com/bronto-community/ai-sre-lab. Content is CC BY 4.0; code is MIT. If you build something on top of it, I'd love to hear about it. I've shown how easy it is to build a simple AI SRE agent for fun. If you want to see what a production-ready one looks like, read our post on the Bronto blog, Introducing Bronto's Open-Source Alert Investigation Agent. It's an open-source agent that replicates the Alert Investigation feature in Bronto, and it's essentially our production version of this lab: a Bronto monitor triggers it instead of curl, it sticks to a constrained set of tools and reasoning paths instead of improvising, and it hands you a report with likely root causes and suggested next steps. The code is at bronto-community/bronto-alert-investigation-agent.

By Severin Neumann

How to Migrate From LangChainGo to Genkit Go

If you built a Go AI service in the last two years, there is a good chance it runs on LangChainGo. It was the obvious choice: the name was familiar, the concepts came straight from Python, and it had the biggest provider catalog in the Go ecosystem. That choice has aged badly. LangChainGo's last release was v0.1.14 in October 2025, its last commit to main was January 2026, and it is still on 0.1.x after three and a half years. Meanwhile, the ecosystem moved: models were retired, SDKs were replaced, and APIs changed shape. LangChainGo did not move with them, and parts of it are broken today as a result. Genkit Go is where that workload belongs now. It is GA, it is on v1.13.1, and it is actively maintained by Google. This guide is the complete migration path: every LangChainGo concept mapped to its Genkit equivalent, with before and after code for each one. Follow it top to bottom, and you can move a real service in an afternoon. If you want the case for why rather than the how, read the companion article: LangChainGo vs Genkit Go: where Genkit shines. Why Migrate Now Three things in LangChainGo are broken today, and none of them will be fixed, because nobody is upstream to fix them. 1. The default model no longer exists. DefaultOptions() in llms/googleai/option.go hardcodes gemini-2.0-flash, which Google retired. A fresh googleai.New(ctx) with no options fails on the first call: Plain Text googleapi: Error 404: This model models/gemini-2.0-flash is no longer available. Please update your code to use models/gemini-3.6-flash for the latest features and improvements. 2. The Google provider runs on an end-of-life SDK. llms/googleai imports github.com/google/generative-ai-go, which Google replaced with google.golang.org/genai. Its README is explicit: End-of-Life Date: All support for this repository (including bug fixes) will permanently end on November 30, 2025. 3. Streaming is broken on every current Gemini model. Because of that dead SDK, any streaming call through the Google provider fails: Plain Text error in stream mode: invalid character ']' looking for beginning of value This fails on gemini-2.5-flash, gemini-3-flash-preview, gemini-3.5-flash and gemini-3.6-flash. Non-streaming calls still work, and the OpenAI provider streams fine, so this is specific to Google on the legacy SDK. The Trap: Observability Breaks Your Agents This one catches people out, so it is worth knowing before you debug it the hard way. LangChainGo turns on streaming automatically whenever you attach a callbacks handler. From chains/options.go: Go if opts.StreamingFunc == nil && opts.CallbackHandler != nil { opts.StreamingFunc = func(ctx context.Context, chunk []byte) error { opts.CallbackHandler.HandleStreamingFunc(ctx, chunk) return nil } } agents/mrkl.go does the same thing. Since streaming is broken, attaching the only supported observability hook is also what breaks your agent. Same agent, same tool, one line different: Plain Text WITHOUT callbacks handler: err=<nil> out="The current year is 2026." WITH callbacks handler: err=error in stream mode: invalid character ']' looking for beginning of value On LangChainGo with Gemini today, you can have working agents, or you can have instrumentation. Not both. What You Need Go 1.24+ (this guide was written against Go 1.27)A Google AI API key, free from Google AI StudioOptionally, an OpenAI API key, if you want cross-vendor fallbackThe Genkit CLI, for the Developer UI: Shell npm install -g genkit Add Genkit to your module: Shell go get github.com/firebase/genkit/go@latest The Concept Map This is the whole translation table. Keep it open while you work. LangChainGoGenkit GoWhat changesllms.Model + provider New()plugin + genkit.InitModels become string references like googleai/gemini-3.5-flashllms.GenerateFromSinglePromptgenkit.GenerateTextDirect swapprompts.PromptTemplateai.WithPrompt or a .prompt fileDotprompt files keep prompts outside the binarychains.LLMChain + chains.Rungenkit.DefineFlowA typed Go function instead of a map[string]anychains.NewSequentialChainordinary Go codeJust call the next functionoutputparser.NewDefined[T]genkit.GenerateData[T]Real schema instead of prompt-and-parsetools.Tool interfacegenkit.DefineTool[In, Out]Typed arguments with a JSON schemaagents.NewOneShotAgent (ReAct)ai.WithTools, or genkitx.DefineAgentNative function calling instead of text parsing. Genkit also has a real agent primitive with sessionsmemory.NewConversationBufferresp.History(), or an agent session storeExplicit history, or aix.WithSessionStore if you want the buffer backembeddings + vectorstoreslocalvec or a vector store pluginGenkit has a local store, LangChainGo does notcallbacks.Handlerbuilt-in tracingDelete the handler entirelynothingmiddleware.Retry, middleware.FallbackRetry and fallback stop being your code Step 1: Set Up Genkit In LangChainGo, each provider is a concrete struct you construct and hold: Go llm, err := googleai.New(ctx, googleai.WithDefaultModel("gemini-3.5-flash"), googleai.WithDefaultEmbeddingModel("gemini-embedding-001")) if err != nil { log.Fatal(err) } defer llm.Close() In Genkit, plugins register with one runtime object and models become string references: Go g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}), genkit.WithDefaultModel("googleai/gemini-3.5-flash")) That difference matters more than it looks. Registering a second vendor is one more plugin: Go g := genkit.Init(ctx, genkit.WithPlugins( &googlegenai.GoogleAI{}, &openai.OpenAI{}, )) Now both vendors live in the same registry, and a model is just a name, which is what makes swapping and falling back a configuration change later on. Step 2: Migrate Your Model Calls A plain text generation is close to a direct swap. Before: Go out, err := llms.GenerateFromSinglePrompt(ctx, llm, "In one sentence: what is Go's error handling philosophy?") After: Go out, err := genkit.GenerateText(ctx, g, ai.WithPrompt("In one sentence: what is Go's error handling philosophy?")) If you need the full response rather than just the text, use genkit.Generate, which gives you resp.Text(), resp.History() and usage data. Step 3: Turn Chains Into Flows A LangChainGo chain wraps a prompt template and takes a map[string]any: Before: Go prompt := prompts.NewPromptTemplate( "Triage this support ticket.\n{{.format}\n\nTicket: {{.ticket}", []string{"format", "ticket"}, ) chain := chains.NewLLMChain(llm, prompt) raw, err := chains.Predict(ctx, chain, map[string]any{ "format": parser.GetFormatInstructions(), "ticket": ticket, }) A Genkit flow is a typed Go function. Your input and output types are the contract, and the compiler enforces them: After: Go triage := genkit.DefineFlow(g, "triage", func(ctx context.Context, ticket string) (*Triage, error) { out, _, err := genkit.GenerateData[Triage](ctx, g, ai.WithPrompt("Triage this support ticket: %s", ticket)) return out, err }) out, err := triage.Run(ctx, "I was charged twice for order A-1029.") This step deletes the most code, because most of what a chain does is pass a map around. Go does not need that. Sequential chains have no equivalent, and do not need one. chains.NewSequentialChain becomes calling the next function: Go summary, err := summarize.Run(ctx, doc) if err != nil { return nil, err } return classify.Run(ctx, summary) Step 4: Migrate Structured Output Prioritize this step. In LangChainGo, structured output is prompt engineering. In Genkit, it is a schema. Before: Generate format instructions from your struct, paste them into the prompt, parse the text back. Go type Ticket struct { Category string `json:"category" describe:"one of: billing, bug, feature, other"` Severity int `json:"severity" describe:"1 (low) to 5 (critical)"` Summary string `json:"summary" describe:"one sentence"` } parser, err := outputparser.NewDefined(Ticket{}) // ...put parser.GetFormatInstructions() into the prompt... out, err := parser.Parse(raw) After: One call, with the schema sent to the model as a real response schema. Go type Ticket struct { Category string `json:"category" jsonschema:"enum=billing,enum=bug,enum=feature,enum=other"` Severity int `json:"severity" jsonschema_description:"1 (low) to 5 (critical)"` Summary string `json:"summary" jsonschema_description:"one sentence"` } out, _, err := genkit.GenerateData[Ticket](ctx, g, ai.WithPrompt("Triage this support ticket: %s", ticket)) Note the tag change: describe: becomes jsonschema_description:, and enums move into a jsonschema: tag where the model actually enforces them. Step 5: Migrate Your Tools The LangChainGo tool interface is Call(ctx context.Context, input string) (string, error). One string in, one string out, no schema. A tool that needs typed arguments has to describe its schema in prose and unmarshal by hand: Before: Go func (RefundTool) Name() string { return "issue_refund" } func (RefundTool) Description() string { return `Issues a refund. Input MUST be a JSON object string with exactly these keys: {"order_id": string, "amount_cents": integer}. Example: {"order_id":"A-1","amount_cents":2500}` } func (RefundTool) Call(ctx context.Context, input string) (string, error) { input = strings.TrimSpace(input) input = strings.TrimPrefix(input, "```json") input = strings.TrimPrefix(input, "```") input = strings.TrimSuffix(input, "```") var a RefundArgs if err := json.Unmarshal([]byte(strings.TrimSpace(input)), &a); err != nil { return fmt.Sprintf("error: input was not valid JSON (%v). Got: %s", err, input), nil } return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil } In Genkit, the tool function is generic over its argument type, and the schema comes from the struct: After: Go type RefundArgs struct { OrderID string `json:"order_id"` AmountCents int `json:"amount_cents"` } refund := genkit.DefineTool(g, "issue_refund", "Issues a refund for an order.", func(ctx *ai.ToolContext, a RefundArgs) (string, error) { return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil }) Delete the fence stripping, the unmarshalling, and the prose schema. The model receives a real JSON schema, and the arguments arrive typed. Step 6: Replace the Agent A LangChainGo ReAct agent drives tools by parsing the model's text output for Action: and Action Input: lines, wrapped in an executor with an iteration cap: Before: Go agent := agents.NewOneShotAgent(llm, []tools.Tool{RefundTool{}, agents.WithMaxIterations(6)) out, err := chains.Run(ctx, agents.NewExecutor(agent), "Refund order A-1029 for 25 dollars and 50 cents.") In Genkit, native function calling means a one-shot agent is just a generate call with tools attached: After: Go out, err := genkit.GenerateText(ctx, g, ai.WithPrompt("Refund order A-1029 for 25 dollars and 50 cents."), ai.WithTools(refund)) No executor, no iteration cap, no text parsing. Genkit runs the tool loop for you. When You Want a Real Agent, Genkit Has One The generate call above replaces a OneShotAgent. For anything that has to hold a conversation, Genkit Go ships an actual agent primitive with sessions and persisted state, which LangChainGo has no equivalent for: Go import ( aix "github.com/firebase/genkit/go/ai/exp" "github.com/firebase/genkit/go/ai/exp/localstore" genkitx "github.com/firebase/genkit/go/genkit/exp" ) g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}), genkit.WithDefaultModel("googleai/gemini-2.5-flash"), genkit.WithExperimental()) store := localstore.NewInMemorySessionStore[State]() agent := genkitx.DefineAgent(g, "taskAgent", aix.InlinePrompt{ ai.WithSystem("Manage a task list. Use tools when changing tasks."), ai.WithTools(addTask), }, aix.WithSessionStore(store), aix.WithDescription[State]("Task management assistant"), ) Three constructors are available: DefineAgent for an inline prompt, DefinePromptAgent to wrap a prompt already in the registry, and DefineCustomAgent to replace the prompt loop with your own code. Options cover a session store, WithStateTransform to reshape state on the way out to a client, and WithStreamTransform for chunks. You drive one with Run, RunText, or Connect for a streaming connection. These live in the exp packages. Agents are experimental in Genkit Go v1.13.1, and calling genkitx.DefineAgent without genkit.WithExperimental() on genkit.Init panics with a message telling you so. The API may change between minor releases, so weigh that before putting one in production. Step 7: Migrate RAG Check this step before planning your timeline, because it has an infrastructure consequence. LangChainGo ships fifteen vector stores: AlloyDB, Azure AI Search, Bedrock Knowledge Bases, Chroma, CloudSQL, Dolt, MariaDB, Milvus, MongoDB, OpenSearch, pgvector, Pinecone, Qdrant, Redis and Weaviate. Every one of them needs an external service running. There is no in-memory or local option, and the pull request adding one has been open since February 2024. If you needed a local store, you wrote it yourself. Genkit ships localvec, a file-backed local store, so the same thing is three calls: After: Go embedder := googlegenai.GoogleAIEmbedder(g, "gemini-embedding-001") store, retriever, err := localvec.DefineRetriever(g, "policy", localvec.Config{Dir: "/tmp/gkvec", Embedder: embedder}, nil) if err != nil { return err } // Index your documents once. localvec.Index(ctx, docs, store) // Retrieve at request time. found, err := genkit.Retrieve(ctx, g, ai.WithRetriever(retriever), ai.WithTextDocs(ticket)) Then hand the documents straight to the model: Go out, _, err := genkit.GenerateData[Triage](ctx, g, ai.WithDocs(found.Documents...), ai.WithPrompt("Triage this ticket using the policy documents: %s", ticket)) If you were on a hosted store, Genkit has Pinecone, Weaviate, Postgres, and AlloyDB plugins. Step 8: Migrate Memory LangChainGo keeps conversation history in a buffer attached to the chain: Before: Go c := chains.NewConversation(llm, memory.NewConversationBuffer()) chains.Run(ctx, c, "My name is Xavi and I work in Go.") chains.Run(ctx, c, "What is my name and what language do I use?") Genkit passes history explicitly: After: Go resp, err := genkit.Generate(ctx, g, ai.WithPrompt("My name is Xavi and I work in Go.")) if err != nil { return err } resp, err = genkit.Generate(ctx, g, ai.WithMessages(resp.History()...), ai.WithPrompt("What is my name and what language do I use?")) For a plain generate call, this is the one mapping where LangChainGo is less typing. The trade-off is that Genkit has no hidden buffer to keep in sync, which is what you want the moment your conversation state lives in Redis or Postgres and your handlers are stateless. If you want the buffer back, that is what an agent's session store is for. Give the agent a store and pass a session ID, and the history is the framework's problem again: Go store := localstore.NewInMemorySessionStore[State]() agent := genkitx.DefineAgent(g, "taskAgent", aix.InlinePrompt{ ai.WithSystem("Manage a task list. Use tools when changing tasks. Be terse."), ai.WithTools(addTask), }, aix.WithSessionStore(store)) for _, turn := range []string{ "Add 'buy milk' to my list.", "What did I just ask you to add?", } { out, err := agent.RunText(ctx, turn, aix.WithSessionID[State]("demo-session")) if err != nil { return err } fmt.Printf("> %s\n%s\n\n", turn, out.Message.Text()) } Plain Text > Add 'buy milk' to my list. Task added. > What did I just ask you to add? You asked me to add 'buy milk' to your list. Nothing in that loop passes the first turn into the second. The session ID does it. localstore.NewInMemorySessionStore is the one that ships for local development; the SessionStore interface is what you implement to put sessions in Redis or Postgres. So the honest version of this row is: Genkit gives you both ends. Explicit resp.History() when you want stateless handlers, and a store-backed session when you want the buffer. Step 9: Delete Your Resilience Code This is the step that usually surprises people with how much it removes. LangChainGo has no middleware layer. callbacks.Handler is seventeen methods that all return nothing: Go HandleText(ctx, text) HandleLLMGenerateContentStart(ctx, ms) HandleLLMGenerateContentEnd(ctx, res) HandleToolStart(ctx, input) // ...and thirteen more You can watch a call. You cannot wrap it, retry it, rewrite its request, or substitute a model, because there is no next(). There is no retry, no fallback, and no OpenTelemetry anywhere in the library, so if your service has those, you wrote them. In Genkit, they are configuration: After: Go out, _, err := genkit.GenerateData[Triage](ctx, g, ai.WithModel(googlegenai.GoogleAIModelRef("gemini-3.5-flash", nil)), ai.WithDocs(found.Documents...), ai.WithTools(refund), ai.WithPrompt("Triage this ticket and act on it: %s", ticket), ai.WithUse( &middleware.Retry{MaxRetries: 2}, &middleware.Fallback{Models: []ai.ModelRef{openai.ModelRef("gpt-5-mini", nil)}, ), ) Register the middleware plugin at init, and those two lines are live: Go g := genkit.Init(ctx, genkit.WithPlugins( &googlegenai.GoogleAI{}, &openai.OpenAI{}, &middleware.Middleware{}, )) Point the primary at a dead model, and the request heals onto the other vendor on its own: Plain Text WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND" {Category:billing Severity:2 PolicyQuote:Duplicate charges must be refunded in full within 5 business days, no manager approval required. Action:Issued full refund of $25.50 (2550 cents) for the duplicate charge on order A-1029.} Gemini went down, OpenAI picked it up, the tool still ran, and the typed struct came back intact. Delete your callbacks handler too. Genkit traces flows, generate calls, tool calls, and retrieval automatically. Run the Developer UI to see them: Shell genkit start -- go run . Step 10: Wrap the Entrypoints Finally, expose your flows over HTTP. Where LangChainGo left you writing net/http handlers around chains, Genkit turns a flow into an endpoint: Go mux := http.NewServeMux() mux.HandleFunc("POST /triage", genkit.Handler(triage)) log.Fatal(server.Start(ctx, "127.0.0.1:3400", mux)) The flow's input and output types define the request and response shapes, so the whole path is typed end to end. Before You Commit The migration is not free. Check these three things first: Provider and vector store catalog. LangChainGo has seventeen model providers and fifteen vector stores. Genkit Go covers Google AI, Vertex, OpenAI, Anthropic, Ollama, and the OpenAI-compatible vendors, plus Pinecone, Weaviate, Postgres, and AlloyDB. If you are on Dolt, MariaDB, or Bedrock Knowledge Bases as a vector store, confirm you have a path. Conversation buffers. There is no drop-in ConversationBuffer. You pass history explicitly, as in Step 8. Specialized chains. Helpers like chains.NewSQLDatabaseChain and the map-reduce summarisation chains have no one-to-one equivalent. In Genkit, they are Go functions you write, which is usually shorter than the chain was, but it is work. Conclusion The migration is smaller than it looks, because most of a LangChainGo program is plumbing that Go does not need: maps passed between chains, format instructions pasted into prompts, JSON unmarshalled out of tool strings, and a resilience layer you wrote yourself. Typed flows, typed tools, schema-backed output, and middleware delete all four. The reason to start now is not ergonomics. It is that the library your service depends on has a retired default model, an end-of-life SDK underneath its Google provider, broken streaming, and an observability hook that breaks the agents it is supposed to observe. Those are not problems you can wait out. For the side-by-side numbers and the full picture, read the companion: LangChainGo vs Genkit Go: where Genkit shines. Further reading: Genkit documentationGenkit Go pluginsLangChainGo repositoryGoogle Gen AI SDK for Go, the replacement for the SDK LangChainGo still usesStop using Python for GenAI, use Genkit GoMastering Genkit: Go Edition

By Xavier Portilla Edo

CORE

MQTT, Kafka, NATS, or Zenoh? Choosing Edge-to-Cloud Messaging by Traffic Pattern

Edge to cloud is not one integration problem. It is four: telemetry going up, control going down, sites syncing sideways, and data reaching people. Each has different physics. Most failed projects picked one technology for all four. Every enterprise with factories, stores, vehicles, hospitals, or devices faces the same architectural question. How does data move between the edge and the data center or cloud, in both directions, reliably, securely, and at acceptable cost? The vendor landscape answers with dozens of overlapping options: MQTT platforms, hyperscaler services, event streaming, event mesh, messaging fabrics, delivery platforms, and protocol proxies. This article structures the option space by traffic pattern instead of by product category, maps the vendor categories onto those patterns, and closes with a decision framework you can apply to your own architecture. The categories here sit alongside the wider integration landscape rather than inside it. For that broader map across event streaming, API, and batch, see the Data Integration Landscape 2026. What Does Edge-to-Cloud Communication Mean? Edge-to-cloud communication is the movement of data between distributed physical locations and centralized computing environments, in both directions, across networks the architect does not control. The word edge hides four very different tiers: The device edge means microcontrollers, sensors, and actuators with kilobytes of memory.The gateway edge means industrial PCs, vehicle computers, and protocol converters.The site edge means a factory, store, hospital, or ship with local compute and its own network boundary.The near edge means regional data centers close to many sites. A technology that fits one tier often fails at another. Edge to cloud splits into four traffic patterns: northbound telemetry, southbound control, site-to-site sync, and outbound delivery to people. The traffic crossing these tiers falls into four patterns: Northbound telemetry moves sensor readings, machine states, transactions, and logs from the edge toward central platforms for analytics, AI, and business systems.Southbound control moves commands, configuration, software updates, and increasingly AI models from central platforms down to machines and gateways.Site-to-site traffic synchronizes data between sites, between a site and a core data center, or between clouds.Outbound delivery pushes live data from central platforms to the applications people use: dashboards, mobile apps, and trading screens. Direction plus consumer type plus constraint determines the right technology. Vendor preference does not. Why Do Network Constraints Decide the Edge-to-Cloud Architecture? The network between edge and cloud is the one component you cannot refactor. Its properties dictate the architecture before any product evaluation starts. Connectivity is unreliable. Ships, vehicles, and remote sites go offline for hours or days. Any viable architecture is offline first: local operation continues, data buffers locally, and synchronization resumes when the link returns. This is why store and forward is the single most important capability on any edge evaluation checklist. Inbound connections are forbidden. Industrial sites, retail networks, and vehicle fleets sit behind NAT, firewalls, and cellular carriers. Security teams do not open inbound ports for messaging. Every serious edge technology therefore uses the same trick: the edge dials out. MQTT clients dial out to brokers. NATS leaf nodes dial out to hub clusters. Azure IoT Operations dials out from the factory to the cloud broker. Different stacks, identical answer, because the constraint is identical. Hardware is constrained and heterogeneous. An 8-bit microcontroller, a fanless gateway PC, and a site server need different footprints. Some technologies scale down to bare microcontrollers. Others assume a Kubernetes cluster. Matching footprint to tier is half the selection process. Footprint is not the only axis. In fixed installations like plants and warehouses, the device edge can stay thin, because the uplink is reliable and most intelligence lives upstream. In the field, agriculture, cranes, rural grid, remote or contested sites, the uplink is wireless and often degraded, saturated, expensive, or absent for days. There, the edge cannot just buffer and forward. It has to run enough local intelligence, pattern matching, image recognition, autonomous decisions, to operate correctly on its own and sync opportunistically when the link returns. That pushes compute onto the device tier for reasons that have nothing to do with protocol size, and it changes the pattern, not just the product. Data residency is now an architecture requirement, not a legal afterthought. European manufacturers increasingly must prove what data leaves the site, country, or jurisdiction. Topic design, routing rules, and retention policies become compliance controls. And one constraint is usually invented rather than measured: latency. Teams assert real-time requirements that the business case does not contain. A large share of edge traffic is perfectly fine as periodic batch over a metered link. Keep that thought. It returns in the site-to-site section. Northbound: How Does Telemetry Get from OT and Devices into the Core? Northbound telemetry is the pattern the industry knows best, and the one with the most settled consensus. At the machine level, OPC UA provides the semantic information model. It defines what a temperature, a batch, or a machine state means. MQTT provides the efficient transport, and Sparkplug adds topic structure, birth and death certificates, and state awareness on top of MQTT. The two standards stopped being rivals. OPC UA Pub/Sub over MQTT is the convergence point: OPC UA semantics carried over MQTT infrastructure. Edge gateways and industrial DataOps products, for example HighByte or Ignition with its MQTT modules, convert brownfield protocols like Modbus and S7 into that stack. The Unified Namespace (UNS) pattern then organizes all site data as a live, hierarchical topic structure on a central broker. I covered UNS design and its trade-offs in depth in Unified Namespace vs. Data Product in IT/OT for Industrial IoT and the broader MQTT market in MQTT Market Trends: Cloud, Unified Namespace, Sparkplug, Kafka Integration, so this section stays short. The real decision is the broker and its operating model. Self-managed platforms carry the northbound load for most industrial estates: HiveMQ and EMQX at the enterprise end, Mosquitto and NanoMQ where a lightweight or embedded footprint matters. They give you deployment control, protocol depth, and edge variants that run on gateways. Managed cloud services remove the operations burden and integrate natively with cloud analytics. The dividing line is usually sovereignty and site autonomy: if the broker must run inside the factory, on the ship, or in a specific jurisdiction, self-managed wins. If all sites have reliable connectivity and the data may leave, managed wins on cost of operations. The hyperscalers themselves tell an asymmetric story. Google retired Cloud IoT Core in 2023 and exited the category. AWS IoT Core remains the steady incumbent for managed device connectivity. Azure went furthest architecturally: Azure IoT Operations runs a distributed MQTT broker on Arc-enabled Kubernetes at the site edge and bridges it to the Event Grid MQTT broker in the cloud, which now supports Sparkplug natively. That is a symmetric edge plus cloud MQTT pair from a single vendor, and it is a new pattern. One hyperscaler retreated, one held, one doubled down. The lesson for architects: managed convenience is real, but so is the risk that a cloud provider deprecates your ingestion layer. The open standards underneath, MQTT and OPC UA, are the insurance policy. Choose products that speak them, and migration remains possible. Southbound and Outbound: How Do Data and Commands Flow Back to Machines and People? Almost all edge content covers ingestion. The return direction is harder, and it is where architectures break, because the return direction has two completely different consumers. Machines need commands and state. Request-response over pub/sub topics handles commands: reboot this gateway, close this valve, change this setpoint. Retained messages and device twins handle state: a device that reconnects must immediately learn its desired configuration without waiting for the next command. Fleet-wide rollout needs staged delivery, and the payloads keep growing. Configuration files became firmware images, and firmware images are becoming AI models. Shipping a quantized model to ten thousand gateways is a southbound data movement problem, not an MLOps footnote. The same fabric increasingly carries traffic for agents at the edge: AI agents that perceive local data, act locally, and coordinate through the messaging layer. NATS and the MQTT platforms are both positioning for exactly this workload. People need the current picture. A plant manager opening a dashboard, an operator reconnecting after a network drop, and a trader loading a screen mid-session all have the same requirement: current state on connect, then live updates. A log is not a user interface. Machines just consume the stream. People do not. This is the last-mile delivery problem, and it has its own vendor category: Lightstreamer, Ably, and PubNub sit between the central platform and many concurrent browsers and mobile apps, handling fan-out, per-client throttling, and entitlement checks. The category is maturing fast. For instance, Lightstreamer's Kafka Connector 2.0, released in July 2026, moved snapshot management into the connector itself, so every reconnecting client receives materialized state followed by the live stream. Delivery to humans deserves a dedicated article. Here it earns its place as the fourth traffic pattern that most edge architectures forget until the first user complains. Site-to-Site: How Do Sites, Data Centers, and Clouds Stay in Sync? The third direction is sideways: keeping data consistent between many sites and one core, or between multiple clouds. For durable event streams, Kafka-style replication is the default. Open source MirrorMaker 2 or vendor-specific cluster linking copies topics between an edge cluster and the central cluster, preserving ordering and history. Apache Kafka is the open-source option; Confluent, Amazon MSK, Aiven, and other solutions are the managed ones. This suits sites large enough to operate a streaming cluster, such as a factory with local analytics that also feeds the global platform. For thousands of small sites, the streaming cluster per site model collapses under its own operational weight. This is where NATS is architecturally distinct rather than just another broker. The entire server is a single small binary with no external dependencies. Each store, vehicle, or clinic runs a leaf node that dials out to the central cluster, keeps serving local applications when the WAN drops, and uses JetStream mirroring to synchronize persisted streams when connectivity returns. Retail chains run this pattern across thousands of locations. The same design serves both the intermittent case, hours or days offline with a bulk sync on reconnect, and the scheduled-batch case where sites replicate on a timer rather than continuously. Its own maintainers warn against one trap: do not stretch a single stream cluster across unstable edge links, because the cluster's consensus breaks every time the network does. Instead, let each site persist its data locally and keep serving through an outage, then mirror to the center asynchronously when the link returns. For enterprises whose sites run applications rather than device fleets, an event mesh covers the same ground at the application layer. Solace links its brokers across data centers and clouds so an event published anywhere reaches any interested subscriber, with topic routing and governance built in. It is the natural fit when the endpoints are SAP, mainframes, and microservices rather than sensors. And a large share of site-to-site traffic does not need streaming, even if streaming is often the stronger long-term architecture. Nightly file synchronization, batch uploads over metered links, and change data capture from edge databases remain perfectly valid when there is no real-time requirement. This is especially true when the existing OT and IT systems on both ends are batch-oriented anyway, and no business case justifies the cost and effort of making them real time. Real time is a requirement, not a default. Stream when you need fresh data, or when you build greenfield and carry no legacy batch systems to preserve. An architecture that streams everything pays streaming costs for batch problems. Which Category Serves Which Edge to Cloud Pattern? Seven vendor categories map onto the four patterns. No single category covers all of them, and messaging fabrics, event mesh, and protocol proxies deliberately span several. No single vendor covers every direction. Seven categories divide the field, and the real picture is that some sit in one pattern while others deliberately span several. MQTT platforms (EMQX, HiveMQ, Mosquitto, NanoMQ) serve both northbound telemetry and southbound control, because MQTT is bidirectional by design. Hyperscaler services (AWS IoT Core, Azure IoT Operations) do the same for the managed-cloud case, ingesting telemetry and pushing desired state through device twins. Event streaming (Apache Kafka, Confluent, Aiven, and more in the data streaming landscape) owns site-to-site replication. Last-mile delivery(Ably, Lightstreamer, PubNub) owns outbound delivery to people, and it is the only pattern with a category built solely for it. Three categories span multiple patterns. An event mesh (Solace) routes site-to-site and out to applications. Messaging fabrics (NATS, Zenoh) carry northbound, southbound, and site-to-site in a single fabric. Protocol proxies (Zilla) translate across all four, replacing several specialized components with one layer. The underlying distinction is worth stating plainly: a protocol standard is not a deployment architecture. MQTT standardizes the wire, which is why so many products speak it and your devices stay portable. NATS and Zenoh standardize the topology, which reduces moving parts and increases commitment. Zenoh, born from DDS research and adopted by automotive initiatives including General Motors' uProtocol and by the ROS 2 robotics ecosystem, goes furthest: it routes named data along the shortest path between producer and consumer, aiming to make the cloud versus edge placement debate irrelevant. These spanning options are not equivalent. A fabric like NATS or Zenoh reduces moving parts and unifies security by being one technology end to end, but that reach is also the lock-in: the more it carries, the more expensive it is to leave. A protocol proxy like Zilla does the opposite. It spans the patterns by translating between existing technologies, which keeps your exit options open at the cost of another component in the path. The right choice depends on how many patterns you actually need and how much lock-in you can accept, not on which vendor markets hardest. How to Choose: Five Questions That Pick Your Edge-to-Cloud Architecture Product evaluations start too late. Answer five questions first, and the shortlist mostly writes itself. Five-question decision framework for choosing an edge-to-cloud architecture, from consumer type to data residency Who consumes the data, machines or people? People need current state on connect, then live updates, which means a last-mile delivery layer over the backbone. Machines just consume the stream.Does this flow actually need real time? Measure before you architect. If a fifteen-minute batch satisfies the business case, use file sync or change data capture. Stream when you need fresh data, because streaming infrastructure you did not need is the most expensive kind.What is the most constrained node in the path? A microcontroller means MQTT or Zenoh, the two with clients small enough for the device itself, not a JVM. A gateway opens the field to lightweight NATS or full brokers, and a site server adds event meshes or a small streaming cluster.How many sites, and can each one operate a cluster? A few large sites with local analytics justify edge Kafka clusters with MirrorMaker or cluster linking. Hundreds or thousands of small sites demand hub-and-spoke: NATS leaf nodes or hierarchical MQTT bridges, with store and forward at every spoke. Enterprises whose sites run applications rather than devices fit an event mesh.Where must the data live? If residency or sovereignty rules apply, the broker runs on site or in jurisdiction, self-managed, and routing policy becomes a compliance control. If data may leave and connectivity is solid, managed cloud services win on operations cost. Two checks apply on almost every branch, at every step. The edge must dial out and never accept inbound connections to the site. And local operation must survive offline, with a buffer sized deliberately, not discovered in the first outage. Five questions, four patterns, one map. The technologies then sort themselves into it. Moving Data Is Not Automating the Business Every pattern and category in this article solves the same problem: moving data between edge, data center, and cloud. That movement is necessary but not sufficient. When telemetry arrives, something must decide what happens next: trigger the maintenance workflow, roll back the failed firmware deployment, replicate the dataset, open the ticket, retrain the model. Deciding what happens next is workflow orchestration. It spans the same edge-to-cloud boundary as the data movement beneath it. Data integration moves the events. Orchestration, with the process intelligence to see how those flows actually run, decides what the business does about them. And increasingly, agentic AI acts on those decisions at the edge. These are not three separate topics. They are one stack, and edge to cloud is where they meet the physical world. Architectures that move data and improvise everything above it end up with the automation logic scattered across broker plugins, cron jobs, and cloud functions. That is the integration spaghetti this whole discipline set out to remove. Edge to cloud is not one problem. It is four. Architect for the pattern, not the product.

By Kai Wähner

CORE

Member Spotlight: Shai Almong

What does it take to build and sustain an open-source project for more than a decade? For Shai Almog, creator of Codename One, it takes a willingness to challenge conventional approaches to software development and a continued commitment to the developers who rely on the work. I recently caught up with Shai to discuss the inspiration behind Codename One, the realities of maintaining an open-source project, and his perspective on how the Java ecosystem has evolved over the years. Check it out below! 1. What is Codename One, and what originally inspired you to create it? Codename One is what we always wanted Java to be: write once, run anywhere, without accepting the lowest common denominator. It’s what you would get if Spring Boot, Flutter, Go, and Node.js had a baby that inherited the best qualities of each. You get Java’s mature ecosystem, Flutter-style control over the UI/portability, Go-like native binaries and footprint, and the full-stack client/server connection people associate with Node.js. With Codename One, one Java codebase can target iOS, Android, macOS, Windows on x64 and ARM64, Linux on x64 and ARM64 with either glibc or musl, and even the browser. A small self-contained desktop GUI application can be around 4–5 MB, with no JVM to install. Underneath, Codename One uses different pipelines for different targets. On our ahead-of-time compiled platforms, ParparVM analyzes the reachable Java bytecode, removes unused code, translates what remains to C, and hands the result to the platform’s native compiler. Codename One also provides the portability libraries, UI toolkit, tests, theming, and build system needed to handle the ugly platform-specific details. We don’t pretend every platform is identical. Our builder architecture assembles the native pieces each application needs, so developers can use platform-specific capabilities without fragmenting the entire codebase. The original inspiration was frustration with Java’s unfulfilled promise. Java gave us portability, but conventional Java application deployment became increasingly heavy. We wanted the language and ecosystem without requiring a large runtime or giving up native performance, platform access, or control of the UI. Now we’re taking that architecture to the backend. The goal is to bring the startup time, footprint, and native compilation work we developed for constrained devices into serverless and microservice workloads. I want Java developers to get the full-stack productivity they expect from Spring Boot, with deployment economics closer to Go and Node.js. 2. Codename One has been around for more than a decade. What has kept you motivated? The community is the biggest reason. People have built businesses and products on Codename One. When someone has trusted your work with something that important, walking away isn’t an option. For a long time, competing in this market was brutal. Our competitors were not normal startups. They were Google, Facebook, Microsoft, and companies with teams larger than our entire community. There was a limit to how much a small team could build, test, document, and support. Agentic development changed that equation. It lets us scale engineering work in a way that simply wasn’t possible a few years ago. We now maintain an enormous cross-platform test matrix and can move at a pace that is competitive with much larger organizations. On most feature dimensions, I would argue that we are already ahead of the big-name alternatives. We still have gaps. We’re a small team, and pretending otherwise would be ridiculous. But our size is also an advantage. The person reading a bug report can be the same person who changes the compiler. We can respond to developers in hours where a large company may need three teams, a quarterly roadmap, and permission from a product manager. The other reason is simpler: we love Java, and we use the competing tools. Every time we do, we run into one of the compromises Codename One was created to remove. 3. What’s one of the biggest misconceptions developers have about maintaining an open source project? That maintaining open source is mostly about accepting code contributions. A pull request looks like free work from the outside. For a maintainer, it can be the beginning of a large amount of work. Is the code correct? Does it meet the project’s standards? Does it work on every target? Who maintains it five years later? How do you tell a volunteer who spent a week on something that the design is wrong? How many rounds of changes can you reasonably ask them to make? We recently stopped accepting outside pull requests. AI wasn’t the reason. If anything, AI improved the baseline quality of many contributions. It also made some of them enormous, because generating another thousand lines became almost free while reviewing those thousand lines did not. The real problem was validation. Codename One currently has more than 60 GitHub Actions workflow files. A small framework change can affect Android, iOS, JavaScript, desktop, bytecode translation, native compilation, and screenshot baselines. That matrix represents roughly $15,000 worth of monthly CI capacity, which GitHub currently provides to us. At that point, validating and repairing an outside patch often took longer than implementing the confirmed change ourselves. So today we ask people to contribute through reproducible issues, benchmark counterexamples, design discussions, testing, and documentation feedback. Those contributions are often more valuable than code because they expose problems we cannot reproduce in our own environments. Also sorry for all the GitHub Actions downtime ;-) 4. You’ve watched Java and the broader development ecosystem evolve considerably. What changes have surprised you the most? The biggest surprise is that Java effectively surrendered the idea of being small. GraalVM is a remarkable piece of engineering, so we aren’t literally alone (although it's a very different project that hasn't really taken the same direction and hasn't taken it as far). But mainstream Java spent years treating bigness as evidence of seriousness: larger runtimes, larger frameworks, more dynamic machinery, longer warmups, and more memory. The industry began treating that weight as inevitable. I'm not against comprehensive/large; Codename One itself is HUGE... But not in runtime, just in API and scope. Most successful newer platforms moved in the opposite direction. Go made self-contained native binaries normal. Node.js made application deployment almost trivial. Mobile and serverless made startup time and memory consumption economically important again. Java largely responded by optimizing large systems rather than questioning why every system needed to be large. Modules are a good example. They help organize and encapsulate a large dependency graph. They do not eliminate that graph or turn it into a small, closed-world native program. Codename One puts a real Java GUI application or full server into a roughly 5 MB binary. What surprises me is not that this works. We have spent years proving that it works. What surprises me is how lonely that position still is. Java could be one of the best languages for small native applications and low-cost serverless workloads. Most of the ecosystem simply stopped trying to make that happen and let Go eat our lunch. 5. After spending so much of your career in technology, what do you enjoy doing when you completely step away from a screen? Completely step away from a screen? I’m probably the wrong person to ask. I just got back from Thailand with my kids. We had a wonderful time. I also had my laptop with me and was doing pull requests on the beach. This is what I enjoy doing. I don’t have an unwind button. Vacations often leave me more exhausted than work because sitting still doesn’t recharge me. If I have an idea and I can build it, fix it, or argue about it with another developer, that is how I unwind. Explore more of Shai's content here.

By Dominique Pugh

Introduction to Error Prone: How to Catch Bugs at Compile Time

The Java compiler’s job is to check whether code follows the rules of the language. Trying to remove an element from a collection that can never contain it, or using the wrong letter in a date pattern, does not break those rules, so nothing stops the code from compiling. In short, any code that follows the rules compiles, even when it does not do what its author intended. What Is Error Prone? Error Prone is an open-source static analysis tool that Google built to fill this gap. Unlike most static analysis tools, it does not run as a separate step; it plugs into javac as a plugin. Using the information the compiler produces while analyzing the code, it looks for patterns that follow the rules of the language but are most likely wrong, and reports them as compiler warnings or errors. The idea is simple: a bug costs the least to fix when the code is written. Error Prone catches bugs during compilation, before they reach code review, CI, or production. Google has used the tool on its huge Java codebase for years, but its use is not limited to Google. NullAway, the null safety tool developed by Uber, runs as an Error Prone plugin. Spring Framework 7 checks its own codebase with NullAway during the build. Popular libraries such as Caffeine also use Error Prone in their builds. The examples in this article, which introduces you to Error Prone, are available in this repository. How It Differs From Other Static Analysis Tools The Java world has well-established tools such as SpotBugs, PMD, Checkstyle, and SonarQube. Three key features set Error Prone apart from them: It runs inside the compiler. It needs no separate build step or server. Because it is a plugin for javac, it works with any build tool that calls javac (Maven, Gradle, Bazel, Ant).It knows everything the compiler knows. The analysis runs on the AST that javac builds, after the type attribution and flow analysis phases. This gives Error Prone access to all symbol and type information. For example, it knows exactly what the generic type argument of an expression is, or which overload was chosen.It does not stop at a diagnosis; it suggests a fix. Most checks come with a suggested fix, which can be applied to the source code with a single command. How Does Error Prone Help Developers? The strength of Error Prone is that it can stop code that compiles but behaves incorrectly, right at compile time. Let’s take a closer look at the following example from the official documentation: Java Set<Short> s = new HashSet<>(); for (short i = 0; i < 100; i++) { s.add(i); s.remove(i - 1); } System.out.println(s.size()); This code seems to remove the previous element at each step, so you would expect it to print 1. It prints 100 instead, because the expression i - 1 is an int and is boxed to Integer. Since the Set<Short> contains no Integer, remove removes nothing. And because the remove(Object) signature accepts any type, the compiler does not complain either. Error Prone, however, refuses to compile this code: Plain Text [ERROR] ShortSet.java:[22,21] [CollectionIncompatibleType] Argument 'i - 1' should not be passed to this method; its type int is not compatible with its collection's type argument Short Now let’s look at the following code: Java private static final DateTimeFormatter FORMATTER = DateTimeFormatter.ofPattern("YYYY-MM-dd"); String reportName(LocalDate date) { return "report-" + FORMATTER.format(date) + ".csv"; } The code looks right when you read it, and it even works correctly for most of the year. However, Y in the pattern does not mean the calendar year; it means the week-based year. The calendar year is written with a lowercase y. When the last days of the year fall into the first week of the next year, the two values diverge. For example, for LocalDate.of(2025, 12, 29) the result is report-2026-12-29.csv, so the date jumps a year ahead. When the bug shows up also depends on the locale. The same code starts producing the wrong year on December 28 in the en-US locale, and on December 29 in the tr-TR locale. For the rest of the year the tests pass, and in code review the difference between YYYY and yyyy is easy to miss. Error Prone recognizes this pattern, stops the compilation and suggests the fix as well: Plain Text [ERROR] ReportDate.java:[16,83] [MisusedWeekYear] Use of "YYYY" (week year) in a date pattern without "ww" (week in year). You probably meant to use "yyyy" (year) instead. Did you mean 'private static final DateTimeFormatter FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd");'? Note that in both examples the code is valid Java and compiles without any problem. At best, your IDE shows a warning; the build does not break. Since the code can also pass the tests, the bug can live unnoticed for a long time. With Error Prone, you do not have to rely on anyone’s attention to catch these subtle bugs: the build fails, the error message explains the problem, and it often suggests the fix as well. Extending Error Prone Error Prone is not a closed set of rules. Where the built-in checks fall short, you can add your own rules on the same infrastructure they use. There are two ways to do this: Writing transformation rules with Refaster. Rules of the form “when you see this code, replace it with that” are written as plain Java templates.Writing your own BugChecker. Team-specific bug patterns are defined with the same API that Error Prone uses for its own checks. Refaster Refaster is a refactoring tool that is part of Error Prone, built for large-scale transformations across big codebases. You do not write the rule against an AST API, but in plain Java: “when you see this, replace it with that.” A Refaster rule is an ordinary Java class with at least two methods. @BeforeTemplate defines the pattern to look for, and @AfterTemplate defines the code to put in its place: Java class StringIsEmpty { @BeforeTemplate boolean before(String s) { return s.length() == 0; } @AfterTemplate boolean after(String s) { return s.isEmpty(); } } The template is matched with type information, not as text. That is why both name.length() == 0 and user.getEmail().length() == 0 are caught, while the same expression on a type other than String that also has a length() method is not. By writing more than one @BeforeTemplate, you can combine different patterns that lead to the same result in a single rule. Using a rule takes two steps: The rule class is compiled with the RefasterRuleCompiler plugin, which produces a .refaster file. The plugin lives in the error_prone_refaster artifact and is passed to the compiler with the -Xplugin:RefasterRuleCompiler --out rules.refaster argument.The generated .refaster file is passed to Error Prone with -XepPatchChecks:refaster:<file>. Error Prone writes the matches to a patch file, or applies them directly to the source code with -XepPatchLocation:IN_PLACE. Refaster is ideal for tasks such as migrating from one API to another, removing a deprecated method, or standardizing an idiom within a team. Picnic’s open-source Error Prone Support project also offers a way to run Refaster rules on every build, like a regular Error Prone check. Custom BugCheckers Error Prone’s built-in checks do not cover every rule. Alongside widely shared rules such as “sensitive data like national ID numbers, credit card numbers and email addresses must not be written to logs” or “do not use new BigDecimal(double) for monetary calculations", each team may also have its own rules, such as "this internal API must no longer be called". Error Prone lets you write these rules on the same infrastructure as its built-in checks. A custom check is a Java class that extends BugChecker. It declares which AST nodes it is interested in through the *TreeMatcher interfaces it implements. Error Prone calls the check for each such node, and the check returns a match and an optional fix. The following example implements the sensitive data rule. Although the rule is general, only the team knows which data is sensitive. That is why the team marks fields such as national ID numbers or email addresses with its own @SensitiveData annotation: Java public record Customer( String name, @SensitiveData String nationalId, @SensitiveData String email, @SensitiveData String cardNumber) {} The check reports an error when a log call is given a marked field, or an object whose toString() would print marked fields: Java @AutoService(BugChecker.class) @BugPattern(summary = "Sensitive data must not be written to logs", severity = ERROR) public final class SensitiveDataLogging extends BugChecker implements MethodInvocationTreeMatcher { private static final String SENSITIVE_DATA = "org.jugistanbul.SensitiveData"; private static final Matcher<ExpressionTree> LOGGER_CALL = instanceMethod() .onDescendantOfAny( "java.lang.System.Logger", "java.util.logging.Logger", "org.slf4j.Logger"); @Override public Description matchMethodInvocation(MethodInvocationTree tree, VisitorState state) { if (!LOGGER_CALL.matches(tree, state)) { return Description.NO_MATCH; } tree.getArguments().stream() .flatMap(argument -> stringifiedValues(argument, state)) // "..." + x → "...", x .filter(value -> isSensitive(value, state)) .forEach(value -> state.reportMatch( buildDescription(value) .setMessage("'%s' contains sensitive data and must not be written to logs" .formatted(state.getSourceForNode(value))) .build())); return Description.NO_MATCH; } private static boolean isSensitive(ExpressionTree value, VisitorState state) { // customer.email(): the field or accessor itself is annotated var symbol = ASTHelpers.getSymbol(value); if (symbol != null && ASTHelpers.hasAnnotation(symbol, SENSITIVE_DATA, state)) { return true; } // customer: the class, or one of its fields, is annotated, so toString() leaks it var type = ASTHelpers.getType(value); if (type == null || type.isPrimitive() || !(type.tsym instanceof ClassSymbol valueClass)) { return false; } return ASTHelpers.hasAnnotation(valueClass, SENSITIVE_DATA, state) || valueClass.getEnclosedElements().stream() .anyMatch(member -> member.getKind() == ElementKind.FIELD && ASTHelpers.hasAnnotation(member, SENSITIVE_DATA, state)); } // stringifiedValues(...) splits a string concatenation into its operands; see the repo } In the demo’s CheckoutService, the first line passes because it only logs the customer's name, while the other two do not compile: Java LOG.log(Level.INFO, "Order " + orderId + " placed by " + customer.name()); LOG.log(Level.DEBUG, "Order " + orderId + " placed by " + customer); LOG.log(Level.INFO, "Receipt for order " + orderId + " sent to " + customer.email()); Plain Text [ERROR] CheckoutService.java:[25,67] [SensitiveDataLogging] 'customer' contains sensitive data and must not be written to logs [ERROR] CheckoutService.java:[28,90] [SensitiveDataLogging] 'customer.email()' contains sensitive data and must not be written to logs The check looks at the expression passed to the logger, so a value first copied into a local variable (String email = customer.email();) is not tracked; following it would need data-flow analysis. This short class shows all the parts of a custom check: @BugPattern: Defines the rule's name, description, and severity (ERROR or WARNING). The rule can be turned off, or its severity changed, by its class name (-Xep:SensitiveDataLogging:WARN), and it can be suppressed with @SuppressWarnings.Matchers: Ready-made building blocks such as MethodMatchers express conditions like "any method of a System.Logger, java.util.logging or SLF4J logger" with type information, in a single expression.Type and symbol information: ASTHelpers gives access to the symbol and type of the logged value. Annotations are resolved by their fully qualified names, so an annotation with the same name in another package does not get mixed in. A text search cannot do this.@AutoService: Makes the check discoverable through ServiceLoader. Adding the jar that contains the checks to the annotation processor path is enough. This check only reports the error, but checks can also suggest fixes with SuggestedFix. BigDecimalDoubleConstructor in the repository is a short example that suggests BigDecimal.valueOf(double) instead of new BigDecimal(double). Checks are tested with CompilationTestHelper. // BUG: Diagnostic contains: comments in the test source mark the line where an error is expected. For checks that suggest fixes, BugCheckerRefactoringTestHelper compares the output of the fix with the expected source code; BigDecimalDoubleConstructorTest in the repository is an example. Running the Examples All the examples in this article are available in runnable form in the error-prone-in-action repository: ShortSet and ReportDate, which trigger the built-in checks, three custom checks with their tests, and the StringIsEmpty Refaster rule. The repository's README explains which command runs each example. Conclusion The Java compiler checks whether code follows the language rules, not what its author intended. Error Prone fills this gap from inside the compiler. Using javac’s type information, it catches code that compiles but is most likely wrong, and often suggests the fix as well. Where the built-in checks fall short, you can write transformation rules with Refaster and team-specific checks with the BugChecker API. In my view, this approach is more valuable today than ever. As AI coding assistants speed up the pace of writing code, tools that catch bugs at their cheapest moment, at compile time, are turning from a luxury into a necessity.

By Hüseyin Akdoğan

CORE

Why Vulnerability Management Fails Without Reliable Asset Data

Vulnerability management depends on knowing what needs protecting. Asset management provides that foundation by maintaining a current picture of IT infrastructure, its owners, and its business importance. Without reliable asset data, teams can invest in scanners and patching tools while leaving systems outside assessment scope and findings without anyone responsible for fixing them. The relationship works both ways: asset records guide vulnerability assessment and remediation, while scanning can reveal systems missing from those records. Keeping this information aligned with infrastructure changes is what connects vulnerability detection to action. Where Asset Data Actually Comes From Vulnerability scanners provide part of the asset picture. A reliable inventory combines data from several sources: SIEM systems link security events to specific hosts.Network traffic analysis and detection tools reveal devices through their activity.Active Directory and the CMDB provide existing device and ownership records.Cloud APIs and orchestration platforms report resources created and managed in those environments.Endpoint management tools provide device details and agent status, but may miss unmanaged devices.Manual records cover assets that automated discovery has yet to reach. No single source provides the whole picture. Each contributes part of it. The task is to bring those parts into one consistent inventory and assign owners who can confirm or correct what the tools report. Collecting records is only useful if someone validates them. What an Asset Record Should Include Before categorizing assets, agree on what each record should contain. CIS Safeguard 1.1 provides a practical starting point: a network address (where static), a MAC address, a hostname, an owner, a department, and an authorization status. For vulnerability management, three fields deserve particular attention: Criticality This helps determine the remediation SLA: how quickly vulnerabilities on the asset need to be addressed. Nothing else in the record does more work. Remediation Owner A named person or team responsible for implementing the fix. A general label like “IT” provides little direction. Last Successful Assessment Date The most underrated field in most inventories. Without it, an asset with no reported vulnerabilities can look the same as one the scanner has not assessed in three months. In many environments, these fields are left blank. Filling them in and keeping them current is a large part of what “process maturity” means in practice. The record should also identify the service the asset supports, its environment, and its exposure. This context provides asset intelligence that connects a host to its application and distinguishes production infrastructure from a temporary development environment. Keep the last observation time separate from the last successful security assessment. Asset identity needs similar care. IP addresses and hostnames can be reused, causing findings from retired systems to be linked to new infrastructure. Retain platform resource identifiers, together with their account or cluster context, when matching records across tools. Why Asset Classification Matters Say you have mapped out the events you are trying to prevent (a breach of customer data, an outage of the payment path, etc.). The next question is which assets sit in the attack paths leading to those events, and which ones are the targets themselves. That is what lets you prioritize remediation with a reason behind it instead of a severity score alone. Take a public-facing web server. Ideally, your tooling separates it automatically from an identical server sitting in a test environment. Most mature VM platforms support grouping by criteria like this: perimeter-facing assets land in one group automatically, and every new server on the perimeter inherits the group's SLA the moment it appears. The question driving the grouping is simple: does this asset sit on a path to an outcome that would actually hurt? Assigning Asset Criticality A practical approach is to use three tiers: High: Perimeter web servers, critical network devices, VPN gateways, and target systems that commonly appear in attack chains.Medium: Assets that provide indirect access to internal systems.Low: Assets whose compromise has limited impact and provides little access to important systems. Workstations deserve particular attention. A standard workstation and the CFO’s laptop may look similar in an inventory, but their access and business importance can differ significantly. A workstation with access to financial systems or sensitive data may warrant high criticality regardless of its device type. When an asset’s criticality is unknown, treat it as potentially important until you assess it. NIST CSF 2.0’s ID.AM-05 calls for prioritizing assets by classification, criticality, and business value. CIS Controls also calls for addressing unauthorized assets. In practice, an unknown asset should trigger investigation rather than receive a low priority by default. Asset criticality also needs vulnerability context. Evidence of active exploitation can change remediation urgency, while fix availability affects how the team can respond. An internal system with privileged access or sensitive data may need urgent attention even without direct internet exposure. A criticality tier should support that decision, not replace it. What the Standards Say About Asset Management These frameworks and standards share a common foundation: maintain accurate asset records and keep them aligned with the infrastructure. Framework or standard Asset management provisions CIS Controls v8.1, Control 1 Maintain an accurate inventory of enterprise assets, including mobile, remote, and cloud assets. Review and update it at least every six months. Maintain a weekly process for addressing unauthorized assets. CIS Controls v8.1, Control 2 Maintain a software inventory and review and update it at least every six months. Verify support status at least monthly. Necessary unsupported software requires a documented exception covering mitigating controls and acceptance of residual risk. NIST CSF 2.0, Asset Management (ID.AM) Maintain inventories of hardware, software, systems, and services, including supplier services. Maintain inventories of data and corresponding metadata for designated data types, plus representations of authorized network communications and data flows. Prioritize assets by classification, criticality, resources, and mission impact. Manage assets throughout their lifecycle. ISO/IEC 27001:2022, A.5.9 and A.5.12 Maintain an inventory of information and associated assets, including their owners. Classify information according to confidentiality, integrity, availability, and relevant stakeholder requirements. CIS specifies minimum review frequencies; infrastructure changes may require more frequent updates. NIST CSF describes cybersecurity outcomes rather than prescribing implementation requirements. ISO Annex A controls are selected and justified through the organization’s risk treatment process. Asset Management From the Security and IT Perspectives Security and IT have complementary responsibilities in keeping asset records current. From the security side, the work involves collecting and categorizing asset data to prioritize remediation, especially for vulnerabilities under active exploitation. Security teams also need to identify gaps in assessment coverage and ensure that findings reflect current configurations and assets still in use. From the IT side, the work involves registering assets when they are provisioned, recording configuration changes, and confirming when systems have been decommissioned. Those lifecycle updates need to reach both the CMDB and the vulnerability management platform. Decommissioning is where coordination often breaks down. A new server usually has a request, an owner, and a deployment process. Its removal may receive less attention. A VM can disappear from the hypervisor while remaining active in the CMDB and vulnerability platform. Its old findings then keep appearing as unresolved vulnerabilities, distorting remediation metrics and wasting time on systems that no longer exist. The opposite failure is more dangerous: an asset marked as decommissioned may still be running. Close findings on the basis of decommissioning only after confirming that the resource has actually been removed from service. Depending on the system, this includes checking platform status, revoking unnecessary credentials, and verifying that remaining storage has been securely deleted or retained under an approved policy. Asset Types That Traditional Inventories Often Miss Even a few years-old asset taxonomy no longer covers the full picture. New realities have changed what teams need to track. The list itself needs rethinking, beyond simply adding a few entries. Cloud Resources Nobody Turned Off A forgotten physical server occupies space and consumes power. A forgotten cloud resource can keep running with little visibility beyond the bill. Harness’s FinOps in Focus 2025 report projected $44.5 billion in wasted cloud infrastructure spending, equivalent to 21% of that spending. It also reported an average of 31 days to identify and eliminate cloud waste without sufficient automation. These are cost findings, rather than measurements of vulnerability exposure, but the operational concern carries over: forgotten resources may retain vulnerable software or unnecessary access. SaaS Sprawl Marketing signs up for an email platform. Engineering adopts a project tracker. HR adds an interview scheduling tool. These services can hold company data without passing through IT or appearing in its inventory. Once a service is used for company work, it needs an owner and a record of the data and access involved. AI Systems and Agents AI governance adds further reasons to maintain reliable records. ISO/IEC 42001 establishes a framework for managing AI systems, while NIST’s voluntary AI Risk Management Framework explicitly addresses AI inventories under GOVERN 1.6. The EU AI Act’s Article 49 sets registration obligations for specified categories of AI systems and organizations. These provisions differ in scope and legal status, but reinforce the need to know which AI systems the organization develops or uses. Containers, Kubernetes, IoT, and OT These environments need discovery methods suited to how they operate. Containers may disappear between scheduled scans, some devices cannot run agents, and sensitive operational systems may require restricted scanning. Combine platform inventories, passive discovery, and appropriate assessments to cover those gaps. Connecting Asset Data to the Attack Surface Gartner identified attack surface expansion as a leading security trend in 2022. Digitalization, hybrid work, AI, and cloud adoption continue to expand potential entry points. Connected systems and IoT devices add exposure. Applications built with open source components and cloud services inherit dependencies and potential vulnerabilities. Attackers can also reach organizations through compromised suppliers. Viewed this way, asset management provides an internal view of the attack surface. It helps map the resources and dependencies that continuous threat exposure management (CTEM) examines for exploitable paths, adding ownership and business context to potential entry points. How Inventory Gaps Develop Vulnerability management is only as good as the asset data it feeds on. Here is an example that illustrates the problem: an ERP system ran on roughly 500 nodes, all regularly scanned, with findings passed to IT. During modernization, the number of nodes grew, and some of the new ones became accessible from the internet. Those additions never reached the CMDB that security relied on. Some records were populated automatically, but these nodes required manual registration. That step was missed, and the gap went unnoticed. The lesson extends beyond this case: a central inventory still needs independent validation. Comparing monitoring and orchestration records against the CMDB could have revealed the missing nodes, provided those sources had visibility into them. Treat monitoring and orchestration tools as asset data sources. They may contain records of running resources that have never reached the CMDB or scanning scope. Work with the infrastructure team to identify those sources and establish a regular process for investigating discrepancies. A Practical Maturity Check Asset management’s real test is whether the inventory reflects the environment and supports remediation. Start with these questions: Are all subsidiaries, business units, and acquired companies represented?Does coverage include every host type: desktops, laptops, and servers?What about remote laptops that never connect to the VPN? Do they still report to your management and security tools?Does coverage extend across operating systems? Windows records in Active Directory alone cannot establish coverage of Linux, macOS, or Windows devices outside the domain.Are all servers covered, both domain members and standalone systems, on isolated and connected networks, locally and in the cloud, including anything recently migrated?What about the less obvious assets: ERP hosts, virtualization and container platforms, network equipment, IP phones, cameras, and badge readers?Can you distinguish assets with recent successful assessments from those with failed scans or stale results? Once you assess coverage, document any gaps and assign responsibility for addressing them. Gaps accepted temporarily should have an approved justification and a review date. A quick way to test this is to pick an asset from the inventory and ask who owns it and when it was last successfully assessed. If the answer takes several days and three forwarded emails, only to end with “that used to be Tom’s thing, but he left,” the process needs serious attention. Measure Coverage Alongside Remediation A falling count of open vulnerabilities can reflect lost visibility rather than successful remediation, and reading that number correctly takes some critical thinking. Track the percentage of known assets assessed within the required interval, the number without verified owners, and the time from provisioning to first assessment. Report failed assessments and stale records separately. Make the scope clear: high coverage of known assets does not prove the inventory is complete. Assessment methods must fit the environment. A Kubernetes workload may disappear between scheduled network scans. Scan container images and check deployment configurations, then link findings to the workloads using them. Keep these results distinct: an image scan does not establish that the deployed workload is securely configured. For systems that cannot tolerate active scanning, agree on an alternative with the operational owner. Passive discovery, software version records, and vendor advisories can help identify potential vulnerabilities without disruptive probes. Document what remains unverified and review those limitations regularly, so the dashboard does not present partial visibility as complete assessment coverage. The Bottom Line Asset management outranks vulnerability management in importance, not the other way around. Without a working picture of what you have, you cannot prioritize remediation with any confidence, and the gaps in your inventory are exactly where an attacker gets in, because they are the parts of the environment nobody is watching.

By Alex Vakulov

CORE

New York City Taxi Trips: Putting Neo4j and DuckDB to Work

In the previous article, we built a custom Neo4j procedure that embeds DuckDB directly into the Neo4j JVM and makes it callable from Cypher. We showed how it works and ran a few examples to prove the concept. Now it's time to put it to work on something real. In this article, we'll use that procedure on the NYC Taxi and Limousine Commission (TLC) trip record data for January 2024. The dataset covers nearly three million yellow cab trips and more than fifty thousand green cab trips across 265 taxi zones. Our goal is to aggregate those trips into a traversable graph in Neo4j — zones as nodes, routes as relationships — and then query it in ways that neither engine could handle as naturally on its own. The full source code is available on GitHub. Getting the Code Before starting, ensure that you've worked through the previous article and have the duckdb.query procedure built and installed in Neo4j. Then get this project: Shell cd ~ git clone --filter=blob:none --sparse https://github.com/VeryFatBoy/neo4j.git cd neo4j git sparse-checkout set neo4j-duckdb-taxi mv neo4j-duckdb-taxi ../neo4j-duckdb-taxi cd ../neo4j-duckdb-taxi The Dataset The NYC TLC publishes monthly trip records in Parquet format. Each row represents a single trip and includes the pickup and dropoff zone identifiers, fare amount, trip distance, and duration. Alongside the trip files, the TLC provides a zone lookup table that maps each of the 265 LocationID values to a borough and neighborhood name and a shapefile that defines the zone boundaries. We'll download all of this programmatically in a Jupyter notebook. Once the files are in place, DuckDB can query the Parquet files directly — no import step needed. Python RAW_DIR = os.path.expanduser("~/neo4j-duckdb-taxi/data/raw") for filename in ["yellow_tripdata_2024-01.parquet", "green_tripdata_2024-01.parquet"]: path = os.path.join(RAW_DIR, filename) size_mb = os.path.getsize(path) / 1024 / 1024 result = duckdb.sql(f"SELECT COUNT(*) AS row_count FROM read_parquet('{path}')").fetchone() print(f"{filename}") print(f" Size: {size_mb:.1f} MB") print(f" Row count: {result[0]:,}") print() DuckDB reads the Parquet file directly and returns the row count without requiring a separate import into a database. The same loop covers the green cab file. This is DuckDB in its natural habitat — fast analytical queries over columnar data without a server or a schema to maintain. Preparing the Zone Nodes Before loading any trips, we need the zone nodes in Neo4j. Each node represents one of the 265 taxi zones and carries the zone name, borough, and service zone. Separately, the first notebook computes a centroid for each zone from the shapefile, which uses a projected coordinate system (NY State Plane in feet). To get latitude and longitude values suitable for a web map, we reproject each centroid and save the results to a JSON file for the dashboard. Cypher CALL duckdb.query( 'SELECT LocationID, Zone, Borough, service_zone FROM read_csv_auto("{ZONE_CSV}") ORDER BY LocationID' ) YIELD row RETURN row.LocationID AS locationId, row.Zone AS zone, row.Borough AS borough, row.service_zone AS serviceZone Cypher MERGE (z:Zone {locationId: $locationId}) SET z.zone = $zone, z.borough = $borough, z.serviceZone = $serviceZone The first statement reads the zone lookup CSV through duckdb.query and the second merges one node per row. With all 265 zones in place, we can move on to the trips. Aggregating Trips With DuckDB The raw trip files are too granular for graph storage. We don't want three million individual trip nodes — we want one relationship per route that summarizes all the trips that ran that route in January. DuckDB handles the aggregation. For yellow trips, we work from a 100,000-row sample to keep the graph build fast; green trips are small enough to aggregate in full. SQL WITH sample AS ( SELECT * FROM read_parquet("{YELLOW_PARQUET}") USING SAMPLE 100000 ROWS ) SELECT PULocationID, DOLocationID, COUNT(*) AS trip_count, ROUND(AVG(trip_distance), 2) AS avg_distance, ROUND(AVG(fare_amount), 2) AS avg_fare, ROUND(AVG((epoch_ms(tpep_dropoff_datetime) - epoch_ms(tpep_pickup_datetime)) / 60000.0), 2) AS avg_duration_minutes, SUM(passenger_count) AS total_passengers FROM sample WHERE PULocationID IS NOT NULL AND DOLocationID IS NOT NULL GROUP BY PULocationID, DOLocationID The sample gives us one row per route with counts and averages already computed. We run the same query for green cab trips, substituting the green Parquet file and the appropriate timestamp column names, without sampling since the green dataset is much smaller. Writing to Neo4j With duckdb.query Now comes the part the previous article set up: using duckdb.query to run the aggregation from Cypher and write the results into Neo4j as relationships. The procedure executes a DuckDB SQL query and streams the results back as maps, which Cypher can process directly. Cypher CALL duckdb.query( 'WITH sample AS ( SELECT * FROM read_parquet("{YELLOW_PARQUET}") USING SAMPLE 100000 ROWS ) SELECT PULocationID, DOLocationID, COUNT(*) AS trip_count, ROUND(AVG(trip_distance), 2) AS avg_distance, ROUND(AVG(fare_amount), 2) AS avg_fare, ROUND(AVG((epoch_ms(tpep_dropoff_datetime) - epoch_ms(tpep_pickup_datetime)) / 60000.0), 2) AS avg_duration_minutes, SUM(passenger_count) AS total_passengers FROM sample WHERE PULocationID IS NOT NULL AND DOLocationID IS NOT NULL GROUP BY PULocationID, DOLocationID' ) YIELD row RETURN row.PULocationID AS pu, row.DOLocationID AS do, row.trip_count AS tripCount, row.avg_distance AS avgDistance, row.avg_fare AS avgFare, row.avg_duration_minutes AS avgDurationMinutes, row.total_passengers AS totalPassengers Cypher UNWIND $batch AS row MATCH (pu:Zone {locationId: row.puId}) MATCH (do:Zone {locationId: row.doId}) MERGE (pu)-[t:YELLOW_TRIP]->(do) SET t.count = row.count, t.avgDistance = row.avgDistance, t.avgFare = row.avgFare, t.avgDurationMinutes = row.avgDurationMinutes, t.totalPassengers = row.totalPassengers DuckDB reads the Parquet file, aggregates the rows, and returns them through a single CALL. The notebook maps the returned columns into the relationship properties, then passes the results to the second statement in batches of 500. Cypher matches the zone nodes and writes the relationships. The two engines are cooperating directly — DuckDB does what it is good at, Neo4j does what it is good at, and the boundary between them is the duckdb.query call. We run the same pattern for green cab trips, replacing the SQL source with the green Parquet file (using the lpep timestamp columns and no sampling) and the relationship type with GREEN_TRIP. The Graph Schema After both loads complete, the graph looks like this: 265 Zone nodes, each with locationId, zone, borough and serviceZoneYELLOW_TRIP relationships connecting pickup zones to dropoff zones, with count, avgDistance, avgFare, avgDurationMinutes and totalPassengersGREEN_TRIP relationships with the same properties Each route represented in the yellow sample or the full green dataset is now a first-class edge in the graph. We can traverse it, filter on it, and aggregate across it using Cypher. Querying the Graph The following examples are orchestrated from the accompanying notebook. Where a query produces zone IDs or route pairs, the notebook passes those results into the next statement. 1. Graph Filter -> DuckDB Aggregate Cypher first fetches all Manhattan zone IDs from the graph. DuckDB then reads the raw Parquet, filters to those zones, and aggregates trip stats. Cypher MATCH (z:Zone {borough: 'Manhattan'}) RETURN z.locationId AS locationId ORDER BY z.locationId Cypher CALL duckdb.query( 'SELECT PULocationID AS location_id, COUNT(*) AS trip_count, ROUND(AVG(trip_distance), 2) AS avg_distance_miles, ROUND(AVG(fare_amount), 2) AS avg_fare_usd FROM read_parquet("{YELLOW_PARQUET}") WHERE PULocationID IN ({id_list}) GROUP BY PULocationID ORDER BY trip_count DESC LIMIT 10' ) YIELD row RETURN row.location_id AS locationId, row.trip_count AS tripCount, row.avg_distance_miles AS avgDistanceMiles, row.avg_fare_usd AS avgFareUsd 2. Graph Routing -> DuckDB Enrichment Cypher finds the top 10 busiest routes from the aggregated relationships already in the graph. DuckDB then goes back to the complete raw Parquet to compute average trip duration for those exact zone pairs. Because the graph was built from a sample, the duration values here draw on the full dataset rather than the sampled one. Cypher MATCH (pu:Zone)-[t:YELLOW_TRIP]->(do:Zone) RETURN pu.locationId AS puId, pu.zone AS puZone, do.locationId AS doId, do.zone AS doZone, t.count AS tripCount ORDER BY t.count DESC LIMIT 10 Cypher CALL duckdb.query( 'WITH pairs(pu_id, do_id) AS ( VALUES {values} ) SELECT t.PULocationID AS pu_id, t.DOLocationID AS do_id, ROUND(AVG((epoch_ms(tpep_dropoff_datetime) - epoch_ms(tpep_pickup_datetime)) / 60000.0), 2) AS avg_duration_minutes FROM read_parquet("{YELLOW_PARQUET}") t JOIN pairs p ON t.PULocationID = p.pu_id AND t.DOLocationID = p.do_id GROUP BY t.PULocationID, t.DOLocationID ORDER BY avg_duration_minutes DESC' ) YIELD row RETURN row.pu_id AS puId, row.do_id AS doId, row.avg_duration_minutes AS avgDurationMinutes 3. DuckDB Scan -> Graph Context DuckDB reads the raw Parquet to find the top 5 pickup zones by volume. Cypher then pulls each zone's name, borough, and connectivity from the graph. Cypher CALL duckdb.query( 'SELECT PULocationID AS location_id, COUNT(*) AS trip_count FROM read_parquet("{YELLOW_PARQUET}") WHERE PULocationID IS NOT NULL GROUP BY PULocationID ORDER BY trip_count DESC LIMIT 5' ) YIELD row RETURN row.location_id AS locationId, row.trip_count AS tripCount Cypher UNWIND $ids AS locId MATCH (z:Zone {locationId: locId}) OPTIONAL MATCH (z)-[:YELLOW_TRIP|GREEN_TRIP]->(neighbor:Zone) RETURN z.locationId AS locationId, z.zone AS zone, z.borough AS borough, count(DISTINCT neighbor) AS connectedZones ORDER BY connectedZones DESC These queries traverse the graph rather than scanning rows. The aggregation is already baked into each relationship, so the work Neo4j does here is pattern matching and property lookup — exactly what a graph engine is built for. 4. Yellow vs. Green Comparison Across Shared Routes Cypher CALL duckdb.query( 'WITH yellow AS ( SELECT PULocationID, DOLocationID, COUNT(*) AS trip_count, ROUND(AVG(fare_amount), 2) AS avg_fare, ROUND(AVG(trip_distance), 2) AS avg_distance FROM read_parquet("{YELLOW_PARQUET}") WHERE PULocationID IS NOT NULL AND DOLocationID IS NOT NULL GROUP BY PULocationID, DOLocationID ), green AS ( SELECT PULocationID, DOLocationID, COUNT(*) AS trip_count, ROUND(AVG(fare_amount), 2) AS avg_fare, ROUND(AVG(trip_distance), 2) AS avg_distance FROM read_parquet("{GREEN_PARQUET}") WHERE PULocationID IS NOT NULL AND DOLocationID IS NOT NULL GROUP BY PULocationID, DOLocationID ) SELECT y.PULocationID AS pu_id, y.DOLocationID AS do_id, y.trip_count AS yellow_trips, y.avg_fare AS yellow_avg_fare, y.avg_distance AS yellow_avg_distance, g.trip_count AS green_trips, g.avg_fare AS green_avg_fare, g.avg_distance AS green_avg_distance FROM yellow y JOIN green g ON y.PULocationID = g.PULocationID AND y.DOLocationID = g.DOLocationID ORDER BY y.trip_count DESC LIMIT 10' ) YIELD row RETURN row.pu_id AS puId, row.do_id AS doId, row.yellow_trips AS yellowTrips, row.yellow_avg_fare AS yellowAvgFare, row.yellow_avg_distance AS yellowAvgDistance, row.green_trips AS greenTrips, row.green_avg_fare AS greenAvgFare, row.green_avg_distance AS greenAvgDistance This is the cross-file join that sets query 4 apart: a single CALL reads both Parquet files inside DuckDB, joins them on shared zone pairs, and returns a side-by-side fare and distance comparison for the top ten routes. The two files never leave DuckDB — Neo4j just receives the result rows. 5. Graph and Raw File Counts Side by Side Cypher MATCH (z:Zone) WITH count(z) AS zones MATCH ()-[y:YELLOW_TRIP]->() WITH zones, count(y) AS yellow MATCH ()-[g:GREEN_TRIP]->() RETURN zones, yellow, count(g) AS green Counts what's in the graph: zone nodes and aggregated route relationships. Cypher CALL duckdb.query( 'SELECT (SELECT COUNT(*) FROM read_parquet("{YELLOW_PARQUET}")) AS yellow_rows, (SELECT COUNT(*) FROM read_parquet("{GREEN_PARQUET}")) AS green_rows' ) YIELD row RETURN row.yellow_rows AS yellowRows, row.green_rows AS greenRows This uses duckdb.query to count the raw rows in the Parquet files. Running both of these queries together shows the relationship between the summarized graph and the source data it was built from. The graph holds 265 zones, 7,018 yellow route relationships and 5,314 green route relationships, while the underlying Parquet files contain nearly three million yellow trips and more than fifty thousand green trips. What the Data Shows A few things stand out from the query results. The busiest pickup zone in the full yellow dataset is JFK Airport (zone 132) with over 145,000 departures, and it connects to 236 distinct drop-off zones — a wider reach than Midtown Center (zone 161), which connects to 120 zones despite nearly as many departures. The Upper East Side dominates the top routes: the four busiest zone-to-zone pairs are all within or between Upper East Side North and South, with the short Upper East Side South to Upper East Side North run clocking the highest volume at 779 sampled trips. Among the shared routes examined, green cabs generally show higher average fares than yellow cabs for comparable trip distances — the Upper East Side North to Upper East Side North intra-zone route, for example, shows a green average fare of $8.51 versus $6.86 for yellow over a nearly identical distance. The Dashboard We built a small interactive dashboard to explore the graph visually, as shown in Figure 1. Figure 1. Dashboard It connects to Neo4j and runs Cypher queries live against the data we loaded. The page opens with three summary tiles showing the zone, yellow route, and green route counts. Below them are a map of pickup zone activity, a bar chart of the top pickup zones, a scatter plot of average fare against average distance, a grouped bar chart comparing yellow and green average fares on shared routes, and a table of the busiest routes. A sidebar lets us choose Yellow, Green, or Both and set how many routes and zones to show. Changing either control re-runs the relevant Cypher queries and re-renders the views in place. The map plots each zone as a circle marker, sized by trip volume and colored by borough. Hovering a marker shows the zone name, borough, and trip count. Because the zone centroids were computed and saved during the data preparation step, no additional spatial processing is needed at query time. The dashboard code is included in the GitHub repository alongside the notebooks. What Each Engine Contributes It's worth being explicit about the division of labor, because it illustrates why the two-engine approach is worth the setup cost. TaskEngineRead and aggregate Parquet filesDuckDBColumnar filtering and GROUP BYDuckDBStore zone nodes and route relationshipsNeo4jRoute traversal and relationship pattern matchingNeo4jBridging the two from Cypherduckdb.query procedure DuckDB never touches the graph. Neo4j never touches the Parquet files. The procedure is the seam between them, and it keeps the aggregation inside Cypher rather than in a separate ETL script. Summary We started with raw trip records in Parquet format and ended with a traversable graph that answers routing and fare questions in Cypher. DuckDB handled the heavy lifting on the file side — reading columnar data, filtering and aggregating millions of rows. Neo4j handled the graph side — storing zones and routes, serving pattern-match queries, feeding a live dashboard. The duckdb.query procedure connected the two engines without a separate ETL service. One CALL aggregated the data, and Cypher wrote the relationships. The full source code is available on GitHub.

By Akmal Chaudhri

CORE

PWA on iOS, Part 2: HTTPS, Manifest and Service Worker

This is Part 2 of the PWA on iOS series, and it’s the most practical one. There will be almost no history or corporate politics here, but there is groundwork you need for the rest of the series to make sense. You’ll learn why a PWA works on a laptop but not on an iPhone, and how HTTPS, the manifest, the Service Worker lifecycle, and two caches decide that. The typical story starts at a desk. The frontend runs on a laptop, the phone is on the same Wi-Fi network, Safari on the iPhone has an address like http://192.168.1.20:5173 open, and everything looks fine: the interface loads, the buttons work, and our PWA's icon sits next to the others on the phone's Home Screen. Then someone turns on airplane mode, taps the icon, and instead of the app, sees a message saying the page can't be opened. Meanwhile, everything is green in Chrome on the laptop: the worker is registered, the cache is full, and the team spends the first half hour hunting for a bug in the manifest, even though the phone has never seen that worker. No official tool will give you the answer, either. In iOS 26, Apple removed the last formal installation requirements, and any site added to the Home Screen now opens as a web app by default. The WebKit blog put it bluntly: "Put simply, Safari now has no requirements at all for installation." Google, for its part, stopped using Lighthouse as a PWA checker. Until Lighthouse 12, it had a separate PWA category: a set of automated checks that produced a score and showed whether an app met the criteria of an installable progressive web app. Lighthouse 12.0, released in April 2024, removed the PWA category, following Chrome's relaxation of its installability criteria. Even earlier, the Chrome blog acknowledged that the service worker check wasn't producing the intended result: sites were adding a worker with an empty fetch handler just to pass it. The green "this is a PWA" checkmark is gone on both platforms, and there is now only one way to tell whether you have an app or just a website with an icon: pick up a phone and test. Behind every web app icon on your phone's screen are three things: HTTPS, a service worker, and a manifest file. People usually think of HTTPS as the padlock in the address bar, but here it's needed for something else: without it, the browser simply won't agree to run code that acts as the bridge between the page and the network. The manifest is the simplest of the three: a small JSON file linked from <head> that records the app's name, start page, display mode, scope and icon. And the most interesting layer, the Service Worker, is a separate script the browser runs alongside the page but without access to its interface. The app's network requests pass through it, and it decides whether to go to the network, serve a saved copy, or show a fallback page when there's no connection. Why a Service Worker Needs HTTPS and a Secure Context A service worker is dangerous precisely because it's long-lived: once installed, it keeps running on later visits to the site and can intercept page requests within its scope. That's why registration is allowed only over HTTPS. The worker effectively sits between the web app, the browser, and the network, and it can alter requests and responses. If sw.js could be registered over plain HTTP, an attacker on public Wi-Fi could swap that file just once. A malicious worker would then stay in the user's browser and could keep serving altered HTML and JavaScript on later visits, harvest API data, or redirect requests to another server. Moving the site to HTTPS afterward wouldn't be an instant fix either: requests are handled first by the installed service worker, which can return a forged response from its cache or change how the network is accessed. The risk of this kind of attack is one of the reasons browsers require HTTPS for service workers. That's where the secure context requirement comes from. A page counts as secure if it was delivered over HTTPS or from an address the browser trusts on its own: localhost and its subdomains, the 127.0.0.0/8 range and ::1. Home or office network addresses like 192.168.x.x and 10.x.x.x aren't on that list. To the browser, that's plain HTTP, however much you trust your router. The sneakiest part is the way the browser says no. In the specification, navigator.serviceWorker is marked as available only in secure contexts, and on an HTTP page it simply isn't there. The check 'serviceWorker' in navigator, which almost every registration example starts with, quietly returns false, the code skips registration, and not a single error shows up in the console. That explains the story from the start of this article: on the phone, the worker didn't fail with an error. It never even tried to start. Let’s Encrypt and Free HTTPS Certificates for PWAs About ten years ago, many teams heard "use HTTPS" as "spend money and deal with hassle." You bought a certificate from a certificate authority and renewed it by hand, and for an internal tool or side project, that meant an extra line in the budget and an extra date on the calendar that everyone forgot until the morning the site started scaring visitors with a red warning. Let's Encrypt, a nonprofit certificate authority run by ISRG, changed that. On December 3, 2015, it entered public beta: invitations were no longer required, certificates were free, and the announcement stated the goal plainly: "We want HTTPS to be the default." Cost was only half of the problem. The server proves on its own that it controls the domain and obtains a certificate on its own via the ACME protocol, with no human involved and no correspondence with a certificate authority, and in March 2019 the IETF made that protocol official as RFC 8555. Let's Encrypt certificates are valid for 90 days by default, and such a short lifetime is practical only because there's no need to renew them by hand. These days, getting HTTPS for a worker takes about one toggle in a hosting panel, or a Certbot container next to nginx. I think this quietly ended many of the PWA arguments on internal projects: the money argument is gone, and what's left is the laziness argument. Testing a PWA on iPhone With HTTPS, Tunnels and mkcert On the laptop itself everything works, because http://localhost is on the trusted list and browsers on macOS will happily register a worker with no certificate at all. Trouble starts when you pick up the phone. On an iPhone, localhost means the iPhone itself. The laptop can only be reached by its network address, and to Safari that address is ordinary insecure HTTP. There are two workable paths: route traffic through a tunnel with a real certificate, or teach the phone to trust your own. The tunnel is easier. The command cloudflared tunnel --url http://localhost:5173 brings up what Cloudflare calls a quick tunnel. You get a random subdomain on trycloudflare.com, requests to it travel through Cloudflare's network to your local server, and the address has a public certificate, so the iPhone trusts it with no setup. ngrok http 5173 does the same through its own service. Cloudflare states plainly that quick tunnels are intended only for development and testing, and points to a different tunnel type for production. The tunnel has a catch that people only notice a week later. The random address changes on every launch, and a new subdomain means a new origin to the browser (the scheme, host, and port by which a browser tells one site from another). To an iPhone, every such address becomes a separate app: yesterday's icon points to a subdomain that no longer exists, today's starts with empty storage and no worker, and a row of identical icons for your PWA gradually grows on the test phone. If you plan to test an update on top of an old version, set up a permanent address: a named tunnel or your own domain. The second path goes through a local certificate authority, and the easiest way to set one up is with mkcert. The command mkcert -install creates a root certificate on your Mac and adds it to the system trust store, and mkcert 192.168.1.20 localhost issues a certificate for the dev server. The Mac is now happy. The phone isn't yet. You need to move the rootCA.pem file from the folder shown by mkcert -CAROOT to the iPhone via AirDrop, email, or a local web server, and install it as a profile. Then comes a step almost everyone skips: open Settings > General > About > Certificate Trust Settings and turn on your certificate under "Enable Full Trust for Root Certificates." Apple's instructions say explicitly that for profiles downloaded from a website or sent by email, trust for SSL/TLS has to be enabled manually, and without that switch the installed profile changes nothing. This setup has a price, and the mkcert README names it plainly: the rootCA-key.pem file gives full power to intercept secure requests from your machine, and it must never be shared. A root certificate on the phone works in the same direction. As long as it's trusted, whoever holds the key can issue a certificate for any domain, and the iPhone will accept it. I keep such profiles only on test devices and remove them when the test setup is no longer needed. Clicking through a certificate warning doesn't count as trust. A page with a self-signed certificate will open if you push through the warning screen, but registering a worker on it usually ends in an error, and until someone surfaces that error on screen, developers keep fixing the manifest. How to Register a Service Worker and Read Registration Errors Once you have HTTPS, registration itself looks modest, and almost everything interesting in it happens in the catch branch: // app.js if ('serviceWorker' in navigator) { window.addEventListener('load', async () => { try { const reg = await navigator.serviceWorker.register('/sw.js', { scope: '/', updateViaCache: 'none', }); console.info('[sw] scope:', reg.scope); } catch (error) { console.error('[sw] registration failed:', error); } }); } else { console.warn('[sw] navigator.serviceWorker is missing: HTTP or an old browser'); } The else branch handles that silent refusal on HTTP: better one warning line in the console than half an hour of guessing. I hook registration to load so that downloading and installing the worker doesn't compete with the first render for the network, which is noticeable on slow mobile connections. The scope parameter sets the scope, meaning the prefix of addresses whose requests the worker will intercept. By default it equals the directory where the script itself lives. A worker at /assets/sw.js can't take control of / unless the server sends a Service-Worker-Allowed: / header along with the script, so it's simpler to keep sw.js in the root. We'll come back to updateViaCache in the section on caches; it has a surprising wrinkle. Don't leave the catch branch empty or skip it. Its errors usually tell a specific story that you won't see on a laptop. The specification forbids redirects when loading a worker script, so an sw.js that the server redirects to the www version of the domain, to a CDN or to a login page won't register at all. If the server serves the script with the wrong MIME type (the content type from the Content-Type header), registration fails with a SecurityError. This happens most often in SPAs, single-page apps where JavaScript handles all navigation and the server answers any unknown path with index.html: instead of a script, the browser gets HTML. One more small thing helps with debugging. The browser adds a Service-Worker: script header to the request for a worker script, so such requests are easy to find in logs or to single out with a separate rule on a CDN, the network of caching servers in front of your backend. Installing is up to the user. iOS has no beforeinstallprompt event and no way to trigger the system install dialog from code. The person opens the Share menu and picks Add to Home Screen, and in iOS 26 sees a toggle there, "Open as Web App," which is on by default. What happens in the system at that moment, where the icon and splash screen come from and what MDM has to do with it, we'll cover in Part 4 of this series. Web App Manifest Fields Safari Supports on iOS The manifest is hooked up with one line in <head>: <link rel="manifest" href="/manifest.webmanifest">. The file can have any name, and manifest.json works too, because the browser follows the link and doesn't look for a file by name. The right response type is application/manifest+json. It's registered with IANA together with the .webmanifest extension, and the specification recommends serving the manifest with it. Plain application/json is accepted in practice as well, but I'd go with the first so the next person on the team doesn't have to guess what's behind that address. Safari 15.4 brought a small but welcome change: the browser now downloads the manifest along with the page, whereas before it did so only at the moment the user added the site to the Home Screen. The manifest request is now visible in Web Inspector right away, and you don't have to install the app every time just to check. Here's the file for our imaginary app: { "id": "/", "name": "Field Journal", "short_name": "Journal", "start_url": "/?source=home-screen", "scope": "/", "display": "standalone", "theme_color": "#0d1b2a", "background_color": "#f4f7f9", "icons": [ { "src": "/icons/icon-192.png", "sizes": "192x192", "type": "image/png" }, { "src": "/icons/icon-512.png", "sizes": "512x512", "type": "image/png" } ] } Going by MDN's compatibility tables, Safari on iOS reads noticeably less from this file than Chrome does. It has understood name, short_name, start_url, scope and display since iOS 11.3, the very beginning of manifest support. theme_color arrived in iOS 15, icons in 15.4, and id in 16.4. Icons come with a condition described in the Safari 15.4 release notes: Safari takes them from the manifest only if the HTML has no apple-touch-icon and the icon either has no purpose or has any. WebKit added the id field in the same release as Web Push, so that notifications and badges would work for several installs of the same app, for example a work copy and a personal copy with different accounts. According to MDN, the iPhone ignores most of the rest. orientation, description and share_target aren't supported on iOS, and Safari understands shortcuts only on macOS, starting with version 17.4. Of the display values, standalone and browser work on iPhone, while fullscreen and minimal-ui aren't supported. The exception is background_color: Safari uses it, together with name, when creating the web app's launch screen. You don't need to remove the other fields from a shared manifest. They're useful on Android and Chrome, and they don't bother Safari. But don't count on them for anything critical on iPhone. short_name is the label under the icon, and it's meant to survive the cramped Home Screen: "Field Journal" might not fit there, "Journal" will. start_url doesn't have to point to the site root. A parameter like ?source=home-screen is handy for analytics, so the server can see launches from the icon, but don't use it to check authorization, because anyone can type such an address. Relative paths inside the manifest are resolved against the manifest's own address, and a start_url from a different origin must, per the specification, be ignored, with the current page's address used instead. id comes with a trap that's easy to miss. If the field is missing, the app's identity per the specification is start_url, and a harmless tweak of an analytics parameter can look to the browser like the appearance of a different app. I set id explicitly in the first release and never touch it afterward. "id": "/" is resolved against the origin and becomes the site root, a stable name that survives any experiments with the start page. display: "standalone" asks for a separate window without an address bar, and theme_color sets the color of the system elements around the page. Both fields are requests to the browser and have nothing to do with your interface's CSS. The browser will carry them out in its own way. Why Apple Meta Tags Still Matter in 2026 Apple taught the iPhone to open sites without browser chrome long before the W3C took up the manifest, and it did so with its own meta tags. They haven't gone anywhere, and I still keep both sets in <head>: <link rel="manifest" href="/manifest.webmanifest"> <meta name="theme-color" content="#0d1b2a" media="(prefers-color-scheme: dark)"> <meta name="theme-color" content="#f4f7f9" media="(prefers-color-scheme: light)"> <meta name="apple-mobile-web-app-capable" content="yes"> <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"> <link rel="apple-touch-icon" href="/icons/apple-touch-icon-180.png"> apple-mobile-web-app-capable historically asked iOS to launch the site from the icon without Safari's interface. In iOS 26 that became the default behavior for any site, but duplicating it is still useful if some of your users are on older iOS versions or on corporate devices that are rarely updated. apple-mobile-web-app-status-bar-style controls the status bar, the one with the clock and battery, and Apple's old reference describes three values for it. With default and black, the page starts below the status bar, while with black-translucent it takes the whole screen and the status bar partly overlaps it. It looks great right up until the first header slides under the clock, so check your insets with env(safe-area-inset-top) as soon as you use that value. The most useful thing in this bunch, I think, is apple-touch-icon. Since Safari 15.4 it explicitly takes priority over manifest icons, and WebKit explains this as a wish to preserve the behavior of sites that have a separate icon drawn for iOS. Use an opaque 180×180 PNG: a transparent background was historically handled badly on iOS, and on older versions the alpha channel turned into a black backing. If the tag isn't there at all, iOS, according to Apple's documentation, looks for files with the apple-touch-icon prefix in the site root on its own. That's where the endless 404s for /apple-touch-icon.png in your server logs come from, the ones nobody can usually explain. Safari won't use an icon whose only purpose in the manifest is "maskable", since that value comes from Android's icon conventions. theme-color arrived in Safari 15 and turned out to be more noticeable than expected. In Safari on iOS 15 it colors the status bar and the area visible when you scroll past the edge of the page, and with a media attribute you can set different colors for light and dark themes, as in the example above. After that the story gets murkier: MDN's tables note that as of Safari 26, the theme-color color is used only in installed web apps. I didn't find that wording in Apple's release notes, so check on whichever iOS version you have in hand, and don't promise your designer a colored header in a Safari tab. Service Worker Lifecycle and skipWaiting on iPhone The most persistent misconception about workers is that a worker runs in the background all the time, like a daemon on a server. The browser starts it for an event (a page request, a push notification, a message from a tab) and may stop it right after the event is handled. A worker has no DOM and no window, and global variables live only until the next stop, so a "sync in progress" flag stored in a variable will be empty after the worker wakes up. Anything that has to survive the worker being stopped should be written to Cache Storage or IndexedDB. A service worker's lifecycle is designed so that a new version doesn't break pages that are already open. After register(), the browser downloads the script and compares it byte for byte with the installed one, and if there's no difference, that's the end of it. A new script gets an install event, in which it usually puts the app shell into the cache, meaning the HTML, styles and scripts without which the interface can't be drawn. Then it waits in the waiting state while the old version serves open tabs, and only when none are left does it get activate, clean up old caches and start receiving fetch events from pages in its scope. On the first run there's one more confusing detail: the page that registered the worker isn't controlled by it until the next navigation, unless the worker calls clients.claim(). // sw.js const VERSION = 'shell-v7'; const SHELL = ['/', '/assets/app.8d91.js', '/assets/app.3c2e.css']; self.addEventListener('install', (event) => { event.waitUntil(caches.open(VERSION).then((cache) => cache.addAll(SHELL))); }); self.addEventListener('activate', (event) => { event.waitUntil( caches.keys().then((keys) => Promise.all(keys.filter((key) => key !== VERSION).map((key) => caches.delete(key))) ) ); }); self.addEventListener('message', (event) => { if (event.data?.type === 'skip-waiting') self.skipWaiting(); }); event.waitUntil() looks like a housekeeping detail, but without it the browser doesn't know the event is still in progress and is free to stop the worker before the cache is written. On the iPhone, where background processes inevitably get put to sleep, that's a very practical risk. cache.addAll() has a quirk of its own: it's atomic, and if even one file in the list returns a 404, the whole installation fails. The new worker then simply doesn't start, users stay on the old version, and your logs are empty, because the error happened on the phone. I don't write the list of app shell files (everything the app can't launch without) by hand. The build tool generates it from the real files with hashes in their names, so it can't contain an address that doesn't exist on the server. Optional images I cache in a separate step, so that one unavailable file can't fail the installation. The waiting state is often mistaken for a bug, though it's protection against shooting yourself in the foot. If a new worker takes over a tab where old JavaScript is already running, you get a mix of the old interface with the new cache or a new API contract. Calling self.skipWaiting() on every install, together with clients.claim() in activate, removes the waiting state entirely, and for a static site or strictly backward-compatible releases that's a fine choice. For an app with an IndexedDB migration or a changing offline data format, I don't do that, and in the code above the worker skips waiting only on a message from the page. The page, noticing registration.waiting, shows a "New version available" banner, sends reg.waiting.postMessage({ type: 'skip-waiting' }) on tap, and reloads in the controllerchange event handler. In my experience, on iOS this scheme noticeably reduces the number of mysterious bug reports from people who come back to the app after a couple of days in the background. Background work on the iPhone is more modest than tutorials promise. Background Sync, Periodic Background Sync and Background Fetch aren't available in Safari or other browsers on iOS. So a photo upload queue that will "send the files by itself ten minutes from now, while the phone sits in your pocket" won't work. A Service Worker wakes up only when the app is open or when an event arrives that iOS is willing to deliver, such as a push. Send what has accumulated when the app returns to the foreground, and show the user that an operation is waiting for a connection. We'll cover updates, push, and IndexedDB storage in detail in Part 5 of this series. For now, remember this: a worker wakes up briefly and remembers nothing between wake-ups. Service Worker Cache Strategies and Opaque Responses In the fetch handler, the worker decides the fate of every request, and the most common mistake here is answering everything from the cache because that's what the tutorial did. For a hashed /assets/app.8d91.js, the Cache First strategy is nearly perfect: a file with that name will never change, and it doesn't need the network. For GET /api/orders, the same strategy will show yesterday's order as today's, and the user won't even know. In practice, there are three strategies, and each negotiates freshness with the user in its own way. The simplest, Cache First, returns the copy and goes to the network only on a miss. It belongs on versioned files, fonts, and icons. For HTML documents and data where freshness matters more than milliseconds, I use Network First: network first, and the saved response only when the network is down. The third strategy, Stale-While-Revalidate, is trickier. The user immediately gets the saved copy while the worker requests a fresh version in the meantime and puts it in the cache for next time. The response arrives quickly but may lag by one visit. For avatars, configs and feeds, that's usually fine. In the fetch handler, the strategy is chosen by what kind of data it is: self.addEventListener('fetch', (event) => { const { request } = event; const url = new URL(request.url); if (request.method !== 'GET' || url.origin !== self.location.origin) return; if (request.mode === 'navigate') { event.respondWith(networkFirst(request, '/')); } else if (url.pathname.startsWith('/assets/')) { event.respondWith(cacheFirst(request)); } else if (url.pathname.startsWith('/api/feed')) { event.respondWith(staleWhileRevalidate(event)); } }); async function cacheFirst(request) { return (await caches.match(request)) ?? fetch(request); } async function networkFirst(request, fallbackUrl) { const cache = await caches.open(VERSION); try { const response = await fetch(request); if (response.ok) cache.put(request, response.clone()); return response; } catch { return (await cache.match(request)) ?? caches.match(fallbackUrl); } } async function staleWhileRevalidate(event) { const cache = await caches.open(VERSION); const cached = await cache.match(event.request); const fresh = fetch(event.request).then((response) => { if (response.ok) cache.put(event.request, response.clone()); return response; }); event.waitUntil(fresh.catch(() => {})); return cached ?? fresh; } Requests for which the worker didn't call respondWith() go to the network as usual, and I consider that the right default: anything you haven't explicitly classified should behave as if there were no worker. For the same reason, the first line filters out other hosts and everything except GET. For a third-party CDN without CORS, the browser gives the worker what's called an opaque response. Its status is 0, the body and headers are unavailable to the script, and response.ok is false. Code that checks ok won't cache it, and images that "seemed to be cached" will vanish when you're offline. You can put such a response in with put(), but then you won't know whether you've just saved a 404 for a month. Caching third-party resources makes sense only if the CDN sends Access-Control-Allow-Origin and the request goes in cors mode. Otherwise it's easier to proxy the files through your own domain or accept that they load only online. In staleWhileRevalidate, note event.waitUntil(). Without it, the cached response goes to the page, the browser decides the work is done, and it may stop the worker before the fresh version lands in the cache. Tutorials often skip that line, though on iPhone it's exactly what separates a real staleWhileRevalidate from an imitation. I handle page navigations separately, by request.mode === 'navigate'. A SPA opens addresses like /orders/42 that aren't in the cache as files, and if the offline fallback covers only /index.html, opening such a link directly without a connection ends in an error. For a SPA it's reasonable to return the cached shell /, as in the code above, and let the app decide which data to show. A server-rendered site can't do this: if you answer every address with the same shell, the worker will break link previews, diagnostics and SEO, and the team will then blame Safari. There, the fallback should be a separate page like /offline.html. credentials is a trap that can go unnoticed for a long time. fetch(event.request) inherits the settings of the original request, while a new Request assembled by hand sends cookies only to its own origin by default. As long as the API lives on the same domain, there's no difference. When the API moves to api.example.com while the app stays on app.example.com, requests rebuilt in the worker suddenly arrive without cookies, the user looks logged out only on the screens that passed through the worker, and reproducing it on desktop doesn't always work the first time. Responses with personal data are better kept out of the Cache API unless you've thought through sign-out. The cache is tied to the origin and knows nothing about the session, so on sign-out, delete the entries holding the user's data, and don't put responses that depend on a cookie or token into a shared cache out of habit. POST can't be put there at all, since the Cache API accepts only GET, and offline writes have to be built as a queue of operations in IndexedDB with an idempotency key and an explicit sync status. While the worker starts after a long idle period, navigation waits for it, and on iPhone that's sometimes visible as an extra empty frame. Since Safari 15.4 there's Navigation Preload: the browser starts the network request for the page in parallel with starting the worker, and in the handler you pick up the ready response from event.preloadResponse. It's enabled by calling self.registration.navigationPreload.enable() in activate, and it makes sense when documents already go through Network First. HTTP Cache vs Cache API and How sw.js Updates There are actually two caches in the browser, and they know nothing about each other. The first, the HTTP cache, is always there. The browser itself stores server responses in it and decides on its own when they've gone stale, going by the Cache-Control, ETag and Vary headers. You control it only indirectly, through those headers. It can even answer a fetch() from a worker without waiting for the network, and it can evict an entry when it needs space. You can't rely on it offline. The second, the Cache API, exists specifically for the worker. It's storage that you manage by hand: what to put in, under which key, when to delete it, and what to return instead of the network. That's why offline is built on it: the files needed for launch sit where you yourself put them. The Cache API doesn't read headers. Even if the server allowed a file to be cached for a year via max-age=31536000, it won't get into the Cache API on its own, because only your code puts things there. And conversely, an entry in the Cache API won't be updated via ETag until you write that check yourself. The classic cache problem looks like this. A bundle is named /app.js with no hash, the worker put it in the Cache API at install, you shipped a fix, and caches.match('/app.js') keeps returning the old version: the key is the same, after all. The fix is simple: give the file a hashed name, such as /assets/app.8f3a.js, plus the header Cache-Control: public, max-age=31536000, immutable. Then you won't have to ask users to "clear the cache." Nobody deletes the old URL; people just stop requesting it. Hashed names have a flip side, and it hits SPAs especially hard. After a deploy, old files disappear from the server, while the user still has the previous version's shell open, and that shell asks for /assets/app.8d91.js. If the server answers any unknown path with index.html and a 200 status, the browser gets HTML instead of a script, fails while parsing it, and the user sees a white screen. Worse, a strategy that caches responses with a 200 status will save that HTML under the name of a JavaScript file. So I keep old assets on the server for at least one release and don't let the SPA fallback into /assets/: a missing file there must return 404, and nothing else. On nginx you can set it up like this: location = /sw.js { add_header Cache-Control "no-cache"; } location = /manifest.webmanifest { types { } default_type application/manifest+json; add_header Cache-Control "no-cache"; } location /assets/ { add_header Cache-Control "public, max-age=31536000, immutable"; try_files $uri =404; } location / { try_files $uri $uri/ /index.html; } sw.js itself isn't part of the hashing scheme. It needs no-cache so that every check reaches the server, including through CDNs and proxies. This is where the promised detail about updateViaCache comes in. Per the specification, the default is 'imports', and with it the worker's main script is requested bypassing the HTTP cache during an update check anyway, while only files pulled in with importScripts() go through the cache. The value 'none' extends the same rule to imports, and 'all' brings the HTTP cache back for everything. According to MDN, Safari has supported this setting since version 11.1. I set 'none' in projects where several teams touch the cache infrastructure: it does no harm, and it insures against a single forgotten importScripts with a long max-age. The specification describes quite strictly when a browser checks for an update. A check is triggered on navigation to a page within the worker's scope, on functional events such as push if more than a day has passed since the last check, and on a manual call to registration.update(). If the last check was more than 86,400 seconds ago, the request for the script bypasses the HTTP cache in any case, even with updateViaCache: 'all'. For a SPA there's a trap hidden here: transitions through history.pushState don't count as navigation, and an app that's been open on an iPhone since morning may never ask the server about a new sw.js until evening. A "Check for updates" button in settings that calls update() and offers a restart when a waiting worker appears helps more than hoping for the daily timer. The browser will dutifully go to the network, but between it and the server there may be a CDN that someone once told to cache everything with a .js extension for a week. Then half of the iPhones sit for a week on a worker that no longer exists on the server, so check the rule for sw.js on the CDN separately, preferably with a request from a phone. Service Worker Scope Mistakes Including the Trailing Slash The specification compares scope as a string prefix, with no understanding of directories. A worker or manifest with a scope of /app will also capture /app-v2 and /application, and a new storefront will suddenly live on the old dashboard's cache. Write /app/ with a trailing slash, and after registering, look at registration.scope in the console. If it isn't the prefix you expected, you can stop debugging cache strategies. One origin can have several registrations with different scopes, and each page is controlled by the one whose scope matches its address and is longer than the others. That sounds convenient until the landing page has registered a worker on /, the dashboard has its own on /app/, and both start answering shared resources in their own ways. I've seen a test environment where, after such a story, the dashboard behaved differently depending on which page the person first landed on, and sorting it out took longer than writing both workers. It's calmer when an origin has one owner: one worker with routing inside fetch, or separate subdomains if the teams really can't share a file. A subdomain means a different origin, different cookies and a different cache, so it's a real boundary, not a free optimization. There's an API mix-up related to this. navigator.serviceWorker.getRegistration() takes a page address (scope has nothing to do with it) and returns the registration that controls that address, and without an argument it uses the current page's address. If you're looking for the dashboard's registration from the home page, pass an address inside the dashboard. Otherwise you'll get the landing page's registration and wonder for a long time why it has no waiting. Then there's the habit of registering sw.js?v=3 to "reset the cache." Jake Archibald, in an article on the worker lifecycle, explains where that ends up: the HTML of the page, which carries the new address, is itself sitting in the old worker's cache, and the user simply won't see the new address. His wording is worth quoting: "You've put yourself in a position where you need to update the service worker in order to update the service worker." Keep one permanent address, /sw.js, and leave freshness to the headers and to the byte-by-byte comparison that the browser does without you. The manifest's scope has its own mix-up, and it's often explained wrongly. Per the specification, if scope isn't specified, it's computed from start_url: the start address is taken without the file name, query and fragment, and the directory where the manifest lies plays no role here. The manifest can sit in the site root, but with "start_url": "/warehouse/index.html" and no scope, it gets a scope of /warehouse/, and a link to /account takes the person beyond the app's boundary. There the page, per the specification, stops being considered part of the app and is shown with browser elements again. If start_url doesn't lie inside an explicitly specified scope, the browser silently discards that scope and falls back to the default, so a typo in one field changes the boundaries of the whole app with no warning. The way out is simple: an explicit "scope": "/" where the account should be part of the app, or a conscious decision that it lives separately. The second, in my observation, almost nobody chooses. One last pitfall: manifest.webmanifest and sw.js must not hide behind authorization. If the server redirects an unauthenticated request to a login page, the worker won't register, because redirects are forbidden. The manifest case is funnier: Safari gets the login page's HTML in its place, and the icon on the Home Screen may well end up named "Sign In." Both files are public and contain no personal data, so serve them as static files, bypassing the app's router. In the Network tab this takes five seconds to check. If a text/html document arrived in place of the manifest, the installed app will take its name from the page's <title>. First Launch From the Home Screen Icon on iPhone The first visit happens entirely in a Safari tab. The page arrives over HTTPS, Safari downloads the manifest along with it, the script registers the worker, and the worker puts the shell in the cache and activates according to your policy. Then the person adds the site to the Home Screen, and here many expect the app to take along everything the worker managed to store. It won't. A web app on the Home Screen has its own storage, separate from Safari's, with its own cookies, localStorage, IndexedDB, and Cache Storage, so someone who has just signed in within a tab will see the login screen again in the app. On first launch from the icon, the worker registers again, now inside the app, and if there's no connection at that moment, there's nothing to show. That’s the second cause of the broken first launch from the story at the start of the article: even with correct HTTPS, the first launch of an installed app has to happen online. On the second launch, the icon opens start_url, and the already-active worker intercepts requests within its scope. If the network is gone, a well-thought-out strategy returns the shell and clearly shows that the data is offline and how fresh it is. If someone put index.html in the cache a year ago under Cache First and forgot about it, the user gets an interface that looks like it works and is hopelessly out of date, and an honest "data is unavailable right now" page would have been better than that surprise. Storage is also subject to a quota on the user's phone. The popular rule "iOS wipes everything for a PWA after seven days of non-use" wrongly stretches to installed apps a restriction that WebKit introduced in 2020 for site storage in Safari. In the same post, WebKit wrote that web apps on the Home Screen have their own days-of-use counter and that, under normal use, their data shouldn't be deleted. The system can still evict data when space runs short, so sync critical data to the server, show queue status, and test a cold start after a long idle period. How to Test a PWA on a Real iPhone Test on a real device, and test the installed app specifically: a Safari tab and the icon on the Home Screen use different storage, so success in one tells you nothing about the other. I run the same scenario every time: install on a clean phone, open from the icon with the network on, turn on airplane mode, fully close the app (swipe it up in the app switcher), and open it again. Then do the same thing on top of an old version to see how the new worker comes up, and separately, sign out, after which no one else's data should remain in the cache. One successful page reload on a laptop proves absolutely nothing. For debugging on an iPhone, there's Web Inspector. On the phone, it's turned on in Safari's settings, under the advanced options, after which the device connects to the Mac with a cable and its pages appear in Safari's Develop menu on the Mac. The console, network, and storage are all there, but debugging workers is, in places, less pleasant than in Chrome DevTools. I don't rely on the inspector alone, and I log what I'll need later to analyze a bug report: the worker version, the result of cleaning up old caches, and the cases when the offline fallback kicked in. A line like console.info('[sw]', VERSION, 'activate') and the shell version number somewhere on a settings screen save hours of back-and-forth with a user who has no idea which version they have. There's one scenario where none of the above helps. A link to your app sent in a messenger will open not in the installed web app, and not necessarily in Safari, but in the messenger's built-in browser. It has its own storage and its own restrictions, so the cache you've built is unavailable there. Why the same address behaves differently on iPhone in Safari, in a built-in browser, and on the Home Screen, and what WebKit has to do with it, we'll cover in Part 3 of this series.

By Konstantin Shkurko