Coding Resources

DZone's Featured Coding Resources

Chaos Engineering and Kafka In Action: Topic Level Idempotence and Retry Check

Everyone is rushing to build AI agents right now, but GenAI integration really just puts a stronger spotlight on solid system design and backend reliability. An AI agent relies entirely on the APIs and data pipelines underneath it. If your backend drops events or double-processes payments during a network blip, even the best agent will just execute a broken workflow faster. To build resilient systems, you need to see how they break. Reading Kafka documentation on fault tolerance helps, but intentionally introducing network chaos shows how the underlying mechanics actually behave. In this post, we will set up a local testing environment using Spring Boot, Kafka, and Toxiproxy. We'll configure producer properties for topic-level idempotence and walk through common network simulation pitfalls that lead to duplicate data. 1. Setting Up the Chaos Test Architecture To test network issues accurately, stopping the Kafka broker isn't enough. We need to simulate realistic network conditions like latency spikes, packet drops, and bandwidth throttling. Toxiproxy acts as a proxy between our producer and broker to simulate these conditions. The Network Architecture Creating a Local/Dev Kafka Cluster We start by spinning up the Kafka cluster and Kafka UI using Docker. YAML # docker-compose-kafka.yml version: "3.8" #avoid this services: kafka: image: apache/kafka:latest container_name: kafka ports: - "9092:9092" environment: CLUSTER_ID: "4L62xdwCQKWvZs96A39vGA" KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: "broker,controller" KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093" KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL" # Configure distinct internal/external routing ports KAFKA_LISTENERS: "INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" # We will bookmark this line, dear reader! KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:19092" KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_LOG_DIRS: "/var/lib/kafka/data" volumes: - kafka_data:/var/lib/kafka/data kafka-ui: image: provectuslabs/kafka-ui:latest container_name: kafka-ui ports: - "9080:8080" depends_on: - kafka environment: DYNAMIC_CONFIG_ENABLED: "true" KAFKA_CLUSTERS_0_NAME: "local-cluster" KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:29092" volumes: kafka_data: To spin it up, we use Docker Compose: docker compose -f "docker-compose -kafka.yml" up Both your local Kafka cluster and the kafka-ui should spin up. Note that KAFKA_ADVERTISED_LISTENERS includes the external port localhost:19092; we'll see why. A Local Proxy to Sit Between Kafka Broker and Producer We use Docker for this, too! YAML # docker-compose-toxiproxy.yml version: "3.8" # Don’t use this version - it’s deprecated services: toxiproxy: image: shopify/toxiproxy:latest container_name: toxiproxy ports: - "8474:8474" # Toxiproxy Management API - "8000:8000" # Example port exposed for a proxied application service - "19092:19092" networks: - toxiproxy-net toxiproxy-ui: image: buckle/toxiproxy-frontend:latest container_name: toxiproxy-ui ports: - "8080:8080" # We don’t use UI in this article environment: - TOXIPROXY_URL=http://toxiproxy:8474 depends_on: - toxiproxy networks: - toxiproxy-net networks: toxiproxy-net: driver: bridge docker compose -f docker-compose-toxiproxy.yml up Once running, create a proxy tunnel via the Toxiproxy CLI to listen on port 19092 and forward traffic to Kafka on port 9092: ./toxiproxy-cli create -l 0.0.0.0:19092 -u host.docker.internal:9092 kafka-proxy Note: We use the above CLI inside the toxiproxy container. So, you must use docker exec. In some versions, the CLI tool might not be included in the $PATH, and you might need to switch the current directory: Shell /go/bin # ls toxiproxy toxiproxy-cli If you're using this toxiproxy tool for the first time, you can refer to the official Toxiproxy documentation or this starter guide we created. 2. The Kafka Client Producer Code We create a basic application using Spring Boot 4.1 (though we do not have any version restriction until now) and use the following dependency: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-kafka</artifactId> </dependency> We'll keep the Spring Boot service simple. We define a basic payload configuration and direct messages to topic T3 via KafkaConfig.java. Config: Java // KafkaConfig.java package com.soham.selfteaching.kafka.kafkadeepdive.config; import lombok.Data; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Configuration; @Configuration @Data public class KafkaConfig { @Value("${app.topic.name:T3}") private String topic; } Our application.yml is minimal, accepting defaults provided by the Kafka library. Properties files # application.yml server: port: 8082 spring: application: name: kafka-deep-dive kafka: bootstrap-servers: "localhost:19092" producer: acks: "1" # Required to bypass idempotence shield key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: request.timeout.ms: 3000 delivery.timeout.ms: 120000 enable.idempotence: false # We disable it first before we explore producer level idempotence The domain event is defined as a Java record. Java // EventDto.java package com.soham.selfteaching.kafka.kafkadeepdive.dto; public record EventDto(String id,String payload) { } Producer Code The core logic sits in ProducerService.java. We use CompletableFuture to handle delivery timeouts asynchronously without blocking thread execution. If the broker is unreachable initially, max.block.ms handles the immediate block on the main thread while background retries handle transient issues. Java // ProducerService.java package com.soham.selfteaching.kafka.kafkadeepdive.service; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.springframework.kafka.core.KafkaTemplate; import org.springframework.kafka.support.SendResult; import org.springframework.stereotype.Service; import java.util.concurrent.CompletableFuture; @Service @Slf4j @RequiredArgsConstructor public class ProducerService { private final KafkaTemplate<String, Object> kafkaTemplate; public void sendMessage(String topic, String key, Object payload) { log.info("Attempting to send message to topic: {} with key: {}", topic, key); CompletableFuture<SendResult<String, Object>> future = kafkaTemplate.send(topic, key, payload); future.whenComplete((result, exception) -> { if (exception == null) { log.info("Message successfully delivered. Topic: {}, Partition: {}, Offset: {}", result.getRecordMetadata().topic(), result.getRecordMetadata().partition(), result.getRecordMetadata().offset()); } else { log.error("Fatal delivery failure for key {}. Exhausted delivery.timeout.ms.", key, exception); } }); } } 3. Configuring Topic-Level Idempotence During a network timeout, the producer cannot verify whether the message failed to reach the broker or if the acknowledgment (ACK) was lost on the return path. By default, the producer retries the request, which can produce duplicate records without proper safeguards. Exactly-Once Semantics (EOS) and topic-level idempotence address this by assigning a producer ID (PID) and sequence number to each message batch. If the broker receives a retried batch with a sequence number it has already processed, it ignores the duplicate write. Disabling Idempotence to Force Duplicates To demonstrate retry behavior, we can explicitly disable these safety settings in application.yml. Since Kafka 3.0, enable.idempotence defaults to true. To allow duplicates during testing, set acks: "1" and explicitly set enable.idempotence: false. Setting request.timeout.ms to 3000 (3 seconds) creates a tight window to trigger network retries during latency testing. 4. Running the Kafka Producer and Verify We run the Spring Boot application: Java @SpringBootApplication public class KafkaDeepDiveApplication { public static void main(String[] args) { SpringApplication.run(KafkaDeepDiveApplication.class, args); } } And hit a REST API (Hold on! It's a simple HTTP API which may NOT quite follow REST paradigms). curl -X GET http://localhost:8082/api/send Java @RestController @RequestMapping("/api") @Slf4j @RequiredArgsConstructor public class ProducerController { private final ProducerService service; private final KafkaConfig kafkaConfig; @GetMapping("/send") public String send() { long ts = System.currentTimeMillis(); Object payload = new EventDto("p1.t1." + UUID.randomUUID().toString(), ("Hello message " + i)); service.sendMessage(kafkaConfig.getTopic(), "k1", payload); return "Process took :%s ms".formatted(System.currentTimeMillis() - ts); } } In the case when idempotence is disabled, it's fair to expect that duplicate messages will arrive on a topic if a producer does not receive timely ACK from the broker leaders in alignment with the ack flag. But we do not yet see any duplicate messages at all. This is fairly explainable, since you're all set up locally, and there's usually no lost acknowledgment. Introduce Chaos to the Local Network Add 5000ms latency to the proxy. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy Send a message to the endpoint. After an API hit, wait for 2–3 seconds and remove the toxic Remove the Toxic While the Retry Is in Flight Instantly run this command to remove the toxic while the retry is in flight: ./toxiproxy-cli toxic remove -n false_positive kafka-proxy While this should have produced duplicates due to retries by the producer following a timed-out broker ACK, we find NO DUPLICATE MESSAGES even after repeating the process above many times. 5. Let Us Debug Our Disconnect Common Pitfalls During Network Injection Simply starting the proxy and adding latency doesn't always trigger duplicates as expected. Below are two common issues related to how Kafka clients handle metadata and connections. Issue 1: Advertised Listeners Bypass If Toxiproxy is running on port 19092 and a 5000ms latency is emulated, messages sent through the proxy might still complete in milliseconds. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy This occurs because of how Kafka advertises broker addresses during metadata discovery. If you look closely at the internal Kafka NetworkClient logs during startup, you will find out why: Plain Text brokers=[MetadataResponseBroker(nodeId=1, host='localhost', port=9092)] When the producer connects to port 19092 for initial cluster metadata, the broker returns its configured advertised listener (e.g., localhost:9092). The producer client caches this response and routes subsequent produce requests directly to port 9092, bypassing the proxy entirely. We must ensure that our KAFKA_ADVERTISED_LISTENERS explicitly broadcasts EXTERNAL://localhost:19092 so the producer continues routing traffic through Toxiproxy. If updated, clear existing Docker volumes (docker-compose down -v) to avoid using cached listener configurations. Issue 2: Initial Metadata Fetch Blocking If high latency (e.g., 15,000ms) is applied before sending any messages, the application may time out after 60 seconds with a TimeoutException: Topic T3 not present in metadata, rather than failing on delivery.timeout.ms. The delivery.timeout.ms timer only applies once a record enters the producer's accumulator buffer. If the client hasn't fetched topic metadata yet, the call blocks on max.block.ms (defaulting to 60 seconds) while attempting to retrieve metadata through the degraded network path. We apply the fix: Send an initial message without active toxics to populate the local metadata cache before injecting network latency. Finally With latency removed, the retried request reaches the broker successfully. Because acks=1 and enable.idempotence=false, the broker writes the payload again without verifying duplicate sequence numbers. Checking Kafka UI confirms two identical records are present in the topic partition. Simulating network edge cases directly demonstrates why configurations like advertised listeners, metadata timeouts, and idempotence defaults exist in modern Kafka clients. Now we get duplicate messages across a topic partition. Duplicate Messages We can now enable back idempotence flag to see that no duplicate messages are created repeating the above steps with enable.idempotence:true, ensuring that idempotence is preserved enable.idempotence= true only ensures idempotence across a partition only and not the entire topic. We shall explore it in a next article. More

Full-Text Search Meets the Graph: Apache Lucene Inside Neo4j

By Akmal Chaudhri

CORE

We have documents that need full-text search, but those documents also have meaningful relationships. Do we split search and graph traversal across two systems or keep them together? Most developers reach for Elasticsearch or a dedicated search service when they need full-text search. It's a reasonable reaction, but it introduces an extra service to operate, an extra data pipeline to maintain, and an extra hop in every query. What's easy to miss is that Neo4j ships with Apache Lucene built in — the same underlying search library on which Elasticsearch is built — and when our data are already in a graph, adding full-text search on top is a single Cypher command! More importantly, full-text search in Neo4j can do something a standalone search engine can't: it serves as an entry point into the graph. We can find our initial results with Lucene, then traverse relationships to discover what surrounds them. That two-phase pattern — search-then-traverse — is what this article describes. We'll build a research paper explorer using a generated dataset of 200 papers with titles, abstracts, authors, fields, venues, and citation relationships. The same papers appear in the Weaviate chapter of Seven Vector Databases in Seven Days, where we used hybrid vector search to find them. Here we use Neo4j's full-text search instead, and we add something the Weaviate version didn't have: a citation graph that we can traverse after search. The comparison is instructive — same papers, similar queries, but a very different graph layer. The full source code is available on GitHub. Prerequisites AuraDB is Neo4j's fully managed cloud database. A free tier is available with no credit card required. Sign up at Get Started for Free.Create a new AuraDB Free instance.When the instance is created, download or note the credentials — the connection URI, username, and password.Once the instance is running, open the Query tab, and connect to the instance.Confirm it's empty with MATCH (n) RETURN count(n) which should return 0. The code runs in a Jupyter notebook with Python 3.12 or later. Before starting Jupyter, export the connection details as environment variables in your shell: Shell export NEO4J_URI="neo4j+s://xxxx.databases.neo4j.io" export NEO4J_USERNAME="your-username" export NEO4J_PASSWORD="your-password" One version note. This article queries the full-text index with Cypher's SEARCH clause, which replaces the older db.index.fulltext.queryNodes() procedure. The documentation lists SEARCH on full-text indexes as available from Neo4j 2026.09 and it requires Cypher 25. Our AuraDB Free instance accepted it, so the most reliable check is simply to try the clause, which the notebook does. On instances that don't support it, the procedure does the same job but now produces a deprecation warning. The Graph Model The dataset has three node labels and two relationship types: Plain Text (:Author)-[:AUTHORED]->(:Paper)-[:CITES]->(:Paper) Paper nodes carry the properties we'll search and filter on: title, abstract, field, year, venue and citation_count. Author nodes have a single property: name. The AUTHORED and CITES relationships carry no properties — all the meaning is in the graph structure. A conventional search index doesn't give us Neo4j's native graph model and Cypher traversal over relationships. A paper's citation neighborhood, the other work its authors have published, the papers that are frequently cited alongside it — that structure lives in the relationships, and it's what we'll traverse after search. Generating the Dataset We'll generate 200 papers from pools of fields, venues, author names, methods, datasets, and abstract templates, using a fixed random seed so the results are reproducible. The generation code is adapted directly from the Weaviate chapter, so the papers are the same across both articles. Python random.seed(RANDOM_SEED) papers = [generate_paper(i) for i in range(NUM_PAPERS)] Each paper gets between two and five authors, drawn from pools of first and last names. We add two things the Weaviate version didn't have: author nodes and citation edges. Citation edges are generated with three rules that keep the graph plausible: A paper can only cite papers with a lower paper_id and a publication year no later than its own, so nothing cites the futureCitations are biased toward papers in the same or adjacent fields, using a weighted random selectionEach paper makes between three and eight citation attempts, and duplicates are removed Python weights = [ 3 if c["field"] == paper["field"] else 2 if c["field"] in adjacent else 1 for c in candidates ] This gives us 937 citation edges that cluster by field, which makes the traversal results more interesting. Because the relationships are synthetic, the traversal results demonstrate the mechanics of graph-based discovery rather than real bibliometric relationships. Loading Into Neo4j Before loading, the notebook clears any existing data — dropping constraints and indexes first, then deleting nodes in batches of 1,000 to avoid transaction timeouts on Aura. It then creates uniqueness constraints on Paper.paper_id and Author.name, which keep authors unique and make the lookups in the loading queries fast: Cypher CREATE CONSTRAINT paper_id IF NOT EXISTS FOR (p:Paper) REQUIRE p.paper_id IS UNIQUE; CREATE CONSTRAINT author_name IF NOT EXISTS FOR (a:Author) REQUIRE a.name IS UNIQUE; We'll load the data in three passes using UNWIND to batch the inserts. First, the papers: Cypher UNWIND $records AS r CREATE (p:Paper { paper_id: r.paper_id, title: r.title, year: r.year, field: r.field, venue: r.venue, citation_count: r.citation_count, abstract: r.abstract }) For authors, we use MERGE rather than CREATE, since the same author can appear on multiple papers and we want a single node per author: Cypher UNWIND $records AS r MERGE (a:Author {name: r.author_name}) WITH a, r MATCH (p:Paper {paper_id: r.paper_id}) CREATE (a)-[:AUTHORED]->(p) Finally, the citations: Cypher UNWIND $citations AS c MATCH (from:Paper {paper_id: c.from_id}) MATCH (to:Paper {paper_id: c.to_id}) CREATE (from)-[:CITES]->(to) After loading, the graph has 200 Paper nodes, 215 unique Author nodes, 691 AUTHORED relationships and 937 CITES relationships. One caveat about the authors: names are drawn from 15 first names and 15 last names, so there are only 225 possible combinations. "The same author" here just means "the same generated name". Real bibliographic data needs author disambiguation, which is a hard problem in its own right. Creating the Full-Text Index We'll create a single Lucene index across both the title and abstract properties of Paper nodes: Cypher CREATE FULLTEXT INDEX paper_fulltext IF NOT EXISTS FOR (p:Paper) ON EACH [p.title, p.abstract] OPTIONS { indexConfig: { `fulltext.analyzer`: 'standard' } } A single index across multiple properties means a single query scores both together — no merging of separate result sets in application code. The OPTIONS line chooses the analyzer, the component that splits text into searchable terms. By default Neo4j uses standard-no-stop-words, which indexes every word, including "and", "for," and "the." That makes long natural-language queries noisy, because a paper can pick up a score just by containing "and." The standard analyzer drops common English stop words but doesn't stem, so fuzzy matching, which we'll use shortly, still works on the original terms. The index is created asynchronously, so we'll poll until it's ONLINE before querying: Cypher SHOW INDEXES YIELD name, state WHERE name = 'paper_fulltext' RETURN state Basic Full-Text Search We'll query the index with the SEARCH clause, which returns matching nodes alongside a Lucene BM25 relevance score: Cypher CYPHER 25 MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 5 ) SCORE AS score WITH p, score RETURN p.title, p.field, p.venue, p.year, p.citation_count, score ORDER BY score DESC Two details are worth knowing. First, LIMIT is part of the clause: it sets how many top-scoring matches the index returns. Second, the CYPHER 25 prefix selects the Cypher version that provides SEARCH. We show it once here and omit it from the snippets that follow, although the notebook includes it in every query. The notebook builds this text with a small helper that fills in the limit, so every example below reuses the same clause. The score is based on BM25, which considers term frequency, inverse document frequency, and document length. A paper whose title and abstract both mention "transformer models" will score higher than one where the term appears only once in the abstract. Python search_papers("transformer models for natural language understanding") Example output: Plain Text Efficient Transformer Models for Natural Language Processing Natural Language Processing | Symposium on Knowledge Discovery and Data Mining | 2022 | Citations: 18 Score: 9.926 Efficient Transformer Models for Natural Language Processing Natural Language Processing | Transactions on Machine Learning and Data Mining | 2020 | Citations: 196 Score: 9.741 ... The generated titles aren't unique, so the same title can appear twice. They are different papers, which the year and venue make clear. Special Characters in User Input Lucene treats characters such as + - ! ( ) { } [ ] ^ " ~ * ? : \ / as query syntax. If the query text comes from a user, an unbalanced bracket or a stray colon will raise a parse error or quietly change the meaning of the query. Unless we deliberately want operators like phrases, ~ and boosts to work, we should escape the text first: Python def escape_lucene(text: str) -> str: """Escape Lucene query-syntax characters in raw user input.""" return re.sub(r'([+\-!(){}\[\]^"~*?:\\/&|])', r'\\\1', text) search_papers("stream processing (survey") # raises a parse error search_papers(escape_lucene("stream processing (survey")) # works The unescaped query fails with a message that Neo4j could not parse the full-text search string. The escaped one returns the stream processing papers. Boosting Titles Over Abstracts Because the index covers both properties, we can address them by name in the query and weight them. Here a match in the title counts three times as much as a match in the abstract: Python search_papers("title:transformer^3 OR abstract:transformer", top_k=5) The Index Stays in Sync One of the costs of a separate search service is the pipeline that keeps it in sync with the primary database. Here the full-text index is updated as part of the same transaction that writes the node. To see it, we create a paper containing a unique word, search for it straight away, and then delete it: Python with driver.session() as session: session.run(""" CREATE (:Paper {paper_id: 9999, title: 'Quokka Navigation for Robotics', abstract: 'A throwaway paper used to show the index updates on commit.', year: 2024, field: 'Robotics', venue: 'Demo', citation_count: 0}) """) search_papers("quokka", top_k=1) # found immediately with driver.session() as session: session.run("MATCH (p:Paper {paper_id: 9999}) DETACH DELETE p") search_papers("quokka", top_k=1) # gone again The first search returns the new paper (score 2.815) and the second returns nothing. Phrase Search Wrapping a query in double quotes tells Lucene to treat it as a phrase — the words must appear together in that order: Python search_papers('"graph neural networks"', top_k=3) Without quotes, Lucene scores documents that contain any of the words, awarding higher scores for more matches. With quotes, every result must contain the exact phrase. Score Comparison: Keyword vs. Phrase Search The chart below (Figure 1) runs the same query twice, once as keywords and once as a phrase, and plots the scores of the top eight results of each. A bar of zero means the paper is outside that mode's top eight, not that it fails to match. Figure 1. Keyword vs. Phrase Search. The result is worth reading carefully. In this dataset, every paper in the Graph Neural Networks field contains the exact phrase, because the field name appears in its title and abstract. So both queries match essentially the same papers and their scores are close: 6.1 to 6.8 for keywords and 5.4 to 5.65 for the phrase. What changes is which papers make the top eight. Only three of the eight appear in both lists, and the top result differs: keyword search ranks "Efficient Graph Attention Networks for Graph Neural Networks" first (6.774), while phrase search ranks "Efficient Link Prediction for Graph Neural Networks" first (5.649). A plausible reason is that keyword scoring rewards papers that repeat the individual words, such as "graph" and "networks" in a title that also mentions graph attention networks, while phrase scoring only counts the phrase itself. Raw scores aren't directly comparable across the two query forms, so membership and order are the things to compare. On noisier text, where the words in a query don't usually sit together, the phrase search would also drop papers entirely, which is the more familiar reason to use it. Fuzzy Search Appending ~ to a term enables Lucene's fuzzy matching, which finds terms within a given edit distance. A misspelled query like tranformer~ correctly matches transformer: Python search_papers("tranformer~", top_k=3) Example output: Plain Text Efficient Transformer Models for Natural Language Processing Natural Language Processing | Transactions on Machine Learning and Data Mining | 2020 | Citations: 196 Score: 3.015 ... Without the ~, the misspelled query returns nothing. With it, Lucene finds the closest matching terms and scores them. This is useful for any field where exact spelling can't be guaranteed, such as technical terms or partially remembered titles. Filtered Search We can combine the full-text index with filters on structured node properties. There is one thing to be aware of: for full-text indexes, SEARCH doesn't allow a WHERE inside the clause. Filtering is a second step. The index first returns its top-scoring candidates, then a WHERE after it filters them by field, year or citation count: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 200 ) SCORE AS score WITH p, score WHERE p.field = $field AND p.year >= $min_year AND p.citation_count >= $min_citations RETURN p.title, p.field, p.venue, p.year, p.citation_count, score ORDER BY score DESC LIMIT $top_k The two limits do different jobs. The LIMIT 200 inside SEARCH sets the size of the candidate pool, here every paper in the collection, and the final LIMIT $top_k trims the filtered results. If we asked the index for only the top five and filtered afterward, a selective filter could leave us with nothing. On a large collection, we would choose a pool big enough for our filters to leave enough results, without asking the index to score everything. These are the same three filtered queries used in the Weaviate chapter. Weaviate ran them as hybrid searches, combining vectors and keywords, whereas this is keyword-only BM25, so we should expect overlap in the results rather than identical rankings: Python # NLP papers from 2020 onwards with at least 100 citations search_papers_filtered( "attention mechanisms and transformer architectures", field = "Natural Language Processing", min_year = 2020, min_citations = 100, ) # Recent computer vision papers search_papers_filtered( "object detection in real time", field = "Computer Vision", min_year = 2022, ) # Highly cited papers across all fields search_papers_filtered( "self-supervised learning for representation learning", min_citations = 200, ) The first query returns two papers (scores 3.350 and 3.154) and the second returns one (7.250). The third returns five: the three self-supervised learning papers score between 10.273 and 9.537, then the scores drop to 3.543 for the fourth paper, which matches only more generic terms such as "learning". With the default analyzer, an earlier run of the first query also returned papers scoring below 0.1, apparently matching only on common words such as "and". With stop words removed, that tail no longer appears. Search Then Traverse: The Graph Layer Full-text search answers "Which documents match?" Graph traversal answers "What is connected to those matches?" This section is where the two work together. We'll use the Lucene results as an entry point into the graph, then traverse CITES and AUTHORED relationships to discover related content. The pattern is always the same: SEARCH finds the initial set, then Cypher traverses from there. In a synthetic graph, some papers have few connections, so the notebook runs ten candidate queries and measures each top result's neighborhood: how many papers it cites, how many cite it, and how strongly it is co-cited with others. It picks the richest as the demonstration query. That query is "protein structure prediction", whose top result, "Efficient Protein Structure Prediction for Bioinformatics" (score 12.926), cites 3 papers and is cited by 17. What Does a Top Result Cite? Starting from the highest-scoring paper, we'll follow outgoing CITES edges to see what it builds on. Note that citation_count is a property generated independently for each paper — it isn't derived from the CITES relationships in this graph. We use it here as a proxy for importance when ordering results. Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (p)-[:CITES]->(cited:Paper) RETURN p.title AS source_title, score AS source_score, cited.title, cited.field, cited.year, cited.citation_count ORDER BY cited.citation_count DESC LIMIT $top_k Example output: Plain Text Drug Discovery for Bioinformatics: A Scalable Approach Bioinformatics | 2016 | Citations: 483 Improving Bioinformatics with Drug Discovery Bioinformatics | 2016 | Citations: 467 Cryptographic Protocols in Cybersecurity: Challenges and Opportunities Cybersecurity | 2016 | Citations: 283 What Cites a Top Result? Traversing in the opposite direction — incoming CITES edges — finds papers that build on a highly relevant result: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (citing:Paper)-[:CITES]->(p) RETURN p.title AS source_title, score AS source_score, citing.title, citing.field, citing.year, citing.citation_count ORDER BY citing.year DESC LIMIT $top_k The most recent citing papers include "A Unified Framework for Reinforcement Learning using Hierarchical Rl" (2024), "Efficient Malware Classification for Cybersecurity" (2024), and "Meta-Learning for Machine Learning: A Scalable Approach" (2024). Citations reach across fields because the generator only biases them toward related fields; it doesn't confine them. Visualizing the Citation Network We can visualize the citation neighborhood of the top result as an interactive graph with neo4j-viz. In Figure 2, the central red node is the top-scoring paper, blue nodes are papers it cites, and green nodes are papers that cite it. Arrows point from the citing paper to the cited paper, and node size reflects citation count. In the notebook, hovering over a node shows the full title, year, and citation count. Python # One Cypher query returns the top result with the papers it cites and the papers that cite it nodes, relationships = {}, [] nodes[top["id"]] = Node(id=top["id"], caption=short(top["title"]), properties={"role": "Top result", "citations": top["citations"]}) for p in top["cited"]: nodes[p["id"]] = Node(id=p["id"], caption=short(p["title"]), properties={"role": "Cites", "citations": p["citations"]}) relationships.append(Relationship(source=top["id"], target=p["id"])) for p in top["citing"]: nodes[p["id"]] = Node(id=p["id"], caption=short(p["title"]), properties={"role": "Cited by", "citations": p["citations"]}) relationships.append(Relationship(source=p["id"], target=top["id"])) VG = VisualizationGraph(nodes=list(nodes.values()), relationships=relationships) VG.color_nodes(property="role", colors={ "Top result": "#EF553B", "Cites": "#636EFA", "Cited by": "#00CC96"}) VG.resize_nodes(property="citations", node_radius_min_max=(14, 40)) VG.render(height="520px") Figure 2. Citation Network. This kind of neighborhood visualization isn't something we'd get from a search engine, as it requires the graph. Find Other Work by the Same Authors From the top result, we'll traverse AUTHORED edges to the authors, then follow their other AUTHORED edges to find their other papers: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (a:Author)-[:AUTHORED]->(p) MATCH (a)-[:AUTHORED]->(other:Paper) WHERE other <> p RETURN p.title AS source_title, score AS source_score, a.name AS author, other.title, other.field, other.year, other.citation_count ORDER BY other.citation_count DESC LIMIT $top_k The highest-cited results are papers by Chen Singh and Lena Li, two of the authors of the top result, including "Improving Computer Vision with Generative Adversarial Networks" (475 citations) and "Efficient Multi-Agent Systems for Reinforcement Learning" (433). This is the kind of discovery path that's natural in a graph but would typically require additional application-side queries or data modeling outside the search index. Co-Citation Discovery Two papers are co-cited when a third paper cites both of them. Frequently co-cited papers can indicate related research topics, shared foundations, or intellectual proximity — a relationship that doesn't appear in the papers themselves, only in how others use them: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (citing:Paper)-[:CITES]->(p) MATCH (citing)-[:CITES]->(cocited:Paper) WHERE cocited <> p WITH p, score, cocited, count(citing) AS shared_citations RETURN p.title AS source_title, score AS source_score, cocited.title, cocited.field, cocited.year, shared_citations ORDER BY shared_citations DESC LIMIT $top_k For our top result, the most co-cited paper is "Drug Discovery for Bioinformatics: A Scalable Approach" with 6 shared citing papers, followed by "Rethinking Robotics with Imitation Learning" with 5, and three papers with 4 each. The Robotics paper is a reminder that the generator's field bias is only a bias. Co-citation analysis is a classical technique in bibliometrics, and here it is a single Cypher query. Traversing From More Than One Result So far we've started from the single best hit. The same pattern works from the top N results, and aggregation then surfaces something no single result shows: the papers that several of the best matches have in common. Here we ask which papers are cited by the most of the top five results, a rough proxy for foundational work on the topic: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 5 ) SCORE AS score WITH p, score MATCH (p)-[:CITES]->(ref:Paper) RETURN ref.title AS title, ref.field AS field, ref.year AS year, ref.citation_count AS citations, count(DISTINCT p) AS cited_by_hits ORDER BY cited_by_hits DESC, citations DESC LIMIT $top_k Two papers, "Drug Discovery for Bioinformatics: A Scalable Approach" and "Improving Bioinformatics with Drug Discovery," are each cited by two of the five best matches, ahead of papers cited by only one. A standalone search engine would return the five matches and stop there. When to Use This Neo4j's full-text search is worth reaching for when our data are already in a graph, and we want to add keyword search without operating a separate service. The setup is minimal: one Cypher command creates the index, one clause queries it, and we immediately get phrase matching, fuzzy matching, field boosting, and BM25 relevance scoring. Because the index is updated in the same transaction as the data, there is no synchronization pipeline to build or monitor. The search-then-traverse pattern is the real differentiator. If our data have relationships, such as citations, authorship, product categories, or social connections, and we want search to serve as an entry point into those relationships, Neo4j gives us both in one place. It is not the right choice for every workload. A dedicated search engine offers capabilities that weren't part of what we used here, such as faceted aggregations and result highlighting, and it lets us scale search independently of the database. If search traffic is large, if it needs different hardware from the graph workload, or if the data have no meaningful relationships, a separate service may be the better fit. The SEARCH behavior described above also matters at scale: the index returns a bounded candidate set, and filters apply afterward, so queries with very selective filters need a candidate pool sized for them. For purely semantic workloads where we want vector similarity rather than keyword matching, Neo4j also supports vector indexes, which we can combine with full-text search in the same query. But for the common case where keyword search and graph traversal belong together, the built-in Lucene integration is often all we need. Summary We've seen how to create a multi-property full-text index in Neo4j, query it with the SEARCH clause using keyword, phrase, and fuzzy search, escape user input and boost fields, combine the results with structured property filters, and use them as the entry point for graph traversals that follow citations, authorship, co-citation, and multiple matches at once. Further examples of Neo4j's multi-model capabilities can be found in the free online book Neo4j Beyond the Graph. The full source code is available on GitHub. More

Constrained Decoding Models Explained: Encoders, Decoders and When to Use Them

By Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

A Software Factory on Kubernetes: Gemini Builds, jev Decides (Genkit + senro in Go)

By Xavier Portilla Edo

CORE

Beyond Manual Mapping: AI-Orchestrated Governance for Hybrid Cloud

By Ramesh Bellamkonda

Top Gen AI Frameworks for Go in 2026: A Hands-On Comparison

Go has quietly become the language of AI infrastructure. MCP servers, agent runtimes, model gateways, Kubernetes operators that talk to LLMs: a large share of that plumbing is written in Go, because Go is what you reach for when something has to be a small static binary that handles thousands of concurrent connections without drama. What was missing for a long time was the framework layer on top of the raw provider SDKs. In 2026, that gap is closed, and the question is no longer "Go or Python?" but "which Go framework?" This article covers the five frameworks a Go developer will actually run into in 2026: Genkit Go, Eino, Google ADK Go, tRPC-Agent-Go, and LangChainGo. Four of them are alive and moving fast. The fifth, LangChainGo, is the one most Go developers heard of first, and it is now effectively unmaintained. It stays in the list because you will find it in existing codebases, and you need to know what that means for you. Same rules as the Java and JavaScript/TypeScript editions: this is not a benchmark post. Tokens per second go stale in weeks. This is a developer experience and architecture comparison, with working code and the real output of that code for every framework, all against the same model, gemini-3.8-flash, on Go 1.27. Versions used: Genkit Go v1.13.1, Eino v0.9.20, Google ADK Go v2.4.0, tRPC-Agent-Go v1.11.2, LangChainGo v0.1.14. APIs as of September 2026. Genkit Go History and Direction Genkit was announced by Google at I/O 2024 as a TypeScript-first framework. The Go SDK arrived only three months later, with v0.1.0 in August 2024, and unlike the Java SDK, which is a community effort, Genkit Go is an official, first-party SDK maintained by the same Google team that ships the TypeScript runtime. It reached 1.0 in September 2025 and sits at v1.13.1 as of September 2026, on a cadence of roughly one minor release per month. The direction is parity with TypeScript plus Go-native ergonomics: generics instead of Zod schemas, context.Context everywhere, plain net/http handlers for flows, and an agents package, still marked experimental, with session stores, multi-agent delegation, artifacts, and a Skills middleware. The monorepo (genkit-ai/genkit) has 6.4k stars and 80 commits in the last 30 days across all SDKs. If you want the long-form introduction, start with Stop Using Python for Your Gen AI Apps, Use Go and Genkit Instead. What Makes Genkit Go Stand Out Genkit Go gives you three levels of abstraction in a single SDK: direct model calls, typed flows, and agents, all traced by the same OpenTelemetry pipeline and all visible in the same Dev UI. On top of that sits a middleware layer (retry, fallback, tool approval, filesystem, skills) that wraps any of the three. No other Go framework gives you that combination. Supported languages: Go 1.25+ for this SDK. The same framework exists for TypeScript, Python, Dart/Flutter, and, unofficially, Java. Vanilla Generation Go import ( "github.com/genkit-ai/genkit/go/ai" "github.com/genkit-ai/genkit/go/genkit" "github.com/genkit-ai/genkit/go/plugins/googlegenai" ) g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}), genkit.WithDefaultModel("googleai/gemini-3.8-flash")) text, err := genkit.GenerateText(ctx, g, ai.WithPrompt("Explain the CAP theorem in two sentences.")) Plain Text The CAP theorem states that a distributed data system can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Because network partitions are inevitable in the real world, systems must ultimately choose between returning the most up-to-date data (Consistency) or ensuring every request receives a response (Availability). Typed Flows: Observable Pipelines Flows are the heart of Genkit. A flow is a named, typed, traceable unit that you can call in-process, expose as an HTTP endpoint with one line, and run from the Dev UI. The JSON schema for the output comes from the Go struct, and GenerateData returns the struct, not a string you have to parse. Go type TranslateRequest struct { Text string `json:"text"` TargetLanguage string `json:"target_language"` } type TranslateResponse struct { Translation string `json:"translation"` DetectedLanguage string `json:"detected_language"` } translate := genkit.DefineFlow(g, "translateText", func(ctx context.Context, req TranslateRequest) (*TranslateResponse, error) { out, _, err := genkit.GenerateData[TranslateResponse](ctx, g, ai.WithPrompt("Translate %q to %s and detect the source language.", req.Text, req.TargetLanguage)) return out, err }) // Call it in-process... out, err := translate.Run(ctx, TranslateRequest{Text: "Buenos días", TargetLanguage: "English"}) fmt.Printf("%+v\n", *out) // ...or expose it over HTTP (and to the Dev UI). mux := http.NewServeMux() mux.HandleFunc("POST /translateText", genkit.Handler(translate)) log.Fatal(server.Start(ctx, "127.0.0.1:3400", mux)) Plain Text {Translation:Good morning DetectedLanguage:Spanish} INFO server listening addr=127.0.0.1:3400 Shell curl -X POST http://127.0.0.1:3400/translateText \ -H 'Content-Type: application/json' \ -d '{"data":{"text":"Buenos días","target_language":"English"}' # {"result":{"translation":"Good morning","detected_language":"Spanish"} Tools and Agents Tools are typed functions. The input schema is inferred from the struct, and the tool loop (model asks, tool runs, model answers) happens inside Generate. Go type WeatherInput struct { City string `json:"city"` } weather := genkit.DefineTool(g, "getWeather", "Returns the current weather for a city.", func(ctx *ai.ToolContext, in WeatherInput) (string, error) { return fmt.Sprintf("Sunny, 24C in %s", in.City), nil }) resp, err := genkit.Generate(ctx, g, ai.WithPrompt("What's the weather like in Tokyo?"), ai.WithTools(weather)) fmt.Println(resp.Text()) Plain Text The weather in Tokyo is currently sunny and 24°C. When you need memory across turns, the experimental agents package adds sessions on top of the same tools. Enable it with genkit.WithExperimental(): Go import ( aix "github.com/genkit-ai/genkit/go/ai/exp" "github.com/genkit-ai/genkit/go/ai/exp/localstore" genkitx "github.com/genkit-ai/genkit/go/genkit/exp" ) advisor := genkitx.DefineAgent(g, "travelAdvisor", aix.InlinePrompt{ ai.WithSystem("You are a helpful travel advisor. Use tools to give accurate advice. Be brief."), ai.WithTools(weather), }, aix.WithSessionStore(localstore.NewInMemorySessionStore[any]())) first, err := advisor.RunText(ctx, "Should I pack a jacket for my trip to Lisbon?") second, err := advisor.RunText(ctx, "Which city did I ask about?", aix.WithSessionID[any](first.SessionID)) Plain Text It is currently sunny and 24°C (75°F) in Lisbon. You won't need a heavy coat, but packing a light jacket or sweater is recommended for cooler evenings and coastal breezes. You asked about Lisbon. Multi-agent delegation, cross-vendor fallback, and tool approval are covered in Building Agents in Go: Where Genkit Shines vs ADK 2.0. The Dev UI: Same Power as TypeScript The same Genkit Developer UI that TypeScript developers use works with Go, with no code changes. Install the CLI and start your Go program through it: Shell npm install -g genkit genkit start -- go run . Plain Text Genkit Developer UI: http://localhost:4000 The flow above shows up as /flow/translateText next to every Gemini model the plugin registered, and you get: Flow runner: execute any flow with a custom input and inspect the typed output.Trace explorer: OpenTelemetry traces for every Generate and flow call, with latency, token counts, and the exact prompts.Model playground: call any registered model directly.Tool testing: run tools in isolation.Dotprompt editor: edit .prompt files live. This is the single biggest advantage Genkit has over every other framework in this list: a zero-config, local trace explorer that replaces Langfuse or LangSmith during development. Provider Support Official plugins in the Go module: Google AI (Gemini), Vertex AI, Anthropic, Ollama, and an OpenAI-compatible family (compat_oai) with ready-made packages for OpenAI, xAI, DeepSeek, OpenRouter, Kimi, DashScope, and Z.ai. Vector stores: localvec (local, no service needed), PostgreSQL/pgvector, AlloyDB, Pinecone, Weaviate, and Firebase Firestore. Plus MCP (client and server), evaluators, and an A2UI plugin for streaming UI components. Pros and Cons ✅ Pros❌ ConsBest-in-class Dev UI with local trace explorerAgents package still marked experimentalMulti-level abstractions: vanilla, flows, agentsFewer vector stores than LangChainGoMiddleware: retry, fallback, tool approvalGo 1.25+ requiredOfficial Google SDK, monthly releasesTyped output from Go structs, no parsingSame framework in TS, Python, Dart, Java Eino History and Direction Eino (pronounced "aino") comes from ByteDance's CloudWeGo team, the group behind Kitex and Hertz, the RPC and HTTP frameworks that run a large part of ByteDance's Go services. It was open-sourced in December 2024 after internal use, and it is now the most-starred LLM framework in Go at 13.1k stars, ahead of LangChainGo. Releases land every few days: v0.9.20 on September 20, 2026, with v0.10.0 already at alpha 35. Eino's own description is that it draws from LangChain and Google ADK but follows Go conventions. The framework splits into two: compose, for orchestration (Chain, Graph, Workflow), and adk, an Agent Development Kit that borrows Google ADK's vocabulary of agents, runners, and events. Components (models, tools, retrievers, indexers, loaders) live in a separate repo, eino-ext, together with callback handlers for Langfuse, LangSmith, CozeLoop, and ByteDance's APMPlus. The 2026 roadmap is all agents: DeepAgent (planning plus sub-agents), agent middleware, interrupt and resume for human-in-the-loop, skills, A2UI streaming components, and a turn loop with preemption. The English documentation on cloudwego.io is now a full eleven-chapter course, a big change from 2025 when most of it was Chinese-first. What Makes Eino Stand Out Orchestration with automatic stream handling. Every node in an Eino graph can be invoked or streamed, and the framework concatenates, boxes, merges, and copies streams as data moves between nodes. You build the graph once and get both Invoke and Stream for free. Graph inputs and outputs are generic types, so the compiler checks that the nodes fit together. Cross-cutting concerns go through callbacks at fixed points (OnStart, OnEnd, OnError, plus the streaming variants) that apply to components, graphs, and agents alike. Supported languages: Go only. The module declares go 1.18, the lowest bar in this list. One thing to know: Eino depends on ByteDance's sonic JSON library, and on Go 1.27 it prints a warning at startup and falls back to encoding/json. Harmless, but noisy. ChatModel Eino does not own the provider client. You create the genai client yourself and hand it to the component. Go import ( "github.com/cloudwego/eino-ext/components/model/gemini" "github.com/cloudwego/eino/schema" "google.golang.org/genai" ) client, err := genai.NewClient(ctx, &genai.ClientConfig{APIKey: os.Getenv("GEMINI_API_KEY")}) cm, err := gemini.NewChatModel(ctx, &gemini.Config{Client: client, Model: "gemini-3.8-flash"}) msg, err := cm.Generate(ctx, []*schema.Message{ schema.UserMessage("Explain the CAP theorem in two sentences."), }) fmt.Println(msg.Content) Plain Text The CAP theorem states that a distributed data store can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Because network partitions are practically inevitable in distributed systems, architects must choose between keeping data strictly up-to-date (Consistency) or ensuring the system remains responsive (Availability) during a network failure. Chains, Graphs, and Callbacks A chain is a linear pipeline. Here the callback handler is attached at invocation time and fires for the chain, the template, and the model. Go import ( "github.com/cloudwego/eino/callbacks" "github.com/cloudwego/eino/components/prompt" "github.com/cloudwego/eino/compose" ) template := prompt.FromMessages(schema.FString, schema.SystemMessage("You are a translator. Reply with the translation only."), schema.UserMessage("Translate {text} to {language}.")) chain := compose.NewChain[map[string]any, *schema.Message]() chain.AppendChatTemplate(template).AppendChatModel(cm) runnable, err := chain.Compile(ctx) tracer := callbacks.NewHandlerBuilder(). OnStartFn(func(ctx context.Context, info *callbacks.RunInfo, in callbacks.CallbackInput) context.Context { log.Printf("start %s %s", info.Component, info.Type) return ctx }). OnEndFn(func(ctx context.Context, info *callbacks.RunInfo, out callbacks.CallbackOutput) context.Context { log.Printf("end %s %s", info.Component, info.Type) return ctx }). Build() out, err := runnable.Invoke(ctx, map[string]any{"text": "Buenos días", "language": "English"}, compose.WithCallbacks(tracer)) fmt.Println(out.Content) Plain Text start Chain start ChatTemplate Default end ChatTemplate Default start ChatModel Gemini end ChatModel Gemini end Chain Good morning Graphs are where Eino earns its name. Nodes are typed, edges are explicit, and plain Go functions slot in as lambda nodes: Go graph := compose.NewGraph[map[string]any, string]() _ = graph.AddChatTemplateNode("template", template) _ = graph.AddChatModelNode("generate", cm) _ = graph.AddLambdaNode("shout", compose.InvokableLambda( func(ctx context.Context, msg *schema.Message) (string, error) { return strings.ToUpper(strings.TrimSpace(msg.Content)), nil })) _ = graph.AddEdge(compose.START, "template") _ = graph.AddEdge("template", "generate") _ = graph.AddEdge("generate", "shout") _ = graph.AddEdge("shout", compose.END) runnable, err := graph.Compile(ctx) out, err := runnable.Invoke(ctx, map[string]any{"product": "a Go framework for LLM apps"}) Plain Text BUILD PRODUCTION-GRADE LLM APPS AT GO SPEED. Branches, parallel fan-out, and sub-graphs use the same API, and a compiled graph can be wrapped as a tool and handed to an agent. Eino ADK: Agents and Workflows Tools are inferred from a Go struct with jsonschema tags. The agent runs the ReAct loop and emits one event per step, so you see the tool call, the tool result, and the final answer as separate messages. Go import ( "github.com/cloudwego/eino/adk" "github.com/cloudwego/eino/components/tool" "github.com/cloudwego/eino/components/tool/utils" ) type WeatherInput struct { City string `json:"city" jsonschema:"description=The city to look up"` } weather, err := utils.InferTool("get_weather", "Returns the current weather for a city.", func(ctx context.Context, in WeatherInput) (string, error) { return fmt.Sprintf("Sunny, 24C in %s", in.City), nil }) advisor, err := adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{ Name: "travel_advisor", Description: "Helps with trip planning and weather-based advice.", Instruction: "You are a helpful travel advisor. Use tools to give accurate advice. Be brief.", Model: cm, ToolsConfig: adk.ToolsConfig{ ToolsNodeConfig: compose.ToolsNodeConfig{Tools: []tool.BaseTool{weather}, }, }) runner := adk.NewRunner(ctx, adk.RunnerConfig{Agent: advisor}) iter := runner.Query(ctx, "Should I pack a jacket for my trip to Lisbon?") for { event, ok := iter.Next() if !ok { break } if event.Err != nil { log.Fatal(event.Err) } if event.Output == nil || event.Output.MessageOutput == nil { continue } msg, _ := event.Output.MessageOutput.GetMessage() fmt.Printf("[%s] %s\n", msg.Role, msg.Content) } Plain Text [assistant] [tool] Sunny, 24C in Lisbon [assistant] It is currently sunny and 24°C (75°F) in Lisbon. You won't need a heavy coat, but packing a light jacket or cardigan is a good idea for breezy coastal evenings. The first assistant event is the tool call itself, which is why its content is empty. Multi-agent workflows are one constructor away: Go pipeline, err := adk.NewSequentialAgent(ctx, &adk.SequentialAgentConfig{ Name: "content_pipeline", Description: "Research, then write.", SubAgents: []adk.Agent{researcher, writer}, }) Plain Text [researcher] 1. Created at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released in 2009. 2. Statically typed, compiled language with built-in garbage collection. 3. Features native, lightweight concurrency through goroutines and channels. [writer] Created at Google by Robert Griesemer, Rob Pike, and Ken Thompson, the Go programming language was released in 2009. It is a statically typed, compiled language that features built-in garbage collection and native, lightweight concurrency through goroutines and channels. NewParallelAgent and NewLoopAgent complete the set, and DeepAgent adds planning and delegation on top. Developer Tooling Eino's answer to observability during development is the Eino Dev plugin for GoLand and VS Code, backed by the devops module in eino-ext. It renders your chains and graphs visually inside the editor and lets you start execution from any node with mock input. It is an IDE feature rather than a browser Dev UI, so there is no trace explorer for a running service. For that you attach one of the callback handlers (Langfuse, LangSmith, CozeLoop, APMPlus). Provider Support Model components in eino-ext: OpenAI, Claude, Gemini, DeepSeek, Qwen, Ollama, OpenRouter, Ark (Volcano Engine), and Qianfan (Baidu), plus "agentic" variants of the main ones for the newer agentic message format. Retrievers and indexers: Elasticsearch (7, 8, 9), Milvus, Qdrant, Redis, OpenSearch, Dify, and VikingDB. Tools: MCP, Google Search, Bing, DuckDuckGo, SearXNG, Wikipedia, HTTP requests, command line, browser use, and sequential thinking. Pros and Cons ✅ Pros❌ ConsMost active independent Go LLM framework (13.1k stars)Still 0.x with frequent alphas; expect API churnTyped graph orchestration with automatic streamingNo browser Dev UI; visual debugging lives in the IDE pluginCallback aspects for logging, tracing, and metricsVerbose: config structs and explicit error checks everywhereADK-style agents plus DeepAgent and interrupt/resumeByteDance-flavoured corners (Ark, VikingDB, CozeLoop)Broad component ecosystem in eino-extsonic dependency warns on the newest Go toolchainsRuns on Go 1.18+Chinese-first history still shows in some docs and examples Google ADK Go History and Direction ADK, the Agent Development Kit, was announced at Google Cloud Next 2025 with Python as its first language. The Go SDK made its first tagged release, v0.1.0, in November 2025, went 1.0 on March 23, 2026, and shipped a new major module, google.golang.org/adk/v2, on June 30, 2026. It is at v2.4.0 (September 11, 2026), has 8.8k stars, and logged over 100 commits in the last 30 days: the highest commit volume in this list. It is an official Google product with the Python, Java, Kotlin, and TypeScript SDKs as siblings. The design is the same as its siblings: everything is an agent, a workflow agent, or a tool. Gemini is the native model, with openaimodel for anything speaking the OpenAI wire format and an Apigee model for enterprise gateways. Deployment gravity points at Vertex AI Agent Engine, Cloud Run, and GKE, with Pub/Sub and Eventarc triggers built into the launcher. The minimum Go version is 1.26.6, the highest bar of the five. A deeper look at the low-level differences with Genkit is in Genkit Go vs ADK Go 2.0. ADK Go's Position: Agent-Only, Enterprise-Grade ADK Go is an agent framework. There is no vanilla generation primitive and no flow abstraction: you always build an agent, give it to a runner, and iterate the events. In exchange, you get sessions, memory, artifacts, workflow agents, agent-to-agent (A2A) protocol support, MCP toolsets, tool confirmation, and an embedded web UI, all first-party. Supported languages: Go 1.26.6+. Same framework in Python (primary), Java, Kotlin, and TypeScript. Go import ( "google.golang.org/adk/v2/agent" "google.golang.org/adk/v2/agent/llmagent" "google.golang.org/adk/v2/model/gemini" "google.golang.org/adk/v2/runner" "google.golang.org/adk/v2/tool" "google.golang.org/adk/v2/tool/functiontool" "google.golang.org/genai" ) model, err := gemini.NewModel(ctx, "gemini-3.8-flash", &genai.ClientConfig{ APIKey: os.Getenv("GOOGLE_API_KEY"), }) weather, err := functiontool.New(functiontool.Config{ Name: "get_weather", Description: "Returns the current weather for a city.", }, func(_ agent.Context, in WeatherInput) (WeatherOutput, error) { return WeatherOutput{Report: fmt.Sprintf("Sunny, 24C in %s", in.City)}, nil }) advisor, err := llmagent.New(llmagent.Config{ Name: "travel_advisor", Model: model, Instruction: "You are a helpful travel advisor. Use tools to give accurate advice. Be brief.", Tools: []tool.Tool{weather}, }) r, err := runner.NewInMemory("travel-app", advisor) msg := genai.NewContentFromText("Should I pack a jacket for my trip to Lisbon?", genai.RoleUser) for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) { if err != nil { log.Fatal(err) } if event.LLMResponse.Content == nil || !event.IsFinalResponse() { continue } for _, part := range event.LLMResponse.Content.Parts { fmt.Print(part.Text) } } Plain Text It is currently sunny and warm at 24°C (75°F) in Lisbon. You won't need a heavy jacket during the day, but packing a light jacket or sweater is recommended for cooler evenings and coastal breezes. Two things to notice. The runner returns a Go 1.23 iterator, so you range over events with the error as the second value, which is the most idiomatic event loop of the five. And the function tool is generic: input and output structs give the schema; nothing is spelled out twice. Multi-Agent Orchestration Workflow agents compose LLM agents. The OutputKey writes an agent's answer into session state, and the next agent reads it back through a {placeholder} in its instruction. Go import "google.golang.org/adk/v2/agent/workflowagents/sequentialagent" researcher, err := llmagent.New(llmagent.Config{ Name: "researcher", Model: model, Instruction: "List three key facts about the user's topic. Be terse.", OutputKey: "facts", // saved into session state }) writer, err := llmagent.New(llmagent.Config{ Name: "writer", Model: model, Instruction: "Write a two-sentence brief using these facts:\n{facts}", }) pipeline, err := sequentialagent.New(sequentialagent.Config{ AgentConfig: agent.Config{ Name: "content_pipeline", SubAgents: []agent.Agent{researcher, writer}, }, }) Plain Text [researcher] * **Origin:** Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released as open source in 2009. * **Concurrency:** Features built-in, lightweight concurrency primitives known as goroutines and channels, managed by its own runtime. * **Ecosystem Role:** Statically typed and compiled, it is the primary language behind core cloud-native technologies, including Docker and Kubernetes. [writer] Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson and released as open source in 2009, Go is a statically typed, compiled language featuring built-in, lightweight concurrency primitives known as goroutines and channels that are managed by its own runtime. Owing to these strengths, it has become the primary language powering core cloud-native technologies, including Docker and Kubernetes. parallelagent and loopagent follow the same pattern. The Dev UI: ADK Web, Embedded in Your Binary The full launcher turns any agent into a CLI with console and web modes. The web mode takes sub-launchers, and webui serves the Angular ADK Web app straight out of your Go binary via go:embed, so there is nothing to install: Go import ( "google.golang.org/adk/v2/cmd/launcher" "google.golang.org/adk/v2/cmd/launcher/full" ) l := full.NewLauncher() config := &launcher.Config{AgentLoader: agent.NewSingleLoader(advisor)} if err := l.Execute(ctx, config, os.Args[1:]); err != nil { log.Fatalf("%v\n\n%s", err, l.CommandLineSyntax()) } Shell go run . web -port 8080 api webui Plain Text Web servers starts on http://localhost:8080 api: you can access API using http://localhost:8080/api The root URL redirects to /ui/, the Agent Development Kit Dev UI, and /api/list-apps answers ["assistant"]. The launcher also wires a2a, pubsub, and eventarc sub-servers, and an -otel_to_cloud flag exports traces to Google Cloud. Flags go before the sub-launcher names, or the sub-launcher rejects them. Vertex AI Gravity You can run ADK Go anywhere. But the session and memory services with persistence are the in-memory ones, a database-backed session store, and Vertex AI; the evaluation tooling, the managed scaling, and the telemetry defaults all assume Google Cloud. Same trade-off as ADK Java and ADK Python: the best experience is on GCP. Pros and Cons ✅ Pros❌ ConsOfficial Google support, highest commit velocityAgent-only: no vanilla generation, no flowsIdiomatic Go: generics for tools, iterators for eventsGo 1.26.6+ requiredEmbedded ADK Web UI, no extra installStrong pull toward Vertex AI and GCPSequential, parallel, and loop workflow agentsGemini-first; other providers only through OpenAI-compatible endpointsA2A, MCP toolsets, tool confirmation, artifactsNo vector store components; RAG goes through tools or VertexSame mental model as ADK Python/Java/TSConfig structs get long for real agents tRPC-Agent-Go History and Direction tRPC-Agent-Go is Tencent's entry, built by the team behind tRPC, the RPC framework that underpins much of Tencent's backend. Development started in May 2025; the first public release (v0.0.1) landed in August 2025, and the project reached 1.x at the end of December 2025. It is now at v1.11.2 (August 20, 2026), with 1.8k stars and 56 commits in the last 30 days. The README credits Tencent Yuanbao, Tencent Video, Tencent News, IMA, and QQ Music with real-world validation, and I have no reason to doubt it. The framework is unapologetically ADK-shaped: llmagent, runner, chainagent, parallelagent, cycleagent, sessions, memory, OutputKey, {placeholder} instructions. If you know ADK, you already know most of it. What it adds on top is where it gets interesting: a GraphAgent that is a faithful LangGraph in Go (state schemas, conditional edges, checkpoints, time travel), an AG-UI server for streaming agent events to frontends, A2A, MCP, Anthropic-style Agent Skills with sandboxed execution, agent self-evolution, an evaluation package, model failover and hedging, prompt caching, and OpenTelemetry traces and metrics with Langfuse examples. RAG is covered by a knowledge module with embedders for OpenAI, Gemini, Ollama, and Hugging Face, and stores from in-memory and SQLite-vec up to pgvector, Elasticsearch, Milvus, Qdrant, and TcVector. What Makes tRPC-Agent-Go Stand Out The GraphAgent. Nobody else in Go ships a LangGraph-equivalent state graph as a first-class agent type that plugs into the same runner, sessions, and telemetry as the LLM agents. Supported languages: Go only, go 1.21+. Model providers: OpenAI and anything OpenAI-compatible (DeepSeek, Qwen, and friends through variants), Anthropic, Gemini, Bedrock, Ollama, Hugging Face, and Hunyuan. Go import ( "trpc.group/trpc-go/trpc-agent-go/agent/llmagent" "trpc.group/trpc-go/trpc-agent-go/model" "trpc.group/trpc-go/trpc-agent-go/model/gemini" "trpc.group/trpc-go/trpc-agent-go/runner" "trpc.group/trpc-go/trpc-agent-go/tool" "trpc.group/trpc-go/trpc-agent-go/tool/function" ) type weatherReq struct { City string `json:"city" jsonschema:"description=The city to look up,required"` } type weatherRsp struct { Report string `json:"report"` } func getWeather(ctx context.Context, req weatherReq) (weatherRsp, error) { return weatherRsp{Report: fmt.Sprintf("Sunny, 24C in %s", req.City)}, nil } m, err := gemini.New(ctx, "gemini-3.8-flash") // reads GEMINI_API_KEY / GOOGLE_API_KEY weather := function.NewFunctionTool(getWeather, function.WithName("get_weather"), function.WithDescription("Returns the current weather for a city.")) advisor := llmagent.New("travel_advisor", llmagent.WithModel(m), llmagent.WithInstruction("You are a helpful travel advisor. Use tools to give accurate advice. Be brief."), llmagent.WithTools([]tool.Tool{weather})) r := runner.NewRunner("travel-app", advisor) events, err := r.Run(ctx, "user-1", "session-1", model.NewUserMessage("Should I pack a jacket for my trip to Lisbon?")) for evt := range events { if evt.Error != nil { log.Fatalf("%s", evt.Error.Message) } if evt.Response == nil || len(evt.Response.Choices) == 0 { continue } if evt.IsFinalResponse() { fmt.Println(evt.Response.Choices[0].Message.Content) } } Plain Text It is currently sunny and warm in Lisbon at 24°C (75°F), so you will not need a heavy coat. However, packing a light jacket or sweater is recommended for cooler, breezy evenings near the coast. Events arrive on a channel and use OpenAI's response shape (Choices[0].Message or Choices[0].Delta when streaming), which is familiar but means digging through nested structs for every message. Chains The chain agent is ADK's sequential agent under another name, OutputKey and {facts} included: Go import ( "trpc.group/trpc-go/trpc-agent-go/agent" "trpc.group/trpc-go/trpc-agent-go/agent/chainagent" ) researcher := llmagent.New("researcher", llmagent.WithModel(m), llmagent.WithInstruction("List three key facts about the user's topic. Be terse."), llmagent.WithOutputKey("facts")) writer := llmagent.New("writer", llmagent.WithModel(m), llmagent.WithInstruction("Write a two-sentence brief using these facts:\n{facts}")) pipeline := chainagent.New("content_pipeline", chainagent.WithSubAgents([]agent.Agent{researcher, writer})) Plain Text [researcher] 1. Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released in 2009. 2. Statically typed, compiled language featuring automatic garbage collection. 3. Native concurrency support using lightweight threads called goroutines and channels. [writer] Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson and released in 2009, Go is a statically typed, compiled language featuring automatic garbage collection. It is designed with native concurrency support, utilizing channels and lightweight threads called goroutines. GraphAgent: LangGraph in Go A state graph with a routing node, a conditional edge, and two LLM nodes. Short questions go to the quick node, long ones to the deep node. Go import ( "trpc.group/trpc-go/trpc-agent-go/agent/graphagent" "trpc.group/trpc-go/trpc-agent-go/graph" ) sg := graph.NewStateGraph(graph.MessagesStateSchema()) sg.AddNode("route", func(ctx context.Context, state graph.State) (any, error) { return nil, nil // pure routing node, no state change }) sg.AddLLMNode("quick", m, "Answer in one short sentence.", nil) sg.AddLLMNode("deep", m, "Answer thoroughly. Follow the length the user asks for.", nil) sg.SetEntryPoint("route") sg.AddConditionalEdges("route", func(ctx context.Context, state graph.State) (string, error) { input, _ := state[graph.StateKeyUserInput].(string) if len(strings.Fields(input)) > 8 { return "deep", nil } return "quick", nil }, map[string]string{"quick": "quick", "deep": "deep"}) sg.SetFinishPoint("quick") sg.SetFinishPoint("deep") g, err := sg.Compile() router, err := graphagent.New("router", g) r := runner.NewRunner("graph-app", router) Asked "What is Go?" and then a request for two paragraphs on goroutines and channels: Plain Text Go is an ancient strategy board game, as well as an open-source programming language developed by Google. Unlike operating system threads that require megabytes of memory and expensive kernel-level context switching, goroutines are extremely lightweight user-space constructs that start with just a few kilobytes of stack space. [...] The same graph package gives you checkpoints, interrupts for human input, sub-graphs, map-reduce fan-out, per-node callbacks, and a visualization export. It is the most complete workflow engine of the five, and it is the reason to look at this framework even if you never use its LLM agents. Pros and Cons ✅ Pros❌ ConsGraphAgent: a real LangGraph for GoSmallest community of the five (1.8k stars)ADK mental model without the GCP gravityNo Dev UI; observability means OpenTelemetry plus LangfuseAG-UI, A2A, MCP, Agent Skills, evaluationHuge option surface (llmagent alone has dozens)Model failover and hedging built inOpenAI-shaped event structs are verbose to consumeOpenTelemetry traces and metrics first-partyTencent-flavored corners (Hunyuan, tRPC integration)Runs on Go 1.21+Documentation partly Chinese-first LangChainGo History and Direction LangChainGo was the first. Travis Cline started it in February 2023, weeks after LangChain Python went viral, and for two years it was the default answer to "how do I call an LLM from Go". It has 9.7k stars, 17 LLM providers, and 15 vector stores, and the LangChain names (chains, prompts, agents, memory, outputparser) make it instantly readable to anyone coming from Python. And it has stalled. The last release, v0.1.14, is from October 20, 2025. The last commit to main is from January 11, 2026. It is still 0.1.x after three and a half years. There are 243 open issues and 174 open pull requests. The repository is not archived, but nobody is merging. That would be survivable if the code kept working. It does not: The default model is retired. googleai.New(ctx) with no options targets gemini-2.0-flash, and Google turned that model off: Plain Text default model: gemini-2.0-flash err: googleapi: Error 404: This model models/gemini-2.0-flash is no longer available. Please update your code to use models/gemini-3.6-flash for the latest features and improvements. The Gemini provider sits on a dead SDK. llms/googleai is built on google/generative-ai-go, whose support ended on November 30, 2025. Streaming is broken on Gemini. The chunks arrive, and then the call returns an error, so every caller that checks errors treats the request as failed: Plain Text One, two, three, four, five. err: error in stream mode: invalid character ']' looking for beginning of value No middleware, retry, fallback, or OpenTelemetry anywhere. The callback handler is seventeen observe-only methods with no way to change the request. Tools are Call(ctx, string) (string, error): no schema, no typed arguments. The full teardown, with a capstone app built in both frameworks, is in LangChainGo vs Genkit Go: Where Genkit Shines. LangChainGo's Position: Legacy None of this means the basics stopped working. Pin a current model and the classic API still runs: Go import ( "github.com/tmc/langchaingo/llms" "github.com/tmc/langchaingo/llms/googleai" ) llm, err := googleai.New(ctx, googleai.WithDefaultModel("gemini-3.8-flash")) out, err := llms.GenerateFromSinglePrompt(ctx, llm, "Explain the CAP theorem in two sentences.") Plain Text The CAP theorem states that a distributed data store can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Because network partitions are inevitable in real-world systems, architects must fundamentally choose between serving the most up-to-date data (Consistency) or keeping the system operational (Availability) when a network failure occurs. Go import ( "github.com/tmc/langchaingo/chains" "github.com/tmc/langchaingo/prompts" ) prompt := prompts.NewPromptTemplate( "Translate {{.text} to {{.language}. Reply with the translation only.", []string{"text", "language"}) chain := chains.NewLLMChain(llm, prompt) out, err := chains.Predict(ctx, chain, map[string]any{ "text": "Buenos días", "language": "English", }) Plain Text Good morning Tools are strings in, strings out. The agent is a ReAct loop driven by prompt text: Go import ( "github.com/tmc/langchaingo/agents" "github.com/tmc/langchaingo/tools" ) type WeatherTool struct{} func (WeatherTool) Name() string { return "get_weather" } func (WeatherTool) Description() string { return "Returns the current weather. Input: a city name." } func (WeatherTool) Call(ctx context.Context, city string) (string, error) { return fmt.Sprintf("Sunny, 24C in %s", city), nil } agent := agents.NewOneShotAgent(llm, []tools.Tool{WeatherTool{}, agents.WithMaxIterations(3)) out, err := chains.Run(ctx, agents.NewExecutor(agent), "What's the weather like in Tokyo?") Plain Text The weather in Tokyo is currently sunny with a temperature of 24°C. If you are on LangChainGo today, the move is not urgent, but it is inevitable: pin your models, avoid streaming on Gemini, and plan the migration. The concept-by-concept guide is in Migrating from LangChainGo to Genkit Go. Pros and Cons ✅ Pros❌ ConsLargest catalog: 17 providers, 15 vector storesUnmaintained since January 2026; still 0.1.xLangChain vocabulary, easy for Python teamsGemini provider on a retired SDK; streaming brokenConversation memory is ergonomicNo middleware, retry, fallback, or OpenTelemetryRuns on Go 1.24+Tools are untyped stringsMany existing tutorials and examplesEvery vector store needs an external service Head-to-Head Comparison Developer Experience FrameworkDX HighlightsShortcomingsGenkit GoDev UI with trace explorer. Typed output from structs. Flows are one line from HTTP.Agents still experimentalEinoTyped graphs, automatic streaming, callbacks everywhere.Verbose config structs; 0.x churnADK GoIterator-based event loop, generic tools, embedded web UI.Agent-only; Go 1.26.6tRPC-Agent-GoGraphAgent, ADK mental model, resilience built in.OpenAI-shaped events; option overloadLangChainGoFamiliar names, biggest catalog.Unmaintained; broken Gemini streaming Abstraction Levels Genkit Go is the only framework here with all three levels in one SDK: vanilla generation, typed flows, and agents. Eino covers generation and orchestration, plus agents through its ADK module. ADK Go and tRPC-Agent-Go are agent frameworks: every call goes through an agent and a runner. LangChainGo has chains and agents, but no typed pipeline and no session-aware agent. Observability FrameworkLocal DevProductionGenkit GoDev UI with trace explorerOpenTelemetry export, Google Cloud plugin, any OTEL collectorEinoEino Dev plugin for GoLand and VS Code (graph rendering, node debugging)Callback handlers: Langfuse, LangSmith, CozeLoop, APMPlusADK GoEmbedded ADK Web UIOpenTelemetry to Google Cloud (-otel_to_cloud)tRPC-Agent-GoEvents on a channel, no UIOpenTelemetry traces and metrics, LangfuseLangChainGoLogging callbacks onlyNothing built in Provider Support FrameworkModel ProvidersVector StoresGenkit GoGemini (Google AI, Vertex AI), Anthropic, Ollama, OpenAI-compatible family (OpenAI, xAI, DeepSeek, OpenRouter, Kimi, DashScope, Z.ai)localvec, pgvector, AlloyDB, Pinecone, Weaviate, FirestoreEinoOpenAI, Claude, Gemini, DeepSeek, Qwen, Ollama, OpenRouter, Ark, QianfanElasticsearch, Milvus, Qdrant, Redis, OpenSearch, Dify, VikingDBADK GoGemini, OpenAI-compatible, ApigeeNone as components (Vertex AI Search via tools)tRPC-Agent-GoOpenAI-compatible, Anthropic, Gemini, Bedrock, Ollama, Hugging Face, HunyuanIn-memory, SQLite-vec, pgvector, Elasticsearch, Milvus, Qdrant, TcVectorLangChainGo17, including OpenAI, Anthropic, Google AI, Vertex, Bedrock, Cohere, Mistral, Ollama15, all external services How Go Does It Feel? FrameworkMin GoToolsStreamingStyleGenkit Go1.25Generic, schema from structCallback per chunkFunctional options (ai.With...)Eino1.18Generic (utils.InferTool)StreamReader, automatic across nodesConfig structs plus builder chainsADK Go1.26.6Generic (functiontool.New)iter.Seq2 over eventsConfig structstRPC-Agent-Go1.21Generic plus jsonschema tagsChannel of eventsFunctional optionsLangChainGo1.24Untyped stringsCallback per chunk (fails on Gemini)Interfaces plus option funcs Only ADK Go uses range-over-func iterators for its event loop. Genkit and Eino use generics the most naturally; LangChainGo predates all of it and shows its age. Maintenance Snapshot Numbers as of September 21, 2026: FrameworkStarsLatest ReleaseLast CommitCommits, Last 30 DaysGenkit Go6.4k (monorepo)go/v1.13.1, Sep 3, 2026Sep 21, 202680 (all SDKs)Eino13.1kv0.9.20, Sep 20, 2026Sep 20, 20267 on main, plus five v0.10 alphasADK Go8.8kv2.4.0, Sep 11, 2026Sep 21, 2026100+tRPC-Agent-Go1.8kv1.11.2, Aug 20, 2026Sep 21, 202656LangChainGo9.7kv0.1.14, Oct 20, 2025Jan 11, 20260 Stars measure the past. The last two columns measure the present, and they tell the story. Which Framework Should You Choose? Choose Genkit Go if: You want to iterate on your AI fast and get feedback with less back-and-forth. The Dev UI is the best local development loop in Go, by a distance.You need vanilla calls, typed flows, and agents in the same service.Provider neutrality matters: mixing Gemini, Claude, OpenAI, and Ollama should be a one-line change.Your team also writes TypeScript, Dart, or Java and wants one framework story. Choose Eino if: Your application is a real graph: branches, parallel fan-out, sub-graphs, and streaming across all of it.You want the most active independent community in Go and are fine living on 0.x.Callback-based instrumentation into Langfuse, LangSmith, or your own APM is what you already do.You need ByteDance-scale orchestration patterns, or you run on Volcano Engine. Choose Google ADK Go if: You are building multi-agent systems and Google Cloud is your runtime.You want official Google support, the embedded ADK Web UI, and A2A interoperability.Your organization already uses ADK in Python or Java and wants Go services with the same shape.The newest Go toolchain is not a problem for you. Choose tRPC-Agent-Go if: You want LangGraph-style state graphs with checkpoints and interrupts, in Go, plugged into agents.You like the ADK mental model but do not want the GCP gravity.AG-UI frontends, Agent Skills, evaluation, and OpenTelemetry out of the box matter more than a Dev UI.You can live with a smaller community and a large API surface. Stay on LangChainGo only if: You already run it in production, your models are pinned, and you do not stream through Gemini.Even then, plan the migration. Nothing is being merged. Conclusion Go's Gen AI framework landscape in 2026 is richer than most Go developers realize, and it has a clear shape. Genkit Go is the most versatile choice and the best developer experience, with the Dev UI as a category of its own. Eino is the ambitious independent, the framework to pick when orchestration is the hard part of your problem. Google ADK Go is the enterprise agent runtime, at its best on Google Cloud. tRPC-Agent-Go is the dark horse, with the best workflow engine of the five hiding behind a modest star count. And LangChainGo is the framework that got everyone started and now needs a farewell plan. If I had to pick one for a new Go service today with no cloud commitment, it would be Genkit Go, for the same reason as in TypeScript and Java: the combination of multiple abstraction levels, provider neutrality, and local observability shortens the loop between an idea and a working feature more than anything else on this list. Whatever you pick, pick it deliberately. Go finally has the tooling. Use it. Last updated: September 2026. Framework versions referenced: Genkit Go v1.13.1, Eino v0.9.20 (with eino-ext gemini v0.1.36), Google ADK Go v2.4.0, tRPC-Agent-Go v1.11.2, LangChainGo v0.1.14. Go 1.27.1, model gemini-3.8-flash.

By Xavier Portilla Edo

CORE

PWA on iOS, Part 3: Chrome on iPhone Is WebKit Too. How One Engine Sets the Rules for PWAs on iOS

This is Part 3 of an 8-part series about PWAs on iOS. We've covered the history of the technology and the three ingredients every PWA needs: HTTPS, a manifest, and a service worker. Now it's time to look at the engine all of that runs on, and why, on the iPhone, Apple chose it for you long ago. A typical bug report from a tester goes something like this: "Works fine on Android. On iPhone we tried Safari and Chrome, and it breaks the same way in both." The tester dutifully opened two browsers and believes two engines are covered, when in fact they tested the same WebKit twice in different wrappers. Chrome on iPhone renders pages with the same system engine as Safari, and so do Firefox, Edge, Brave, and the rest. They differ in toolbars, bookmark sync, and looks, but in every one of them the page is built and run by WebKit, in whatever version shipped with iOS. It's easy to jump to the opposite conclusion: if there's only one engine, testing Safari must be enough. That's a mistake too, and just as common. When someone says "it works in Safari for us" on a call, it usually means they opened the site in a tab and stopped there. But one engine doesn't mean one environment. On an iPhone, it runs in at least four, and each has its own cookies, its own storage, and its own rules for how long a process lives. Checking a Safari tab and declaring victory is like looking into one room out of four and concluding that the whole house is tidy. We'll go through the rooms below. A Fork That Outlived Its Parent WebKit wasn't written from scratch in Cupertino. Its ancestor was KHTML, the engine of KDE's Konqueror browser, and next to it lived a JavaScript interpreter called KJS. Both projects were open source and distributed under the LGPL, a license that lets you take the code, modify it, and even build it into a closed-source product, but requires you to publish your changes to the library itself. KDE is an international free software community. It began in 1996, when Matthias Ettrich, a student at the University of Tübingen, announced on Usenet a project for a graphical desktop environment for Unix-like systems, and KDE 2.0 in October 2000 introduced Konqueror, which replaced the file manager and doubled as a browser. Apple started work on its fork in the summer of 2001. The reasons given for the choice were very practical: KHTML and KJS were small (under 140,000 lines of code), cleanly designed, standards-compliant, and easier to work with than the other options available. In January 2003, Steve Jobs unveiled Safari at Macworld, and the same day an Apple engineer, Don Melton, wrote to the KDE developers: Safari was built on their code, the modified versions had been made public, and a list of the changes would follow in a separate email. You often read that "Apple open-sourced WebKit in 2003," and that's not quite accurate. In 2003, two libraries were open: WebCore (HTML and CSS parsing, layout and rendering) and JavaScriptCore, both under the LGPL because they had grown out of KDE code released under that license. The WebKit framework itself, the glue code that Safari and other Mac programs used to talk to the engine, was only opened up on June 7, 2005, together with a public repository and bug tracker. Dave Hyatt of the Safari team announced it on his blog. The first iPhone in 2007 shipped with the same engine, which had to be squeezed onto a slow processor and a small amount of memory. In 2010, the project announced WebKit2, an architecture where the interface and the page content live in separate processes. The UI Process handles the window and the buttons, while the Web Content Process parses HTML, runs JavaScript, and can crash without taking the whole app down with it. On a phone this design matters more than on a laptop, and we'll come back to it. By then Google had been shipping Chrome on WebKit for a year and a half, but Chromium's multiprocess model had been built its own way from the start, and maintaining two architectures in one project was getting more and more expensive for both sides. On April 3, 2013, Google announced a fork called Blink and named exactly this reason in the announcement itself. Chrome on Android and Safari on iPhone have been cousins ever since, sharing a common ancestor. But after thirteen years of separate lives, the engines have drifted so far apart that how one behaves tells you almost nothing about how the other will. As for KHTML, where it all began, KDE eventually stopped developing it: work on KHTML ended in 2016, and in 2023 it was officially retired. One Line in the App Store Rules The WebKit requirement fits into a single line of the App Store rules. Guideline 2.5.6 says: "Apps that browse the web must use the appropriate WebKit framework and WebKit JavaScript." Today the guideline adds that you may apply for an entitlement to use an alternative browser engine, with links to the rules for the EU and Japan, but everywhere else the requirement is unchanged. For Google and Mozilla, this meant their iPhone browsers stayed shells for years: their own interface, account sign-in, tab and password sync, and inside it the same WKWebView that any app in the store can use. Chrome on iOS can't enable a feature WebKit hasn't shipped yet, and it can't work around a bug WebKit hasn't fixed yet. If something gets fixed in Chromium on a Tuesday, iPhones get the fix only when Apple ships an iOS update with a newer WebKit, and a fresh version of Chrome from the App Store changes nothing. Apple justifies the rule on security grounds, and I don't think that's a mere excuse. In its developer documentation, Apple calls browser engines one of the most common attack vectors, and that's hard to argue with: an engine constantly processes untrusted code from all over the internet. When there is one engine, and it's part of the system, a single iOS update closes the hole for every browser at once. On top of that, fast JavaScript needs JIT compilation, meaning the ability to create executable code in memory at runtime, and ordinary iOS apps aren't given that permission. I find the decision reasonable for the platform and costly for the web. The security is paid for by making the web on iPhone depend on a single supplier, and that supplier's release schedule automatically becomes everyone else's. Attempts to get out of this arrangement began before Apple allowed anything. In December 2022, press reports appeared saying that Apple was preparing for the EU's Digital Markets Act and was discussing dropping the mandatory WebKit rule. About six weeks later, on January 31, 2023, the Chromium team opened an issue in its public tracker for an experimental port of Blink to iOS. The issue stated up front that they were building a stripped-down test shell, content_shell, rather than a browser for the store, because such a browser would violate the App Store rules. Around the same time there were reports that Mozilla was also trying to get its Gecko engine running on iPhone. I'll say how that turned out below, when we get to the regulators. Four Rooms in One House Even with a single engine, the environment it runs in can differ, and for a web app that difference matters more than the browser's brand. I keep four such environments in mind, and data doesn't flow from one to another on its own. Room 1 The most familiar one is an ordinary Safari tab. Cookies, localStorage, IndexedDB and camera and notification permissions are shared across the tabs of a profile, Web Inspector on a Mac can attach to the tab, and almost every tutorial on the internet was tested right here. Room 2 SFSafariViewController looks almost the same. It's a Safari window with Reader and password autofill that a native app opens on top of itself when you tap a link in a feed or an email. The app itself can't see what's happening inside. Before iOS 11, that window shared cookies with Safari, and a lot of single sign-on setups relied on it. In iOS 11 Apple separated the storage, and each app got its own cookie jar for such windows. Today Apple's documentation explicitly sends anyone who needs data shared with Safari to a different API, ASWebAuthenticationSession. The old behavior still circulates in forums and articles. If someone confidently says that the user is already logged in inside SFSafariViewController, ask which iOS version they tested that on. Room 3 WKWebView is a different story. This embeddable web view is what hybrid apps, the in-app browsers of messengers, and Chrome and Firefox themselves are built on. Its storage belongs to the host app: Safari's cookies don't carry over, but the app itself can read and set cookies through WKHTTPCookieStore, so your session won't be a secret from the host. Even the quota is different. Since iOS 17, WebKit lets a single site in a browser app, meaning one that can be set as the default browser, use up to 60% of the disk, but inside the web view of an ordinary app only up to 15%. Room 4 An icon on the Home Screen, opened as a web app, lives apart from all the others. It has its own container with its own cookies, localStorage, IndexedDB, Cache Storage and its own permissions, and that container can't see Safari or any web view embedded in other apps. Someone who's logged in inside a tab will install the app, tap its icon and find a login form, and camera access granted in the tab has to be granted again. Since iOS 26, a site added to the Home Screen opens as a web app by default. You can turn off the "Open as Web App" switch when adding it, and then the icon becomes an ordinary bookmark that takes you back to Room 1. When a bug report comes from a user, the first thing we need to work out is which room they were in. For that, I usually hide a diagnostics screen in the app, opened by long-pressing the version number, that shows an object like this: JavaScript async function whereAmI() { const standalone = matchMedia("(display-mode: standalone)").matches || navigator.standalone === true; return { standalone, path: location.pathname, sw: navigator.serviceWorker?.controller?.scriptURL ?? "none", caches: "caches" in self ? await caches.keys() : [], build: document.documentElement.dataset.build, ua: navigator.userAgent, }; } navigator.standalone is a nonstandard Apple property, and inside an embedded WKWebView both checks come back false, so they can't tell a Safari tab from an in-app browser inside a messenger. The user agent sometimes helps, since it may contain the name of the host app, but that's a hint for a human reading the report, and I wouldn't build logic on it. I take the address from pathname because the query string sometimes holds one-time tokens that have no business being in logs. From here on, when I say a web app opened from the Home Screen, I mean an actual PWA, not a plain bookmark to a site page that someone has put on the Home Screen. Why Users Log In Three Times The most common story I've run into across different teams goes like this. A corporate single sign-on login works fine in a Safari tab, because the identity provider's cookie is already in the profile. Then a native wrapper appears, and sign-in there is opened through SFSafariViewController, expecting the same cookie to be there. It isn't, so the person types their password a second time. Then the site is put on the Home Screen, and it all happens a third time, because the icon has its own container. Support gets a complaint: "the app keeps logging me out," even though the app never did anything of the kind. Each sign-in the user went through simply happened in a new, separate container. You can't read Safari's cookies from another environment. There's no public API for it, and that's by design. The workable options are more boring. A native app that needs a session from Safari should use ASWebAuthenticationSession, available since iOS 12. It shares persistent cookies with Safari and asks the user for permission before opening the sign-in page, and if you specifically don't want a shared session, the prefersEphemeralWebBrowserSession flag makes the sign-in private. For a PWA, I assume that sign-in starts from scratch after installation, and I try to keep it short: OAuth with PKCE, a redirect back to an address inside the manifest's scope, and a long-lived session on your own domain, so that the user signs in once after installing rather than every week. I also always check on a real device where the provider's page opens and where the redirect brings the user back, because navigating outside scope in an installed app behaves differently than it does in a browser tab. Twenty Billion Dollars for a Search Bar If the WebKit-only requirement rested on security alone, it would be easier to argue about. But Safari has a commercial side too: the right to be its default search engine is worth a great deal. In the US Department of Justice's antitrust case against Google, court documents disclosed in spring 2024 put the amount Google paid Apple in 2022 to be Safari's default search engine at about $20 billion. That doesn't prove WebKit was made mandatory on every iPhone for the sake of money, but it shows what's at stake: Safari is a significant source of revenue for Apple. So anything that could take users away from it on iPhone, whether a browser choice screen or other engines, touches not only architecture but revenue. Critics of the rule that every browser must use WebKit, above all Open Web Advocacy, have argued for years that without competition between engines Apple has little incentive to move quickly on the features that make web apps resemble native ones. As an example, they point to Web Push, which web apps on iPhone didn't get until spring 2023. Apple, for its part, points to Safari's speed, to privacy, and to its view that some APIs from Chromium are dangerous for users. I'm not firmly on either side of this argument. Some of the features missing on the iPhone are ones I wouldn't hand to a random website on my own phone either. Others were delayed for years without a clear explanation, and from the outside it's very hard for me to tell where concern for the user ends and concern for the App Store begins. The Month PWAs Disappeared in Europe The first to break the WebKit-only rule was the European Union. The Digital Markets Act (DMA) required Apple to allow other browser engines on iPhone, and iOS 17.4, released in March 2024, introduced BrowserEngineKit, a set of system interfaces for browsers that run on their own engine. It applies in the 27 EU member states, but the European Economic Area as a whole isn't covered, so a user in Norway, for example, is left with plain WebKit. Apple spelled out the conditions in great detail, and they read more like a list of reasons not to bother. A browser with its own engine must be distributed only in the EU, must be eligible to be set as the default browser, and must pass 90% of the Web Platform Tests and 80% of Test262 even with JIT turned off (for example, when the user has Lockdown Mode enabled). It also has to publish a vulnerability disclosure policy and quickly patch actively exploited vulnerabilities, with Apple naming a 30-day window for straightforward cases. Mozilla reacted right after the announcement, in January 2024. A company spokesperson, Damiano DeMonte, told The Verge that Mozilla was "extremely disappointed" and explained why: "The effect of this would be to force an independent browser like Firefox to build and maintain two separate browser implementations — a burden Apple itself will not have to bear." Formally, Apple had complied with European law. In practice, it had made it prohibitively expensive for any company that develops its own browser engine to bring it to Apple's devices in the EU. For PWAs, the most interesting part happened in the beta. In February 2024, developers in the EU noticed that in the iOS 17.4 beta, PWAs no longer opened in their own window and had turned into bookmarks that launch an ordinary browser tab. Gone with the window were the separate data container and Web Push, which had arrived only a year earlier. Apple blamed the same DMA: since browsers on other engines were now allowed, they would have to be let into the web app mechanism too, and that would require a new integration architecture. Building it, Apple said, was impractical: on top of the law's other requirements, very few people use Home Screen web apps, and the company judged the cost unjustified. Open Web Advocacy, developers and the press kicked up a fuss, the European Commission got involved, and on March 1, 2024, Apple reversed the decision. The final iOS 17.4 brought web apps back in the EU, still running on WebKit. That autumn Apple took the next step: starting with iOS 18.2, a browser with its own engine in the EU can create Home Screen web apps that run on that engine. On paper, that's exactly the freedom people were fighting for. But only a browser with its own engine that has met all of Apple's conditions can use it. London and Tokyo Followed Brussels The UK's Competition and Markets Authority (CMA) wrapped up its study of mobile ecosystems in June 2022 by proposing an investigation into how the mobile browser market works. On November 22 of that year, it launched an in-depth market investigation into browsers and cloud gaming. Apple challenged that decision in court, and after a ruling by the Competition Appeal Tribunal, the investigation was suspended from March 31, 2023. The CMA appealed the ruling; the Court of Appeal upheld the CMA's decision on November 30, 2023, and on January 24, 2024, the investigation resumed. The final report came out on March 12, 2025, and its conclusion is blunt: competition in the UK mobile browser market isn't working well, users and businesses both suffer as a result, and the main concerns relate to Apple's rules for browsers on its devices. The CMA decided to change those rules not through the investigation but under the UK's new digital markets law, the Digital Markets, Competition and Consumers Act, which came into force on January 1, 2025. On October 22, 2025, it designated Apple and Google as having Strategic Market Status in mobile platforms, and it explicitly included browsers and browser engines in its definition of a mobile platform. The regulator now has a tool to demand changes. But as of autumn 2026, there is still no requirement to allow other engines on iPhone: Apple's first commitments to the CMA concern the rules for reviewing and ranking apps in the App Store. Japan went further. The Mobile Software Competition Act (MSCA) was passed in June 2024 and came into full force in December 2025. In the guidelines to it, which got a lot of coverage in summer 2025, the Japanese regulator specifically barred Apple from placing unjustified technical restrictions on alternative engines, from imposing high financial costs on developers, and from steering users away from browsers built on such engines. In effect, Japan ruled out the European scenario in advance, where permission technically exists but using it doesn't pay. On December 17, 2025, Apple announced its Japan changes in iOS 26.2. Users in Japan got a choice screen for browsers and search engines, and browser developers could move to engines other than WebKit, with access to JIT and multiprocessing and, as Apple puts it, under strict security and privacy requirements. As for the prototypes mentioned earlier, they remained prototypes, and ordinary Chrome on iPhone, in Europe and Japan alike, still runs on WebKit. In June 2026, Kyle Pflug of the Microsoft Edge team published benchmark results for a research prototype of a Blink-based browser for iOS, built with BrowserEngineKit. In Speedometer 3.1, the prototype beat Safari on iOS 26.5.1 by about 29%, and Pflug himself stressed that he was showing a research build, not announcing a product. As of autumn 2026, there are almost no full-fledged iPhone browsers running on their own engine. I wouldn't build product plans around their arrival any time soon, but "never" isn't the right word either. So for now, what the web can do on iPhone is determined by WebKit, and you have to plan a product around what WebKit lacks for PWAs on iOS. Some of the gaps are visible right away, while others surface at the most inconvenient moment. The one that stands out first is the lack of an install button: a site can't offer to install itself as an app. There Will Be No "Install" Button Chromium has a beforeinstallprompt event: a site uses it to show its own button and trigger the system install dialog. iOS has no such event, and the only way to put a site on the Home Screen is by hand, through Share and then Add to Home Screen. Because of this, onboarding on iPhone always ends up a little clunky. In a Safari tab, you can show the instructions in words and a picture; in an already installed app, you have to hide them; and in a messenger's in-app browser, they're often useless because the elements they refer to, such as the Share button, simply aren't there. There are no background mechanisms either. Background Sync, Periodic Background Sync and Background Fetch aren't implemented on iOS, so you can't send out a queue of pending requests after the user has closed the app. On the hardware side, Web Bluetooth, Web NFC, and WebUSB are unavailable, and navigator.vibrate is missing even for a short bit of haptic feedback. Take an app for a warehouse worker who scans barcodes with the phone's camera: on Android, a failed scan can be signaled with a vibration, but on iPhone there's no such signal, so the error has to be shown with a screen color or a sound. If the product needs a Bluetooth printer, or has to work with the screen locked, it's more honest to discuss a native app right away than to promise that the web will catch up soon. With no background sync, the whole model for saving data changes. I put operations into a queue in IndexedDB and send them at moments when the app is definitely running: when it comes back to the foreground and when the network reappears. Each operation gets an idempotency key, so that resending after a dropped connection doesn't create a duplicate: JavaScript let flushing = false; async function flushOutbox() { if (flushing) return; flushing = true; try { for (const job of await outbox.pending()) { await api.send(job.payload, { idempotencyKey: job.id }); await outbox.markSent(job.id); } } catch { // no network or the server returned an error: we'll try again on the next event } finally { flushing = false; } } The online event tells you the device has connected to a network, but it doesn't guarantee that a request will reach the server. For example, the phone picks up Wi-Fi on the subway, but there's no internet behind it yet, and the very first request fails. That's why the loop stops at the first error and waits for the next trigger: another online event, or the app returning to the foreground. The second weak spot is in the interface: if you don't show the user that some operations haven't been sent yet, they'll close the app believing everything went through. How a service worker fits into this scheme, and why it doesn't save you when the app is closed, is covered in Part 5 of this series. The Safari Version Is the iOS Version Everything above applies to WebKit in general, but on a specific phone the set of available features also depends on the iOS version. On a Mac, Safari updates separately from the system. On an iPhone, it doesn't work that way: the engine is built into iOS and updates only together with it. Safari 16.4 arrived with iOS 16.4, Safari 26 with iOS 26, and a user can't install a newer browser on an older system. Look at the device report in your web analytics (Matomo, Google Analytics, or your server logs): if a noticeable share of visitors come from iPhones running iOS 16.2, those people have no Web Push. Telling them to "update Chrome" won't help, because Chrome on such a phone runs on the same WebKit from 16.2. Only updating iOS can help. That's why I tie the test plan to iOS versions: on an iPhone, the browser brand says almost nothing about the engine version. I keep two reference points: the lowest version the product still promises to support, and the current one. Detecting the engine version from the user agent (UA) isn't a good idea either. The user agent is a string that a browser sends with every request to introduce itself to the site: its name, its engine, and the operating system. On iPhone it's almost useless: every browser has the word Safari in it, and the AppleWebKit number hasn't changed in years. If the condition that enables push depends on parsing such a string, I ask for it to be removed in code review. In code, check for capabilities, not versions: is there a PushManager (the object a site uses to subscribe to push) and is the app running in standalone mode, meaning launched from an icon on the Home Screen. If there's no PushManager and the app is open in a tab, show the install instructions. If the app is already installed and there's still no PushManager, the iOS version is too old, and you should offer the user a system update. If PushManager is there, show the notification bell. Safari's release cycle has a rhythm worth knowing. The major autumn releases bring what was shown at WWDC (Apple's developer conference in June), but for web apps on iPhone, a lot of the important changes arrive in spring, most often in the x.4 releases. Safari 15.4 in March 2022 brought more than 70 additions, among them <dialog>, :has() and support for icons from the manifest (though apple-touch-icon still takes priority in Safari). Safari 16.4 in late March 2023 added 135 features, including the Badging API and, of course, Web Push for Home Screen web apps. Safari 18.4 in spring 2025 brought Declarative Web Push, and 26.4 in March 2026 added WebTransport. The WebKit team acknowledges the pattern itself in its post about 26.4: "March has a way of bringing a lot of new things to WebKit — and this year is no exception." There are exceptions, too: 17.4 in March 2024 was modest in features, but it contained all the changes that Europe's DMA required. You can find out what the next release will bring in advance, and without any insider contacts. Can I Use and MDN show what Safari already supports, and the Standards Positions page shows where WebKit stands on new specifications. Safari Technology Preview on a Mac lets you try features months before release, though for iPhone it's only a rough guide: it doesn't fully reproduce iOS behavior. The Safari release notes and posts on the WebKit blog list the changes in each release point by point, and the blog also carries notes for every Technology Preview build. On the iPhone itself, in Safari's settings under Advanced, there are switches for experimental features: in iOS 16.4 the item was called Experimental Features, and in newer versions it was renamed Feature Flags. They're handy for trying out an API ahead of time, but before you investigate a bug a user has reported, reset them to their defaults. Otherwise you'll be reproducing the problem in a browser with settings that the user doesn't have. A Link from a Messenger Opens in the Wrong Place Many people reach a site without going through Safari at all: through a link in a messenger, in Instagram, in an email client or in a corporate app. Such apps open the site either in SFSafariViewController or in their own browser built on WKWebView, and in the second case you land in someone else's container, where the host decides what's allowed. There's usually no Add to Home Screen item there, the Share button may not exist at all, and a hint saying "tap Share" leads the person nowhere. For these cases, it's better to show a short text asking the user to open the link in Safari, along with a button that copies the address. Many messengers already offer their own "open in browser" control, but it's safer not to rely on it and to build your own. Debugging all this is a special kind of fun. Web Inspector on a Mac can attach to Safari tabs and to installed web apps, but it will only see someone else's WKWebView if the app's developer has explicitly allowed inspection: since iOS 16.4 there's an isInspectable property for that, and it's off by default. You most likely won't be able to attach your inspector to the browser inside a popular messenger. If a bug reproduces only "from the messenger," I first ask the person to open the same link in Safari, and a good share of such bugs disappear there, because they lived in the host's embedded window, which your service worker might never have reached. JavaScriptCore Does Things Its Own Way Suppose the page opened in the right place: in a Safari tab or in an installed app. Next, the code starts running, and here too things differ from what you know from Chrome. Because Chrome on iPhone runs on WebKit, it executes JavaScript not with the familiar V8 but with WebKit's own engine, JavaScriptCore. V8, the JavaScript engine in Chrome, has little in common with it beyond the language specification. JavaScriptCore has four execution tiers. First, the code is run by an interpreter called LLInt (low-level interpreter). When a function gets hot, meaning it's called or loops often enough, the engine compiles it on the fly, which is known as JIT (just-in-time compilation): first with the simple Baseline JIT tier, then with the optimizing DFG (data flow graph) tier, and finally with the most aggressive one, FTL (faster than light). The engine decides when to move up a tier based on an execution counter and on statistics about value types that it collects as it goes. In an ordinary interface, the difference between JavaScriptCore and V8 is barely noticeable. On heavy computation, the picture varies: one engine is faster on some tasks, the other on others. So it makes sense to profile for iPhone on an iPhone itself: measurements from Chrome DevTools tell you about V8's performance, not JavaScriptCore's. Lockdown Mode changes this picture more than any optimization does. It's an optional mode of extreme protection, introduced in iOS 16: the person turns it on themselves in Settings, under Privacy & Security, and the phone restarts. Apple built it for the very few people who may be personally targeted because of who they are or what they do. Apple says that in this mode some complex web technologies are blocked, so websites may load more slowly, web fonts may not display, and images may be replaced with a missing-image icon. JIT is unavailable in it as well: Apple says so directly in its conditions for browsers on third-party engines, discussed earlier. That means all JavaScript runs on the slowest tier only, the interpreter, and a page that flies for everyone else can be noticeably sluggish for such a user. So if a bug reproduces for only one user while everyone else on the same iOS version is fine, ask them whether Lockdown Mode is on (Safari shows a banner when it is). A site can't detect this itself, and Apple doesn't describe any way to. Ask before you spend a week hunting for a race condition in React: with slow JavaScript, the order of events changes, and an ordinary page starts to behave erratically. You don't need to work around the mode. What you need is an interface that stays usable with slow JavaScript, without web fonts, and with images that may fail to load. JavaScript syntax has its own release schedule, too. Lookbehind in regular expressions (a check of what comes before a match, a construct like (?<=...)) arrived in Safari only in 16.4, in spring 2023, together with Import Maps. Before that, a single such regex in a bundle caused a SyntaxError at parse time, and the user saw a white screen instead of one broken function. As far as I know, transpilers don't rewrite lookbehind for you. You either rewrite the expression by hand or raise your minimum supported iOS version to 16.4. structuredClone, on the other hand, has been available since 15.4, and a polyfill for it (code that imitates a missing function) only adds weight on modern iPhones. The Process Will Die, but the Data Must Survive So far we've looked at how JavaScript executes on iPhone. Now let's look at how long it lives: a page can vanish at any moment, without warning, and for your data that matters more than any difference between engines. It's time to return to WebKit2, which came up in the history of the engine. There, the interface and the page content live in separate processes, and your page is run by the Web Content Process. When the system runs low on memory, jetsam steps in, the iOS mechanism that frees memory by terminating processes, and it simply kills that process. This is especially likely when the user switches away from the app: they're filling in a form, get distracted by a phone call or switch to the camera to photograph a document, and while they're busy, the system removes the page left in the background from memory. The page gets no "about to close" event: neither beforeunload nor pagehide arrives, and everything it held in memory (variables, objects, the not-yet-submitted contents of a form) disappears at once. When the person comes back, they find the app starting from scratch, or an empty form on the screen. For any screen beyond a single short form, my takeaway is a simple rule: write the draft to IndexedDB as the user types, with no "Save draft" button, which people won't press anyway, because they don't know their app can be killed out from under them. JavaScript // The form the user is filling in const form = document.querySelector("#invoice"); // ID of the pending save. While the person is typing, we keep resetting it let timer; // The input event fires on any change to a form field, // that is, on every keystroke form.addEventListener("input", () => { // The person is still typing: cancel the save scheduled by the previous keystroke clearTimeout(timer); // Schedule a new save in 400 ms. If the field changes again before then, // the timer is reset, so the write happens once, after a pause in typing, // and not on every letter timer = setTimeout(() => { // Collect all the form's field values into a plain object like { fieldName: value } const data = Object.fromEntries(new FormData(form)); // Write to IndexedDB under the key "invoice", overwriting the previous draft. // drafts here is a small wrapper over IndexedDB (its code isn't shown in this example). // savedAt lets us show "there's an unsaved draft from 2:02 PM" when restoring drafts.put("invoice", { data, savedAt: Date.now() }); }, 400); }); This pattern is called debouncing: the write is postponed until there's a pause in typing, so the database isn't hit on every letter. The 400 ms delay means the last few characters typed before the process dies can be lost, and for an invoice I'm fine with that. Restoring matters more. You must never silently fill in a half-completed form. The person should see a line like "There's an unsaved draft from 2:02 PM" and choose for themselves whether to continue it or throw it away. Otherwise, sooner or later an old draft will land on top of a new invoice, and sync will take the blame. Two stubborn myths surround storage for PWAs on iOS: the "hard 50 MB limit" and "everything gets wiped after seven days." The first is long outdated: as I mentioned earlier, since iOS 17 WebKit calculates quotas as a percentage of disk size, an installed web app gets the same quota as a site in the browser, and Apple doesn't publish a fixed figure in megabytes. The second grew out of a real change, but one that affected only Safari. In March 2020, WebKit introduced a seven-day limit in ITP (Intelligent Tracking Prevention, Safari's protection against tracking): if someone uses Safari for seven days and never interacts with a site, everything the site wrote with scripts, from localStorage to IndexedDB and service worker registrations, gets deleted. But that rule was written for sites opened in Safari. An installed web app runs outside Safari, and in that same announcement WebKit wrote that it doesn't expect data in such an app to be deleted: that isn't what ITP is meant to do, and it would treat a case of deletion as a serious bug. So the "seven days" story shouldn't be applied to installed PWAs. None of this guarantees that data lives forever, because it can disappear for other reasons. When the device is low on storage, the system can evict a PWA's data regardless of ITP, and if the user deletes the app's icon from the Home Screen, the data goes away with the app. navigator.storage.estimate() gives only an approximate figure. WebKit grants navigator.storage.persist() requests based on heuristics, and one of them, according to WebKit's own description, is that the site is launched as a Home Screen web app. Anything you can't afford to lose should live on the server, and the local database should serve as a cache and a queue. Everything above applies to an app that's already on the Home Screen. But a PWA doesn't get its own storage in advance: it appears at the moment of installation, and every installation gets its own. So in Part 4 of this series, we'll tap Share, choose Add to Home Screen, and see what iOS creates at that moment: where the icon comes from, exactly when that fourth room we talked about above appears, and why the same site can be installed as a PWA twice, leaving you with two apps that know nothing about each other. Other Parts in This Series PWA on iOS, Part 1: PWAs on the iPhone Are Older Than They LookPWA on iOS, Part 2: HTTPS, Manifest and Service WorkerPWA on iOS, Part 4: What Add to Home Screen Really DoesPWA on iOS, Part 5: Service Worker Updates, Cache, and PushPWA on iOS, Part 6: Performance, Rendering, and Keyboard

By Konstantin Shkurko

Guess the Speed: Building a Family Internet Speed Guessing Game

In this blog post, we will see how I turned our family's speed test ritual into a family internet speed guessing game that you can play on your own Wi-Fi. Every family has that one argument. "The internet is slow again." "No, it is fine; you are just streaming in 4K." Nobody really knows who is right, because nobody knows the actual speed. So we started guessing before every speed test. It became a game, and the game became a website. Let's see how it works and how I built it. The author describes developing Guess the Speed, a multiplayer game that transforms family internet speed tests into a competitive activity. Players submit secret download and upload guesses before running a real speed test against Cloudflare's network, then earn points based on accuracy using place rankings, bonuses for close guesses, and a cutoff rule for large misses. The post details features including room codes for multiplayer phone participation, a TV-optimized display mode, live result gauges, and shareable results. It explains the technical implementation using Astro, TypeScript, Cloudflare Workers, and Durable Objects for managing game rooms, alongside privacy measures like automatic data deletion and cookieless analytics. The author reflects on lessons learned about focused game design, the appeal of real data, and privacy as a feature. Why I Built It Speed test night on the living-room TV became our family game night. Everyone shouts a guess before I press go. Here is the funny part. My kids get it right almost every time, and Dad loses on purpose. I started my career in performance testing, then moved into development, so I think I am allowed to be bad at guessing speeds. That is my excuse, anyway. At some point I realized this was a real game. It needed scores, secret guesses, and a proper winner. So I built it. How a Round Works A round takes about a minute. Here are the steps: Add players and pick a face for each one.Everyone locks in a secret download guess and a secret upload guess.Run a real speed test against Cloudflare's network.Compare the guesses with the result and score the closest players. That's it. No accounts, no sign-up, nothing to install. As shown below, the home screen keeps all of this on one page, with the live gauge in the middle and the player cards around it. A Sample Round (With Real Scoring) Let's ground this with one simple example. Say the speed test returns 240 Mbps download and 20 Mbps upload. Three players locked in their guesses: PlayerDownload guessUpload guessMom25022D&J20030Dad50050 Now the scoring rules kick in: Place points: first place earns 3 points, second earns 2, and third earns 1.Spot-on bonus: +1 point for each guess (download or upload) within 5% of the real value.50% cutoff: if your average miss is over 50%, you get no place points. Here is how our round shakes out: Mom is closest overall. Her download guess is only about 4% off, so she also gets the spot-on bonus. Total: 3 + 1 = 4 points.Mia lands second with 2 points.Dad guessed way too high. His average miss is far beyond 50%, so he gets 0 points. Dad loses on purpose, remember? In this round, the cutoff rule made sure he lost on merit too. The Features I am Most Happy With Once the basic game worked, I kept adding things that make it feel good in a real living room. Room codes for everyone's phones. Start a room, share a six-letter code or a QR code, and up to 12 players can join. Everyone guesses from their own phone. The server keeps all guesses private until the results are revealed, so nobody can peek and copy. The host also chooses which device runs the test. TV mode. This is a bigger gauge and bigger text, built for a couch ten feet from the screen. It supports fullscreen and uses a screen wake lock where the browser allows it, so the TV does not fall asleep mid-round. A live gauge with clear stages. You can follow the test through ping, download, and upload with live values and result cards. Watching the needle move is half the fun. Share the winner. The champion dialog shows the winner and the scores next to a Share result button. It uses the device share sheet or copies a link. Festive themes. Pick light, dark, or a seasonal theme, or leave it on Auto (by date) and let the calendar decide. Dev diagnostics. This one is for the curious. It shows the summary, latency samples, individual test requests, a bandwidth percentile, the test plan, and browser details. You can copy or download the latest run as JSON. Add ?dev=1 to the URL to open it. How It Is Built I kept the stack small and boring on purpose: Astro and TypeScript build the static site, which is served from Cloudflare Workers.@cloudflare/speedtest runs the real speed test in the browser.A GameRoom Durable Object manages the multiplayer rooms, using WebSocket hibernation so idle rooms cost almost nothing.A PlayStats Durable Object keeps the aggregate play counter.Game and scoring rules live in src/lib and are covered by Vitest tests. A few things I want to call out. Scoring is plain, testable code. Rules like the 50% cutoff and the tie-break are exactly the kind of thing that breaks quietly. Keeping them in src/lib with unit tests meant I could change the rules without fear. Durable Objects fit this problem well. One room is one object, and it holds the secret guesses until reveal. That is a natural fit for "a small piece of shared state that many phones talk to." Testing without waiting. Real speed tests take time, so I added a simulated mode. Append ?mock=1 for a short fake test. Use ?tv=1 to jump straight into the TV layout. To run it locally: Shell npm install npm run dev And to exercise rooms and the play counter, run the Worker locally: Shell npm run dev:worker Deployment is handled by Cloudflare Workers Builds. wrangler.jsonc runs npm run build, then npx wrangler deploy uploads dist as static assets. The Durable Object migrations run as part of that deploy. Everything below comes from the open-source repo at https://github.com/QAInsights/guessthespeed, so you can read the real code. Architecture at a Glance Astro and TypeScript build the static site, served by a Cloudflare Worker.worker/index.ts is the HTTP entry point. It creates rooms and upgrades WebSocket connections.GameRoom (worker/game-room.ts) is a Durable Object. There is one per room, and it owns the sockets and the stored state.src/lib/room.ts and src/lib/scoring.ts hold the rules as plain TypeScript functions with no Cloudflare APIs, covered by Vitest. The design rule I followed: the Durable Object is a thin shell. It loads state, calls a pure function, persists the result, and broadcasts. All the game decisions live elsewhere. One Durable Object Per Room A room code maps straight to an object with idFromName. Every phone that knows the code reaches the same instance, and every message for that room is handled in one place. That means there is no distributed locking to think about. Room codes are six letters from a 19-letter consonant alphabet, which gives roughly 47 million combinations. Creating a room means picking a code and asking its object to initialize itself. If the object already holds a room, it answers 409, and the Worker tries another code. TypeScript // worker/index.ts (simplified) for (let attempt = 0; attempt < 5; attempt += 1) { const code = generateRoomCode(/* crypto-backed random */); const hostToken = crypto.randomUUID(); const stub = env.ROOMS.get(env.ROOMS.idFromName(code)); const response = await stub.fetch("https://room.internal/init", { method: "POST", body: JSON.stringify({ code, hostToken, now: Date.now(), kind }), }); if (response.status === 201) return jsonResponse({ code, hostToken }, 201); if (response.status !== 409) return jsonResponse({ error: "room_creation_failed" }, 503); } return jsonResponse({ error: "room_code_unavailable" }, 503); The collision check needs no separate database. The object's own storage is the source of truth for whether a code is taken. The hostToken is a secret returned only to the creator. Presenting it on connect is what makes a socket the host. WebSockets That Survive Hibernation A room can sit idle for minutes while a family argues about the result, and I did not want to pay for an always-on object. The WebSocket Hibernation API lets the object be evicted from memory while the sockets stay connected. The catch is that anything held in memory is gone when it wakes up. Two things keep a room correct across hibernation: Game state lives in storage, not in fields. Each message handler starts with loadRoom().Per-socket identity lives in the socket attachment. After the first hello message, the object stores who this socket is with serializeAttachment. TypeScript // worker/game-room.ts (simplified) private async connectWebSocket(): Promise<Response> { if (!(await this.loadRoom())) return Response.json({ error: "room_not_found" }, { status: 404 }); if (!roomHasCapacity(this.ctx.getWebSockets().length)) return Response.json({ error: "room_full" }, { status: 503 }); const pair = new WebSocketPair(); this.ctx.acceptWebSocket(pair[1], ["room"]); return new Response(null, { status: 101, webSocket: pair[0] }); } private async handleHello(socket: WebSocket, message: Record<string, unknown>) { // ...validate clientId against a UUID pattern... const room = await this.loadRoom(); const attachment = { clientId: message.clientId, isHost: typeof message.hostToken === "string" && message.hostToken === room.hostToken, }; socket.serializeAttachment(attachment); this.sendState(socket, room, attachment); } The clientId is a random UUID the browser generates once and keeps in local storage. A returning phone sends the same ID, so the server can match it to its player after a dropped connection or a screen sleep. The reply to hello is always a full state snapshot, not a replay of missed events, so a reconnect cannot drift. One tradeoff worth naming: my per-socket rate limiter is an in-memory WeakMap, so it resets when the object hibernates. I treat the limits as a safety net, not accounting, so losing them on wake is acceptable. The Server Is the Referee Clients never change state directly. They send an action, and the server decides. TypeScript // worker/game-room.ts (simplified) const room = await this.loadRoom(); const result = applyAction( room, attachment.clientId, attachment.isHost, parsed, Date.now(), { onlineOwners }, ); if ("error" in result) { sendError(socket, result.error); this.sendState(socket, room, attachment); // resync the sender return; } await this.ctx.storage.put(ROOM_KEY, result.room); this.broadcastState(result.room); applyAction(room, clientId, isHost, action, now) returns either a new room or an error code. It takes now as a parameter instead of reading the clock, which makes it trivial to test. The room tests run actions in sequence and assert on the result, with no sockets involved. When an action is rejected, the sender gets the error plus the current state. That small habit fixed a whole class of stale-UI problems, because the client always snaps back to the truth. Keeping Guesses Secret If the server broadcast one shared state object, anyone could open dev tools and read everyone's guess. So the server never sends the raw room. It builds a view per recipient. TypeScript // src/lib/room.ts (simplified) const reveal = room.game.phase === "results" || room.game.phase === "champion"; players: room.game.players.map((roomPlayer) => { const { owner, ...player } = roomPlayer; delete player.guessedAt; return { ...player, guess: reveal || owner === clientId ? { ...player.guess } : { down: null, up: null }, mine: owner === clientId, }; }), TypeScript // worker/game-room.ts private broadcastState(room: Room): void { for (const socket of this.ctx.getWebSockets()) { const attachment = getAttachment(socket); if (attachment) this.sendState(socket, room, attachment); } } Each socket gets its own projection: your own guess, nulls for everyone else's, until the results phase. The same function strips internal fields like the owner and the guess timestamp. A room test asserts that other players' guesses stay hidden from both players and the host while guessing and testing, then appear at reveal. Cleaning Up Rooms With a Sliding Alarm Rooms should disappear after six hours of inactivity. My first version called setAlarm after every accepted action, which meant a storage write for every guess and every tap. A later commit replaced that with one rule: only schedule an alarm if none exists, and let the alarm re-arm itself. TypeScript // worker/alarm.ts export function alarmToSchedule( existingAlarm: number | null, deadline: number, ): number | null { return existingAlarm === null ? deadline : null; } TypeScript // worker/game-room.ts (simplified) async alarm(): Promise<void> { const room = await this.loadRoom(); if (room && Date.now() < room.updatedAt + ROOM_TTL_MS) { await this.ctx.storage.setAlarm(room.updatedAt + ROOM_TTL_MS); // still active return; } await this.ctx.storage.deleteAll(); for (const socket of this.ctx.getWebSockets()) { try { socket.close(1000, "Room ended"); } catch { continue; } } } The room stores updatedAt. When the alarm fires, it checks whether the room was active since it was scheduled. If yes, it sets a new alarm for the new deadline. If not, it deletes everything and closes the sockets. The result is a sliding time-to-live with at most one alarm in flight. Scoring as Pure Functions Scoring rules are exactly the kind of code that breaks quietly, so they live in src/lib/scoring.ts with no I/O. The core of it is a relative miss for each metric. TypeScript function metricMiss(guess: number | null, actual: number): number { if (!(actual > 0) || !Number.isFinite(actual)) return 1; return guess === null ? 1 : Math.abs(guess - actual) / actual; } // miss = (downMiss + upMiss) / 2 // bonus = Number(rounded(downMiss) <= spotOnPct) + Number(rounded(upMiss) <= spotOnPct) Looking back at our round: Mom's download miss is |250 - 240| / 240, about 4.2%, which is under the 5% spot-on limit. Her upload miss is 10%, so no second bonus. Place assignment skips players over the cutoff and handles ties the way a race does: tied players share a place, and the next place is skipped. JavaScript let place = 0; for (let index = 0; index < eligible.length; index += 1) { if (index === 0 || compare(eligible[index - 1], eligible[index]) !== 0) { place = index + 1; } eligible[index].place = place; eligible[index].placePoints = settings.placePoints[place - 1] ?? 0; } Two small details earned their own tests: Rounding before comparing. Misses are rounded to four decimals before ties are detected. Without that, floating-point noise decides who is tied.A zero or missing real speed. metricMiss returns a full miss instead of dividing by zero, so scores stay finite. The test file has cases for each rule: 3, 2, 1 points, an exactly 50% miss staying eligible, shared ties, the download-then-upload tiebreak, a player with no guess, and fewer than three players. TypeScript it("awards 3, 2, and 1 for first, second, and third", () => { const scores = scoreRound( [player("one", 100, 20), player("two", 90, 18), player("three", 70, 16)], actual, defaults, ); expect(scores.map(({ place, placePoints }) => [place, placePoints])).toEqual([ [1, 3], [2, 2], [3, 1], ]); }); Race Mode and the Server Clock Race mode changes the rule: the host starts the test, and players can guess while it runs. A guess locked in later earns a bigger share of the points, but a guess that misses the result earns nothing. The interesting design question is who owns the clock. Phone clocks drift and phones lag, so every timestamp comes from the Durable Object. Starting the test stores raceStartedAt: now, each guess stores guessedAt: now, and the result computes the timing bonus from the same server clock. TypeScript // src/lib/scoring.ts (simplified) const duration = endedAt - raceStartedAt; const fraction = duration === 0 ? 1 : Math.max(0, Math.min(1, (guessedAt - raceStartedAt) / duration)); const timingBonus = 0.5 + 0.5 * fraction; const total = Math.round(basePoints * timingBonus); The bonus ranges from 0.5 for a guess at the very start to 1.0 for a guess at the last moment. A table-driven test pins the edges and a rounding case. TypeScript it.each([ [0, 0.5, 25], [50, 0.75, 38], // 37.5 rounds up [100, 1, 50], ])("applies timing at %s%% of the test", (at, timingBonus, total) => { const [score] = applyRaceTiming( [baseScore], [{ id: "racer", guessedAt: at }], 0, 100, ); expect(score).toMatchObject({ basePoints: 50, timingBonus, total }); }); Applied to our round, with place points scaled to 30, 20, 10 in race mode: Mom's base is 30 plus a 10-point spot-on bonus, which is 40. If she locks in at 80% of the test, the bonus is 0.9, and she scores 36. Mia's base is 20, and locking in at 20% gives 0.6, so she scores 12. Waiting is rewarded, as long as you do not wait too long. There is one more leak to close. If players could watch the live speed climb, they would copy it in the last second. In race mode, the progress relay forwards only the step counters and drops the measurements. JavaScript const keys = race ? ["step", "steps"] : ["mbps", "pingMs", "step", "steps", "bytes"]; The relay also accepts progress only from the device assigned to run the test, and it is throttled to ten updates per second. Challenges and Lessons A few things that needed more than one attempt: Redundant alarm writes. Resetting the alarm on every action was correct but wasteful. The sliding alarm in section 7 fixed it.Host handover edge cases. Handing the host role to another phone sounds easy. The rules are: only the host can do it, not while a test is happening, not to an offline player, and not to the same device. The offline case shipped as a follow-up fix after a handover to a disconnected player left a room with no live host.Interrupted tests. If the device running the test reloads mid-test, the room can get stuck in the testing phase. A small pure function, interruptedTestStep, decides what the client does next: resend a result it already holds, or abort the round once.Abuse limits on a public endpoint. Messages are capped at 2 KB, each socket at 20 messages per second, rooms at 12 players, and WebSocket upgrades check the origin. Worker request bodies are size-bounded too.Testing real-time behavior without waiting. Because the rules are pure, most tests never open a socket. For the browser, a ?mock=1 flag runs a short fake speed test, so a full round takes seconds. What I would tell someone starting a similar project: Keep the Durable Object thin and move every decision into functions that take now as an argument.Never broadcast shared state when players should not see each other's data. Project a view per socket.Put the identity in the state in storage, and assume the object can wake up empty at any time.Prefer one self-rescheduling alarm over resetting a timer on every message. Privacy Decisions A game for families should not feel like a tracking machine, so I made a few choices early: In single-screen mode, settings, player names, and scores stay in your browser's local storage.Room names, faces, guesses, and scores are deleted after six hours without activity.The play counter stores totals only, and Cloudflare Web Analytics is cookieless.The speed test itself runs in your browser against Cloudflare's network. You can read the full details on the privacy page. Try It On Your Own Wi-Fi Here is how to play your first round: Head to https://guessthespeed.com/.Add everyone playing and pick a face for each person.Lock in your secret guesses (or have people join from their phones with the room code).Press go and watch the gauge.Check the scores and share the champion. If you want ideas for a full evening, there is a family game-night guide on the site. There is also a guide that explains how the real speed test works, and a 30-minute classroom activity for kids. What I Learned A few takeaways from this build: Small, focused games beat big ideas. The whole concept fits in one sentence, and that makes every decision easier.Real network data makes people care. A made-up number is boring. A real measurement on your own Wi-Fi starts arguments, which is exactly what a game needs.Privacy is a feature. Deleting rooms after six hours and keeping analytics cookieless cost me very little, and they make the game easier to recommend to other families. So that is Guess the Speed. A speed test, a few secret guesses, and a family that cannot stop arguing about the Wi-Fi. Which of your guesses would be furthest off: your download speed or your upload speed? Let me know in the comments below. Happy gaming!

By NaveenKumar Namachivayam

CORE

Spring Security JWT Validation Best Practices

A few months into a large-scale legacy-to-cloud migration for a Fortune 50 insurance carrier, I found something that had been sitting quietly in our codebase for a while: application code that was manually re-checking JWT claims — signature, issuer, audience, expiry — that Spring Security's OAuth2 Resource Server had already validated at the framework level, before the request ever reached that code. Nobody had done this carelessly. Each addition, on its own, looked like good security hygiene. That's exactly why it survived review for as long as it did. Why This Happens So Easily A JWT is a self-contained, signed structure carrying its own claims. Validating untrusted input by hand is a deeply ingrained instinct for engineers, and for good reason — it usually is the right instinct. The problem is visibility. When a developer writes if (claims.getExpiration().before(now)), a reviewer can point at that line and confirm the check exists. What's much harder to see is the framework machinery quietly doing the exact same job, correctly, one layer up. Spring Security's OAuth2 Resource Server support validates signature, expiry, issuer, and audience (when configured) in line with RFC 7519 and RFC 8725. Its JwtAuthenticationProvider runs these checks on every incoming token, and as the identity provider rotates its signing keys, it automatically re-fetches and re-validates against the new keys — no application code involved. Almost none of this is visible unless you've actually read the framework internals, which is exactly why teams tend to reimplement it instead of trusting it. The audience claim was the clearest example in our case. It identifies which resource server a token is meant for, and the framework exposes it as a single config property. We'd been treating it as something that needed hand-written comparison logic — when really, the value had just never been wired into configuration in the first place. The fix wasn't more code. It was one line of YAML. What We Changed We removed the manual validation entirely and let Spring Security's Resource Server own both authentication and authority mapping: YAML spring: security: oauth2: resourceserver: jwt: issuer-uri: https://login.microsoftonline.com/{tenant-id}/v2.0 audiences: api://{application-id} authorities-claim-name: roles authority-prefix: GRANT_ That block replaced manual signature verification, issuer checking, expiry checking, audience checking, and — on Spring Boot 3.3+ — the hand-written role-to-authority converter too. No JwtAuthenticationConverter, no custom Bean. The framework builds the authority mapping straight from the roles claim and applies whatever prefix you configure. (GRANT_ isn't a framework requirement, by the way — pick anything, as long as it matches what your @PreAuthorize expressions expect downstream.) The Objection We Had to Take Seriously This didn't get waved through. It took a few weeks of real back-and-forth, because the pushback was a legitimate security position, not just resistance to change: don't trust any single layer — verify independently, everywhere. Concretely: if issuer-uri got misconfigured — pointed at a stale tenant, or worse, something attacker-influenced — the Resource Server would faithfully validate every token against the wrong authority, and nothing in a minimal-code setup would catch that. A hardcoded, independent check plausibly would. Nobody disputed that the risk was real. The actual question was which layer should own catching it. Our answer: a hardcoded application-layer check and a misconfigured deployment value are different failure modes, touched by different people, on different release cadences. Duplicating validation in code doesn't actually protect against a bad issuer-uri — it just adds a second place that can also drift out of sync. Instead, we added a CI/CD gate that asserts the deployed issuer-uri and audience resolve to the expected tenant, and fails the pipeline on mismatch, on every deploy. Is that still "a second check"? Sure. But it fails differently, and that difference is the whole argument: a broken pipeline gate is a loud, visible build failure someone has to explicitly override. A stale hardcoded value in application code can sit silently wrong for months, because nothing forces anyone to look at it again. What I Can Honestly Say Held Up I want to be straight about this: since the change shipped, we haven't had a key rotation or a deployment misconfiguration that actually tested either mechanism in anger. So I can't tell you the CI/CD gate has caught a real incident, because it hasn't had to yet. What I can say: the manual validation code is gone, no duplicate signature/issuer/audience/expiry logic remains anywhere in the codebase, and in the months since, nothing attributable to the change has come up in review. That's evidence the change didn't introduce a regression — not proof it's been battle-tested under a live failure. Worth being honest about the difference. The Takeaway Treat hand-rolled token validation as a prompt to investigate, not an automatic red flag. In review, ask what the framework's resource-server config already checks before accepting new validation code as necessary.If a function exists purely to re-derive something the framework already validates, the fix is usually a config line, not a code change.Defense-in-depth concerns about misconfiguration are legitimate — answer them directly. Often the better answer is verifying config at deploy time (CI/CD gating), not re-validating tokens at runtime.The two approaches fail differently, and that's the actual argument for preferring the config-time check: a broken gate is loud, a stale hardcoded check is silent. This example is Spring Security and Entra ID, but the underlying review question — what does the framework already check before I add code that checks it again — travels well to any OAuth2/OIDC resource-server setup.

By Alka Nimje

Java 27 Makes G1 the Default Everywhere: How to Test Your Small Containers Before You Upgrade

When JDK 27 was released on 15 September 2026, most of the attention went to compact object headers and post-quantum TLS. JEP 523 received less coverage, partly because its description sounds like housekeeping: G1 becomes the default garbage collector in all environments. For most applications, that changes nothing. For a specific class of container, it changes the collector silently, with no warning in the logs and no failure at startup. This article covers what the fallback did, which workloads lose it, how to identify affected services, and how to run a comparison that produces usable numbers rather than noise. What the Serial GC Fallback Did G1 has been the nominal default since Java 9. What is less widely known is that HotSpot inspected the machine at startup and overrode that choice in constrained environments. The JVM selected Serial GC when either condition was true: ConditionThresholdAvailable processors1Physical memoryLess than 1,792 MB The reasoning was sound at the time. Serial GC is single-threaded with a very small native memory footprint. On a single-core machine, a concurrent collector spends time competing with application threads for the only available core, and the bookkeeping structures G1 maintains were expensive relative to a small heap. G1 has improved substantially since then. Its native memory footprint has been reduced across several releases, and JDK 26 delivered JEP 522, which removed synchronization overhead from G1's write barriers by introducing a second card table. JEP 523 concludes that work. The fallback is removed, and G1 is now selected in every environment regardless of CPU count or available memory. Serial GC remains fully supported and can still be selected explicitly: Shell java -XX:+UseSerialGC -jar application.jar Applications that already specify a collector, whether G1, ZGC, Parallel, or Shenandoah, are unaffected by this change. Which Workloads Are Affected The change only reaches services that never had an explicit collector flag and that run in a constrained environment. In practice, this describes a predictable set of deployments: Sidecar containers. Log shippers, metrics agents, and service mesh components typically receive a fraction of a CPU and a few hundred megabytes of memory. They are rarely tuned.Batch and queue workers. Short-lived jobs scheduled onto whatever capacity is available.Low-traffic internal services. Admin APIs, configuration services, and internal tooling that are deliberately given small pods.CI runners and development containers. Environments where JVM flags are seldom set because the workload is not production.Pods with fractional CPU limits. This case is easy to miss. A container configured with cpu: 500m may report a single available processor to the JVM, which placed it in Serial GC territory on earlier releases. These are the workloads least likely to appear in a performance test plan, which is precisely why the change deserves deliberate verification. Step 1: Identify Your Current Collector Before changing anything, determine which of your services are actually running Serial GC today. The GarbageCollectorMXBean interface exposes this directly: Java import java.lang.management.GarbageCollectorMXBean; import java.lang.management.ManagementFactory; public final class RuntimeFacts { public static void main(String[] args) { Runtime rt = Runtime.getRuntime(); System.out.printf("java version : %s%n", Runtime.version()); System.out.printf("available cpus : %d%n", rt.availableProcessors()); System.out.printf("max heap MB : %,d%n", rt.maxMemory() / (1024 * 1024)); for (GarbageCollectorMXBean gc : ManagementFactory.getGarbageCollectorMXBeans()) { System.out.printf("collector : %s%n", gc.getName()); } } } The collector names identify which implementation is active: CollectorReported MXBean namesSerial GCCopy, MarkSweepCompactG1G1 Young Generation, G1 Concurrent GCParallelPS Scavenge, PS MarkSweepZGCZGC Cycles, ZGC Pauses For a Spring Boot application, registering this as a startup listener puts the information into your existing log aggregation without any additional infrastructure: Java import org.springframework.boot.context.event.ApplicationReadyEvent; import org.springframework.context.event.EventListener; import org.springframework.stereotype.Component; import java.lang.management.GarbageCollectorMXBean; import java.lang.management.ManagementFactory; import java.util.stream.Collectors; @Component public class GcFactsLogger { @EventListener(ApplicationReadyEvent.class) public void logCollector() { Runtime rt = Runtime.getRuntime(); String collectors = ManagementFactory.getGarbageCollectorMXBeans() .stream() .map(GarbageCollectorMXBean::getName) .collect(Collectors.joining(", ")); System.out.printf( "JVM %s | cpus=%d | heap=%,dMB | gc=%s%n", Runtime.version(), rt.availableProcessors(), rt.maxMemory() / (1024 * 1024), collectors); } } Deploy this across your estate, then filter your logs for entries reporting Copy as a collector name. That result set defines the scope of your testing. Step 2: Understand the Native Memory Difference Discussions about garbage collector selection usually center on pause times. For this particular change, native memory consumption is the more consequential metric. G1 maintains bookkeeping structures that Serial GC does not require, including remembered sets, card tables, and per-region metadata. This memory is allocated outside the Java heap, which means it counts against a container memory limit without appearing in heap metrics. Measurements published after the JDK 27 release place G1's native memory overhead at approximately 43 MB, compared with roughly 1.7 MB for Serial GC. Ankur Mundra measured both collectors on Spring Boot 4.1 within real cgroup limits and reported these figures. The absolute difference is modest. The proportional impact depends entirely on container size: Container memory limitApproximate G1 overhead share1024 MB4%512 MB8%256 MB16% On a tightly sized container, this overhead is consumed before the application allocates its first object. Where memory limits were set with minimal headroom, the change alone can produce OOM kills after an upgrade that altered nothing else. Native Memory Tracking provides the per-category breakdown if you want to confirm the figure in your own environment: Shell java -XX:NativeMemoryTracking=summary \ -XX:+UnlockDiagnosticVMOptions \ -jar application.jar Then, against the running process: Shell jcmd <pid> VM.native_memory summary The GC section of the output reports committed native memory attributable to the collector. Step 3: Run a Controlled Comparison A meaningful comparison requires the smallest container size you actually deploy, representative load, and a run long enough to reach steady state. A thirty-second run measures startup behavior rather than collection behavior. Capture GC logs for both configurations: Shell # Pre-Java 27 behaviour java -XX:+UseSerialGC \ -Xlog:gc*:file=serial.log:time,uptime,level,tags \ -jar application.jar # Java 27 default java -Xlog:gc*:file=g1.log:time,uptime,level,tags \ -jar application.jar Three measurements matter, and the third is the one most often omitted: Total pause time. Sum the pause durations across the run rather than comparing individual collections. Serial GC performs fewer, longer pauses; G1 performs more, shorter ones. Only the aggregate is comparable.Tail latency. G1 runs concurrent phases. On a single-core container, those phases contend with request handling for the only available processor. This effect is invisible in a mean and clearly visible at p99.Resident set size under load. This is where the native memory difference becomes operationally relevant. Sample RSS throughout the run rather than reading it once at startup: Shell while true; do ps -o rss= -p "$PID" | awk '{printf "%s %d MB\n", strftime("%H:%M:%S"), $1/1024}' sleep 5 done If RSS approaches your container limit under G1 where it did not under Serial, you have found the problem before it found you. Step 4: Pin the Collector When G1 Loses If measurement shows that G1 performs worse on a constrained workload, the appropriate response is not to defer the JDK upgrade. It is to state the collector explicitly: Shell java -XX:+UseSerialGC -jar application.jar This is a supported configuration. Serial GC is neither deprecated nor scheduled for removal, and JEP 523 explicitly preserves it. In a Kubernetes deployment, express the choice where it will be reviewed: YAML spec: containers: - name: batch-worker env: - name: JAVA_TOOL_OPTIONS value: "-XX:+UseSerialGC" There is an argument for setting the flag explicitly even where G1 performs well. An explicit collector choice is self-documenting. A future engineer reading the manifest can see that a decision was made and tested, rather than inferring what the JVM selected at runtime. Conclusion JEP 523 is a reasonable change. G1 has genuinely closed the gap that justified the fallback, and removing a hidden conditional from the JVM's startup logic simplifies the mental model for everyone. The operational consideration is narrower: a set of small, rarely tested containers will change collectors on upgrade without any signal that it happened. Identifying those containers takes one log statement. Testing them takes one load run. The cost of skipping both is a latency regression or an OOM kill that appears unrelated to any change your team made. Java 27 is not a long-term support release, and many teams will remain on Java 25. That does not remove the relevance of this work. The same default will arrive with the next LTS, and evaluating it on a short-term release carries considerably less risk than discovering it mid-migration. Further Reading JEP 523: Make G1 the Default Garbage Collector in All EnvironmentsJEP 522: G1 GC: Improve Throughput by Reducing SynchronizationJDK 27 Release Notes

By Garima Agarwal

MySQL Instant DDL: Metadata Locks, Waiting Writers, and Query Pileups

A transaction reads one row and then remains open. Later, a deployment adds a column using ALTER TABLE ... ALGORITHM=INSTANT. The migration waits. A new request issues an ordinary SELECT against the same table, and that request waits too. The original query has already finished. The schema change requires no table rebuild. The new query accesses a different row. Yet all three sessions participate in the same blocking sequence. The explanation sits above InnoDB’s row-lock layer. The open transaction retains a metadata lock, the schema change requests an incompatible lock, and that pending request can make later queries queue behind it. Instant DDL reduces the work required to change a table. It does not eliminate the coordination required to begin or complete that change. Reproduce the Queue With Three Connections Use a disposable database and three independent MySQL connections. A tiny table keeps data volume out of the explanation: MySQL CREATE DATABASE mdl_demo; CREATE TABLE mdl_demo.orders ( id BIGINT PRIMARY KEY, status VARCHAR(20) NOT NULL ) ENGINE=InnoDB; INSERT INTO mdl_demo.orders VALUES (1, 'NEW'), (2, 'NEW'), (3, 'DONE'); In Session 1, start a transaction and read one row: MySQL START TRANSACTION; SELECT * FROM mdl_demo.orders WHERE id = 1; -- Leave the transaction open. The query returns immediately. Do not commit yet. In Session 2, request the schema change: MySQL ALTER TABLE mdl_demo.orders ADD COLUMN source VARCHAR(32) NULL, ALGORITHM=INSTANT; After the ALTER begins waiting for a metadata lock, issue another read in Session 3: MySQL SELECT * FROM mdl_demo.orders WHERE id = 2; This later read can also wait, despite accessing a different row. Committing Session 1 allows the migration to proceed, followed by the queued read. MySQL’s verified Bug #106480 documents this three-connection behavior for MySQL 8.0. Test the sequence against your deployed version rather than assuming every DDL operation has identical locking details. Separate Execution Cost From Lock Admission For supported operations, ALGORITHM=INSTANT changes metadata without rebuilding the table. Whether a particular alteration qualifies depends on the operation, table characteristics, and server version. MySQL publishes these restrictions in its online DDL operation matrix. That optimization removes potentially expensive data movement. It does not promise a maximum elapsed time. The migration’s duration includes time spent acquiring required locks. A metadata change that executes quickly after admission can still wait behind an existing transaction for minutes. This distinction should influence deployment planning. Estimating migration risk from table size alone misses the failure demonstrated here. A three-row table can experience the same admission problem as a much larger one. The operational question is therefore twofold: how expensive is the change once admitted, and how long may it wait while affecting other requests? The Blocker Can Be Idle For tables used within a transaction, MySQL retains transactional metadata locks until the transaction ends. Finishing a SELECT does not necessarily release them. See MySQL’s metadata-lock lifetime documentation. Consequently, searching only for slow queries can miss the session responsible. Application code commonly stretches transactions beyond their SQL work. An endpoint reads data, calls a remote service, performs additional processing, and commits afterward. The database query takes milliseconds, but the transaction remains open throughout the external work. A connection can therefore appear idle while still preventing a schema change. Measure transaction age separately from query duration. Review transaction boundaries around network calls, user interaction, and background work. Moving unrelated operations outside a transaction can reduce exposure without changing the migration itself. Preflight checks help identify existing blockers, but they cannot guarantee that another transaction will not start immediately afterward. Why Later Readers Wait Without the pending ALTER, the two reads in the example could coexist. The DDL request changes the scheduling situation. MySQL normally prioritizes metadata write-lock requests over read-lock requests, with documented exceptions. That policy prevents a continuous stream of readers from indefinitely starving a writer. The priority rules appear in the metadata-lock acquisition documentation. On a busy table, this creates a widening queue. One open transaction delays the migration, and the waiting migration delays newly arriving work. Those requests can occupy application connections until the pool fills. Subsequent requests may then wait for a connection before reaching MySQL at all. This is why a migration incident can look like general database saturation even when CPU and storage activity remain modest. The database is coordinating access, not necessarily executing expensive queries. Inspect Ownership and Pending Requests Start with a diagnostic query that connects waiting sessions to potential blockers: MySQL SELECT object_schema, object_name, waiting_pid, waiting_lock_type, blocking_pid, blocking_lock_type FROM sys.schema_table_lock_waits WHERE object_schema = 'mdl_demo' AND object_name = 'orders'; The sys view documentation describes these session and lock relationships. Use the result alongside transaction age and application ownership. Identify which connection belongs to the migration runner and which belongs to the application. For lower-level investigation, inspect performance_schema.metadata_locks and correlate its owner thread identifiers with performance_schema.threads. Distinguish granted locks from pending requests rather than treating every displayed lock as an active blocker. Do not automatically terminate the oldest connection. It may hold legitimate work whose rollback is expensive. When the migration is deferrable, removing its waiting request may be the less disruptive intervention. Set the Correct Timeout The familiar innodb_lock_wait_timeout controls InnoDB row-lock waits. It does not bound the metadata-lock wait responsible for this incident. Configure the migration connection instead: MySQL SET SESSION lock_wait_timeout = 3; ALTER TABLE mdl_demo.orders ADD COLUMN source VARCHAR(32) NULL, ALGORITHM=INSTANT; In MySQL 8.4, lock_wait_timeout governs metadata-lock acquisition attempts and defaults to 31,536,000 seconds. The timeout applies separately to each acquisition attempt, so it is not a universal three-second statement deadline. See the server variable reference. Three seconds is illustrative. Choose a budget based on acceptable application disruption. Apply the setting on the actual connection executing the migration. Setting it on a separate administrative connection has no effect on the migration session. A dedicated connection also avoids leaving unexpected timeout settings in an application pool. Verify That Timeout Releases the Queue Extend the reproduction using a fresh table definition. Keep Session 1’s transaction open. Start the migration with a short metadata-lock timeout, then issue Session 3’s read while the migration is waiting. When the ALTER times out and its pending request is removed, the later read should proceed in this isolated scenario even though Session 1 remains open. That observation identifies the queue boundary. The two reads were compatible; the waiting schema change prevented the later one from advancing. A failed migration must remain visible to deployment automation. Do not mark it complete merely because application traffic recovered. Retry with backoff and a bounded attempt count rather than repeatedly recreating the same queue. After a client disconnect or ambiguous execution result, inspect the actual schema before retrying. Make Admission Part of the Deployment Contract A production migration policy should specify the permitted algorithm, metadata-lock wait budget, retry behavior, and conditions for postponement. Monitor pending DDL alongside transaction age, connection-pool usage, and request latency. Alerting only on migration duration can detect the problem after application capacity has already been consumed. Test both successful admission and blocked admission before deployment. The latter proves that the migration yields when production conditions are unfavorable. Instant DDL remains valuable because it avoids unnecessary table reconstruction. Its safety depends on the surrounding lifecycle: transactions must finish, lock requests must be admitted, and waiting migrations must have a bounded opportunity to interfere with traffic.

By Sagar Bonthu

A Non-Expiring AKS Deployment Token Could Read Secrets Across Three Namespaces

A security review started after connection strings were visible through a Kubernetes management tool. The first request sounded simple: remove Secrets from the cluster and move the values to Azure Key Vault. That request focused on which system stored the values. The access review found a wider problem. Kubernetes stored a deployment service account's legacy token in a kubernetes.io/service-account-token Secret. The token had no expiry in its normal use, and RoleBindings allowed the same identity to operate in three application namespaces. The Role used resources: ["*"] across several API groups and allowed get, list, watch, create, update, patch and delete. The Role included the core API group, so the token could read every Secret in each of those namespaces. The review found no evidence of token misuse. It found an avoidable access path with a long lifetime and a larger scope than the pipeline needed. Moving application passwords to Key Vault would improve their source of truth and rotation. It would not revoke this token, narrow its RBAC permissions or remove the cross-namespace bindings. Those controls needed their own fix. Why the Kubernetes Secrets Existed The application images carried JSON configuration files for local development. Database, cache and message-broker endpoints pointed to localhost. Those defaults were useful on a developer machine and unusable inside the cluster. The shared Helm chart supplied the deployment values at runtime. It rendered an Opaque Secret from a secrets map, then loaded the Secret into the application container through envFrom. The reduced chart looked like this: YAML {{- if .Values.secrets } apiVersion: v1 kind: Secret metadata: name: {{ include "app.fullname" . }-secrets type: Opaque stringData: {{- range $key, $value := .Values.secrets } {{ $key }: {{ $value | quote } {{- end } {{- end } The Deployment then loaded every entry as an environment variable: YAML envFrom: - secretRef: name: {{ include "app.fullname" . }-secrets The application configuration placed environment variables after the file-based settings, so the runtime values won. That ordering made the Kubernetes Secret part of the configuration path. Removing it without replacing that path would leave the containers pointing at local services. Figure 1. The existing deployment used a Kubernetes Secret as the runtime override for local file defaults. The next question was, "Which components can retrieve the value, and how long does each form of access last?" Key Vault Does Not Always Remove the Kubernetes Secret The Azure Key Vault provider for the Secrets Store CSI Driver supports more than one consumption model. A pod can read mounted files directly from a CSI volume. This path can avoid creating a native Kubernetes Secret. The application still needs a configuration method that reads those files and responds safely when their contents rotate. The driver can also copy mounted values into a native Kubernetes Secret through the secretObjects field of a SecretProviderClass. Microsoft documents this as secret synchronisation. It is useful when an application already consumes secretKeyRef or envFrom, which was the case here. It also means that moving the source value to Key Vault does not remove the Secret object from Kubernetes. YAML apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: app-vault spec: provider: azure secretObjects: - secretName: app-runtime-secrets type: Opaque data: - objectName: database-password key: Database__Password The choice affects rotation too. Microsoft notes that an existing process does not receive updates when a container consumes a Kubernetes Secret through an environment variable. The team must restart the pod before the process receives the new value. Key Vault remains a sound source for secrets. The important point is narrower: the storage migration and the Kubernetes access review solve different problems. The Service Account Crossed Namespace Boundaries The next part of the review followed the identities that could read Secrets. Kubernetes allows a RoleBinding in one namespace to name a ServiceAccount from another namespace. That feature is useful for controlled cross-namespace work. It also means that checking only the ServiceAccount's home namespace can understate its reach. The review found a layout equivalent to this generalized example: YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pipeline-deployer namespace: environment-b rules: - apiGroups: ["", "apps", "batch", "autoscaling", "networking.k8s.io", "policy"] resources: ["*"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pipeline-deployer-from-a namespace: environment-b subjects: - kind: ServiceAccount name: pipeline-deployer namespace: environment-a roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: pipeline-deployer Two bindings of this form extended the identity from its home namespace into two more namespaces. The same bearer token could therefore authenticate once and use the permissions that each Role granted. Figure 2. Two cross-namespace bindings extended one long-lived credential into three namespaces. The identity was not a cluster administrator. The Role listed its API groups, and it did not include rbac.authorization.k8s.io. The token could not grant itself new RoleBindings through this Role. That limit mattered, but it did not make the existing scope acceptable. The core API group still covered Secrets, and the wildcard resource rule covered much more than the deployment needed. The following authorization check avoids printing any secret value: Shell identity="system:serviceaccount:environment-a:pipeline-deployer" for namespace in environment-a environment-b environment-c do printf '%s: ' "$namespace" kubectl auth can-i list secrets \ --namespace "$namespace" \ --as "$identity" done For the reviewed identity, the effective result was yes in all three namespaces. The Token Type Made the Scope Harder to Contain The service account had one manually created Secret of type kubernetes.io/service-account-token. Kubernetes describes this as a legacy mechanism that provides a long-lived credential. The current documentation recommends the TokenRequest API for short-lived tokens instead. The difference changes the failure window. A deployment job can request a token that expires shortly after the job finishes. If someone copies a static token into a CI/CD service connection, that credential can remain useful until somebody deletes or rotates it. An old export, agent workspace or copied configuration can preserve the same access during that period. The cluster version already supported projected service account tokens. Another environment-specific service account had no legacy token Secret, which confirmed that the static pattern was not present in every namespace. The First Fix Reduces the Token's Reach The quickest containment step is to remove the two cross-namespace RoleBindings. That immediately reduces the token's potential reach from three namespaces to one without changing the applications. Each environment should then have its own deployment identity, Role and RoleBinding. A pipeline for one namespace should not borrow the service account from another. This keeps the credential, audit trail and revocation boundary aligned with the environment it deploys. The desired shape is simple: Plain Text pipeline A uses a short-lived token for its Role in namespace A pipeline B uses a short-lived token for its Role in namespace B pipeline C uses a short-lived token for its Role in namespace C These deployments require no cross-namespace subject. Figure 3. The target design aligns one service account and its short-lived token with one namespace. A Short-Lived Token Replaces the Static Credential Kubernetes can issue a bounded token with the TokenRequest API. kubectl create token is a convenient way to exercise that API: Shell kubectl create token pipeline-deployer \ --namespace environment-a \ --duration 15m The command covers only one part of a CI/CD design. The runner still needs a trusted way to request the token. The pipeline should receive the credential at run time, the requester should limit its audience, and the token should expire soon after the deployment window. The service connection must stop storing the old bearer token before that change has value. After the new path succeeds, delete the legacy token Secret and confirm that the API rejects the old credential. Explicit RBAC Replaces Wildcard Access resources: ["*"] made future chart changes part of the role automatically. Kubernetes warns that wildcard rules can grant access to resource types that future API extensions add. A deployment role should name the resource types and verbs the release actually uses. The rendered manifests provide the initial resource list: Shell helm template app ./chart \ --namespace environment-a \ --values values.environment-a.yaml \ > rendered.yaml Inventory every apiVersion and kind in that output. Then compare the result with deployment calls in API audit logs or a dry run. The resulting Role may need access to Deployments, Services, ConfigMaps, Secrets, Jobs, Ingresses and autoscaling resources. It should not receive every resource merely because the chart might use one later. If a separate controller owns runtime Secrets, the deployment identity can lose Secret write access as well. If Helm still renders those Secrets, the role needs the exact verbs required by that release process. The policy must match the delivery design instead of hiding a required permission until the next deployment fails. The Review Found Other Paths to the Same Values The cluster token was the widest durable access path that the review found. Several pipeline variables also stored credentials as ordinary variables. Azure DevOps treats secret variables as protected resources. It encrypts their values at rest and masks exact values in logs. A task can still send a value elsewhere, and masking does not always hide partial strings. Marking the variables as secret still removes routine plaintext visibility from the library interface and logs. The remediation plan combined four related controls: Mark credential variables as secret and restrict which pipelines can use each variable group.Remove routine human permission to read Kubernetes Secrets through cluster tools.Rotate the database, cache and broker credentials that broader access paths exposed.Move non-sensitive settings out of the Helm secrets map and into a ConfigMap. Moving non-sensitive settings to a ConfigMap makes the review surface honest. Feature flags, log levels and endpoint names do not need to sit beside credentials merely because the original release populated only one values map. Validation Uses the Identity's Point of View The validation must test permissions from the identity's point of view. For each deployment identity, build an explicit matrix: Plain Text own namespace other namespaces read required objects yes no apply release objects yes no read application Secrets only if required no modify RBAC no no Then verify the old paths: the deleted legacy token fails authenticationeach identity returns no for Secret access outside its namespacethe pipeline can still deploy and roll backvariable-group values appear masked in the interface and logscluster users without an operational need cannot reveal Secret valuesrotated application credentials work and the previous values fail This sequence keeps the deployment usable while shrinking the access path in measurable steps. Secret Location and Secret Access Need Separate Reviews The visible Kubernetes Secret started the investigation, but deleting that object first would have left the long-lived deployment credential untouched. Using CSI secret synchronization could even have recreated the same type of object from a better source. The useful review order was identity, scope, lifetime, source and rotation. It exposed one token that crossed three namespace boundaries, wildcard permissions that exceeded the release surface, and pipeline values that lacked basic secrecy controls. Key Vault can improve storage and rotation. Short-lived credentials and narrow RBAC control who can reach the Kubernetes API. Environment-specific identities contain the effect of a leaked token. These controls work together, and none of them can substitute for the others. References Kubernetes documentation: SecretsKubernetes documentation: Service AccountsKubernetes documentation: Managing Service AccountsKubernetes documentation: RBAC good practicesMicrosoft Learn: Azure Key Vault provider configuration options for AKSMicrosoft Learn: Access Key Vault through the CSI Driver on AKSMicrosoft Learn: Secrets in Azure PipelinesMicrosoft Learn: Manage Azure Pipelines variable groups

By Ahmed Tariq

Deploying Small Language Models on Sovereign Cloud

The enterprise paradigm for artificial intelligence has fractured. While massive cloud-hosted large language models continue to dominate consumer applications, highly regulated sectors like defense, aerospace, and high-tier finance are actively blocking external API calls to public AI services due to strict data sovereignty mandates. The solution is migrating toward small language models (SLMs) localized at the tactical edge. Deploying an SLM into an air-gapped, "sovereign cloud in a box" environment requires absolute cryptographic certainty over the supply chain. A compromised base image running deep inside a disconnected network invalidates the entire security perimeter. The recent release of Docker Hardened Images (DHI) and the expansion of Google Distributed Cloud (GDC) air-gapped appliances give backend engineers the exact toolchain required to host models like Google's Gemma with zero-trust enforcement. This technical guide details an end-to-end framework for curating a zero-CVE container, packing a fine-tuned Gemma SLM, and pushing it to an isolated GDC cluster. The Target Infrastructure: GDC Air-Gapped Google Distributed Cloud air-gapped is an integrated hardware and software platform designed to operate physically disconnected from the internet. It allows enterprises to deploy container-based workloads and managed services like Vertex AI in completely isolated environments. The architecture supports rigorous compliance frameworks, including NIST, ISO 27001, and SOC II. Notably, GDC offers packaged generative AI solutions enabled by Gemma, providing local, conversational search capabilities on-premises. However, when deploying custom fine-tuned checkpoints or building the Java microservices that handle backend inference orchestration- a core focus of my daily work as an IEEE Senior Member and backend developer — we cannot rely on standard public container registries. The container must be hardened from the base layer up. Base Image Curation With Docker DHI To build our localized SLM container, we start with Docker Hardened Images. DHI provides minimal, secure, production-ready container images maintained by Docker. Rather than blindly pulling a generic Alpine or Debian base image and running a vulnerability scanner after the fact, DHI guarantees a hardened foundation. Using the docker dhi CLI, available in Docker Desktop 4.65 and later, we can inspect and mirror these hardened artifacts directly to our enterprise registries. Shell # Verify local DHI CLI installation docker dhi --help # Mirror a hardened Python or Java base image to your local org docker dhi mirror start --org enterprise-registry \ dhi/python,enterprise-registry/dhi-python By switching our Dockerfile FROM instruction to enterprise-registry/dhi-python, we structurally eliminate the attack surface before installing our inference engine dependencies. Generating the SLM Artifact and SBOM As I detailed in my Springer book chapter Introduction to Trustworthy AI, accountability in machine learning relies heavily on technical robustness and traceability. We cannot deploy a black-box container into an air-gapped network. We must generate a cryptographic Software Bill of Materials (SBOM) that catalogs the exact SLM weight hashes, the inference engine binaries, and the OS packages. The DHI CLI handles this transparently. Once you package your Gemma model weights into the hardened container, you can retrieve the exact attestations and SPDX SBOMs attached to the base images. Shell # Retrieve the SLSA provenance and SPDX SBOM for audit docker dhi attestation get dhi/python:3.11 sha256:<digest> -o provenance.json This JSON payload is ingested into the local CI/CD pipeline. The pipeline validates that no unauthorized dependencies were injected during the model packaging phase. If the cryptographic signature fails, the build halts immediately. Deploying to the Sovereign Edge With a cryptographically signed, DHI-backed container housing our Gemma SLM, we transition to the deployment phase on GDC air-gapped. Because GDC air-gapped operates a single cluster encompassing bare metal nodes without internet access, deployment relies on a physical handoff. The hardened container image, the validated SBOM, and the deployment manifests are loaded onto a portable storage device. Once transferred to the secure facility, the infrastructure operator pushes the image to the local, disconnected GDC container registry. GKE on GDC orchestrates the container workload. We apply Kubernetes network policies to isolate the SLM namespace, ensuring that the model can only be queried by authorized internal Java backend services. By enforcing kernel isolation at the container level and physical isolation at the hardware level, we construct a true defense-in-depth architecture. The Gemma model processes highly classified proprietary data while satisfying the strictest data sovereignty requirements. The transition from cloud-first to sovereign-edge AI marks a critical maturation in software engineering. Trustworthy intelligence demands predictable execution boundaries. By merging the zero-trust foundations of Docker Hardened Images with the physical isolation of Google Distributed Cloud, engineering teams can deliver state-of-the-art reasoning capabilities directly to the tactical edge without compromising a single byte of sensitive data.

By Anuj Ashok Potdar

Event Sourcing in Distributed Systems: Architecture, Consistency, and Fault-Tolerant Projections

If you've worked with a relational database for any length of time, you know the drill: state lives in a row, and every UPDATE overwrites whatever was there before. It's simple, it's fast to reason about, and it throws away history by design. Update a bank balance from $1,000 to $900, and the fact that a withdrawal happened and why is gone unless some separate audit log happened to catch it. Event sourcing flips that model on its head. Instead of storing current state, you store the sequence of events that produced it, and you derive state on demand by replaying the log. History isn't a side effect you bolt on afterward; it is the write path. This isn't a niche idea anymore. Netflix runs event-sourced pipelines that process over a billion viewer events a day for personalization and health monitoring. Uber pushes millions of events per minute ride requests, location updates, pricing signals through Kafka-backed streams with exactly-once semantics for fraud detection. LinkedIn built Kafka in the first place to serve as an append-only log for activity data. Slack pipes billions of daily messages through event-driven pipelines that carry full metadata so services don't have to make synchronous calls to each other. What's harder to find is a clear-eyed look at what actually breaks in production and how teams fix it. That's what this article is about: the data model, the CQRS split, the consistency trade-offs, and the part that tends to get glossed over: how to keep read models correct when the same event might get delivered twice, three services need to stay in sync, and any one of them might be down when the message arrives. Events as First-Class State In a CRUD system, a withdrawal looks like this: SQL -- Traditional CRUD (destructive) UPDATE accounts SET balance = 900 WHERE id = 123; In an event-sourced system, nothing gets overwritten. You append a new fact: SQL -- Event sourcing (additive) INSERT INTO events (stream_id, event_type, payload, version) VALUES ('account-123', 'MoneyWithdrawn', '{"amount": 100}', 42); Current state is reconstructed by folding over every event in a stream, in version order. It's a pure function — same events in, same state out, and it maps directly onto a left-fold/reduce if you've ever written one: Python def reconstruct_balance(account_id): events = db.query( 'SELECT * FROM events WHERE stream_id = %s ORDER BY version', [f'account-{account_id}'] ) state = {'balance': 0} for event in events: if event.type == 'AccountOpened': state['balance'] = 0 elif event.type == 'MoneyDeposited': state['balance'] += event.payload['amount'] elif event.type == 'MoneyWithdrawn': state['balance'] -= event.payload['amount'] return state['balance'] The Event Store Schema The event store itself is a boring relational table with one architectural rule bolted on: nothing is ever updated or deleted. SQL CREATE TABLE events ( id BIGSERIAL PRIMARY KEY, stream_id UUID NOT NULL, event_type VARCHAR NOT NULL, payload JSONB NOT NULL, version INT NOT NULL, occurred_at TIMESTAMPTZ NOT NULL DEFAULT now(), UNIQUE (stream_id, version) ); That UNIQUE (stream_id, version) constraint is doing more work than it looks like. It's your optimistic concurrency control: if two processes both try to append version 5 to the same stream, the second one gets a unique-constraint violation instead of silently clobbering the first. It's compare-and-swap at the stream level, for free. And to be clear — this table can live in Postgres, MySQL, Oracle, whatever you're already running. Event sourcing is a write strategy, not a database product. Don't let anyone tell you that you need a specialized event store to do this. Snapshots Replaying thousands of events every time you want to read an entity's state gets expensive fast. The standard fix is snapshotting: periodically materialize current state and store it next to the log. On read, load the latest snapshot and replay only what's happened since. SQL SELECT * FROM snapshots WHERE stream_id = 'account-123' ORDER BY version DESC LIMIT 1; -- Then replay only events after snapshot.version SELECT * FROM events WHERE stream_id = 'account-123' AND version > :snapshot_version ORDER BY version; The CQRS Boundary In practice, event sourcing almost always shows up paired with CQRS (Command Query Responsibility Segregation) — splitting the write path (commands that produce events) from the read path (queries against a denormalized read model). This isn't just architectural tidiness. The event log is optimized for append throughput and sequential replay; it's a bad fit for ad-hoc queries, joins, sorting, or full-text search. ComponentRoleTypical technologyCommand handlerValidates commands, appends eventsApplication service layerEvent storeAppend-only source of truthPostgreSQL, EventStoreDB, KafkaProjectionReads events, builds the read modelBackground worker, Kafka consumerRead modelDenormalized query targetPostgreSQL, Redis, ElasticsearchQuery APIServes reads to clientsREST / GraphQL Consistency Models and the Lag Problem Projection Lag Is Real, Design For It Because the projection runs asynchronously, there's a window between an event landing in the store and the read model reflecting it. A user withdraws money and immediately checks their balance and might see the stale number. This is the central consistency problem in any CQRS/event-sourcing setup, and it needs an explicit answer, not a hope that it won't matter. In healthy systems on Kafka or a comparable bus, projection lag typically sits around 50–300ms. Under load spikes, partitions, or consumer restarts, it can stretch to seconds or minutes. Your design needs to be correct under the bad case, not just the median. Pick Your Battles: Selective Consistency The real design question isn't "How do I eliminate eventual consistency?" It's "Where do I actually need strong consistency, and where can I let the lag ride?" Those two buckets look pretty different: Data typeConsistency requiredWhyAccount balance post-withdrawalStrongFinancial correctnessPayment confirmationStrongRegulatory, user expectationRecommendation feedEventualA few seconds of lag is invisibleNotification badge countEventualApproximate is fineProduct inventory on a listing pageEventualMinor lag is tolerableSearch index for a new entityEventualUsers already expect crawl delay Read-Your-Own-Writes For the strong-consistency bucket, the most common mechanism is read-your-own-writes (RYOW). The command API hands back the version number it just appended; the client sends that version back on its next read; and the query service refuses to answer until the read model has caught up. Markdown # 1. Command returns the appended event version POST /accounts/123/withdraw {amount: 100} → HTTP 200 {status: 'ok', event_version: 42} # 2. Client sends version as constraint on next read GET /accounts/123/balance Headers: X-Min-Version: 42 # 3. Query service checks projection position SELECT last_processed_version FROM projection_checkpoints WHERE name = 'balance_projection'; -- If < 42, wait up to 500ms then retry or return 202 Accepted When You Just Bypass the Read Model For a small set of high-stakes flows such as payment confirmation screens, dispute resolution, regulatory audits, plenty of systems skip the read model entirely and rebuild state straight from the event log. It's always consistent, at the cost of latency. Snapshots (see above) keep that latency bounded even for entities with long histories. Three Ways to Keep Projections in Sync The projection needs to process each event exactly once and update the read model atomically with its own checkpoint. In practice, that comes down to three strategies, each suited to a different infrastructure setup. 1. Same-Database Transaction The simplest option: put the read model and the checkpoint table in the same relational database, and wrap the update and the checkpoint advance in one transaction. SQL BEGIN; UPDATE account_balances SET balance = 900 WHERE account_id = 123; UPDATE projection_checkpoints SET last_event_id = 42 WHERE name = 'balance_projection'; COMMIT; Either both writes land or neither does, which kills the two classic failure modes: processing an event without advancing the checkpoint (you'll reprocess it on restart) and advancing the checkpoint without writing the data (permanent, silent data loss). Great when your whole read model fits in one database. 2. Kafka + Idempotent Consumers Once Kafka is your event bus, consumer offsets live inside Kafka and can't be committed in the same transaction as your read-model write. Kafka only promises at-least-once delivery, so on a consumer restart, some events may show up twice. The fix is to make the consumer idempotent, in which case it's safe to run twice on the same event. SQL INSERT INTO account_balances (account_id, balance, last_event_id) VALUES (%s, %s, %s) ON CONFLICT (account_id) DO UPDATE SET balance = EXCLUDED.balance, last_event_id = EXCLUDED.last_event_id WHERE account_balances.last_event_id < EXCLUDED.last_event_id; That WHERE clause is the whole trick. If event version 42 gets replayed after it's already been applied, 42 < 42 is false and the update is a no-op. ScenarioDB state beforeIncoming eventConditionResultNormal processinglast_event_id = 41v = 4241 < 42 → TRUEUpdate appliedCrash replay (duplicate)last_event_id = 42v = 4242 < 42 → FALSESkipped safelyOut-of-order deliverylast_event_id = 42v = 4142 < 41 → FALSEStale event ignored The INSERT branch only ever fires once, when the entity's stream first shows up. Every event after that takes the ON CONFLICT path, and the version guard is what makes it idempotent. 3. The Transactional Outbox Pattern Things get harder when a projection needs to fan out to several different stores at once, say a relational read model, an Elasticsearch index, a Redis cache, and a webhook receiver. No single database transaction can span all of those. This is where the transactional outbox pattern earns its keep. The projection writes its read-model update and an outbox row in one local transaction. A separate relay process picks up pending outbox rows and delivers them to each downstream target, retrying until each one confirms. Because the outbox write is atomic with the read-model write, the relay always knows exactly what still needs to go out. SQL -- Atomic: both rows commit or neither does BEGIN; UPDATE account_balances SET balance = 900 WHERE account_id = 123; INSERT INTO outbox (id, payload, es_sent_at, redis_sent_at, webhook_sent_at) VALUES (gen_random_uuid(), '{"account_id": 123, "balance": 900, "event_version": 42}', NULL, NULL, NULL); COMMIT; Making the Outbox Relay Fault-Tolerant The relay delivers to each target independently and tracks confirmation per target. The scenario worth designing for: the relay writes successfully to target A, then crashes before marking the row delivered, so on restart, target A gets the same write again. Every target needs to handle that gracefully. Elasticsearch: Let the Document ID Do the Work Elasticsearch indexing is idempotent as long as you supply a stable document ID yourself. Index the same ID twice and you just overwrite the first doc with identical content, having no duplicates. Python # Non-idempotent: ES generates a random ID es.index(index='accounts', body={'balance': 900}) # → Two calls produce two documents # Idempotent: stable ID derived from the outbox row es.index( index='accounts', id=outbox_row.id, # same on every retry body={'balance': 900, 'event_version': 42} ) Redis: Absolute Values, Not Deltas SET in Redis is naturally idempotent — setting a key to the same value twice is the same as setting it once. The catch: your relay has to write absolute values, never deltas. INCRBY and LPUSH are not idempotent and have no business being in a relay path. Python # Non-idempotent: applying a delta twice corrupts state redis.incrby('account:123:balance', -100) # Idempotent: writing the derived absolute value redis.set('account:123:balance', 900) Webhooks: Idempotency Keys External webhook receivers are the trickiest case, because you don't control their implementation. The standard answer is the one Stripe documents publicly: an Idempotency-Key header. The relay sends the outbox row's UUID as that key on every attempt; the receiver keeps a table of keys it's already seen and just replays the original response for a repeat. Getting this right on the receiver side means handling three edge cases: Concurrent duplicate requests: Two requests with the same key arrive before either has been saved. Fix it with an atomic INSERT under a unique constraint; the second insert fails, and that thread waits on the first.Key reuse with a different body: Same key, different payload. Reject with a 422 after comparing a hash of the stored body to the incoming one.Key expiry: Keys can't live forever. Stripe uses a 24-hour TTL, which means your relay's retry window (and dead-lettering) has to finish well inside that boundary. The key record and the business-logic effect have to commit in the same transaction. If the business logic runs but the key never gets saved, a retry re-executes the logic and that's not a missed optimization, that's data corruption. Python def handle_request(request): key = request.headers['Idempotency-Key'] req_hash = sha256(request.body) with db.transaction(): row = db.execute(''' INSERT INTO idempotency_keys (key, request_hash, created_at, expires_at) VALUES (%s, %s, now(), now() + interval '24 hours') ON CONFLICT (key) DO UPDATE SET key = EXCLUDED.key RETURNING key, request_hash, response_status, response_body ''', [key, req_hash]).fetchone() if row.request_hash != req_hash: return 422, 'Key reused with different request body' if row.response_status is not None: return row.response_status, row.response_body result = run_business_logic(request) db.execute(''' UPDATE idempotency_keys SET response_status = %s, response_body = %s WHERE key = %s ''', [result.status, result.body, key]) return result.status, result.body Dead-Letter Escalation Transient failures — timeouts, a target being briefly down — get exponential backoff with per-target retry counters on the outbox row. Permanent failures — a schema mismatch, a decommissioned endpoint — shouldn't be allowed to block the relay forever. After a configurable number of attempts (10 is a common default), move the row to a dead-letter table for someone to look at. Who's Actually Running This in Production CompanyEvent volumePatternOutcomeNetflix>1B events/dayEvent sourcing + CQRS on KafkaPersonalization and monitoring for 260M+ subscribersUberMillions/minuteKafka + Flink, exactly-once semanticsFraud detection and dynamic pricing at global scaleLinkedInOrigin of Kafka (2010)Append-only activity stream logUnified data pipeline, replacing point-to-point integrationsSlackBillions of messages/dayKafka event propagation with full metadataDecoupled delivery and indexing under heavy loadStripeDocumented idempotency-key APITransactional idempotency at the webhook layerFormally specified 24-hour idempotency window with body-hash validation Takeaways A few things worth internalizing if you're building (or maintaining) one of these systems: Eventual consistency is a contract, not a bug. RYOW turns the consistency requirement into an explicit versioned header instead of an assumption. Idempotency keys turn "don't double-process this" into a first-class API contract. The outbox pattern turns delivery into a durable, queryable table instead of a fire-and-forget call. Once it's explicit, you can test it and monitor it — that's the whole point.Idempotency is the load-bearing primitive. It's standing in for distributed transactions, which are either unavailable, too slow, or too fragile for high-throughput event-driven systems. At-least-once delivery plus idempotent receivers gets you effectively-once behavior without any distributed coordination.The pattern underneath is always the same: a unique key derived from something stable (an event version, an outbox row's ID), stored atomically next to the effect it protects, checked before that effect is applied, and expired on a TTL that bounds how long you need to keep retrying.None of this locks you into specific technology. Event store in Postgres, EventStoreDB, or Kafka's compacted log — doesn't matter. Read model in MySQL, Redis, or Elasticsearch — doesn't matter. The outbox just needs to live in the same ACID store as the read model it's paired with. That's a low bar, and it's why teams can adopt this incrementally next to whatever they're already running. The hard part of event sourcing was never the event log where appending immutable rows is the easy 20%. The real engineering work is keeping derived state correct across multiple heterogeneous targets when any one of them can fail at any moment. Idempotency, applied consistently, is what makes that tractable without reaching for distributed transactions.

By Nitin Kishen Choweth Vijayakumar

Build a GitHub Contribution City With Java and Quarkus

GitHub's contribution calendar gives you an already iconic overview of your activity over the year. Each day has a square, and its color changes with the contribution level. And of course, those days and contributions can also be arranged into buildings, with a week becoming a small part of a city and busier weeks adding height to the skyline. I had some spare minutes and built Quarkus Contribution City around this idea, using Java and Quarkus to generate an SVG from GitHub's contribution data. The visual inspiration came from Pink Pixel's skyline, and you can see a version with my real data on my GitHub profile. A scheduled GitHub Action updates the image at night. For a Java developer, this is a small project you can follow from the API request through to the image. You can use the released action on your own profile, then change the drawing code locally once you have a feel for how the city is put together. For this article, I created a special DZone example that adds a small piece of character art above the buildings. Add the City to Your Profile Your GitHub profile README lives in a public repository with the same name as your account. For my account, that repository is myfear/myfear. Open your own profile repository, or create it with a README if you haven't used this GitHub feature before. Copy the project's complete publishing workflow into .github/workflows/contribution-city.yml on the default branch. This example checks out your profile repository and the action's v1 release line. It then generates the image and publishes it to an output branch. The workflow includes the repository write permission needed for that last step. The generation step uses the repository owner's username, so the same configuration can be copied between personal profile repositories. You can add the layout options to that step: - name: Generate contribution city id: city uses: ./.city-action with: github-token: ${{ secrets.CONTRIBUTION_TOKEN || secrets.GITHUB_TOKEN } username: ${{ github.repository_owner } weeks: '53' height: '14' theme: github-dark The local path refers to the action checkout made earlier in the example workflow. The built-in GITHUB_TOKEN is the starting point for reading your visible contribution data. An optional CONTRIBUTION_TOKEN secret can supply a personal token when additional access is needed. The calendar returned by GitHub depends on the token's permissions and your account's contribution visibility settings. After committing the workflow, open Actions in your profile repository, select publish-contribution-city, and choose Run workflow. A successful run creates contribution-city.svg on the output branch. Add this image reference to your profile README, replacing both occurrences of USERNAME: ![My GitHub contribution city](https://raw.githubusercontent.com/USERNAME/USERNAME/output/contribution-city.svg) The example also has a daily schedule for 03:17 UTC. Subsequent runs create a commit when the generated image changes. For this setup, the workflow and its run history can be found in your profile repository, while the Java implementation is maintained in the action's repository. Give Each Week a Building The city is arranged in time order, with the most recent week on the right. Every building is two character cells wide, so all weeks occupy the same horizontal space. Weekly contribution totals determine the height, and daily contribution levels provide the window colors. Month labels beneath the street help you find your place in the year. Height needs a little extra thought. A very busy week can have many times the contributions of a quiet one. With a linear scale, those quieter weeks would be compressed close to the street. The city uses a logarithmic scale, which gives them more room while preserving the order of weekly totals. For each week, the calculation can be expressed as: int height = weeklyTotal == 0 ? 1 : 1 + (int) Math.round( Math.log1p(weeklyTotal) / Math.log1p(maximumWeeklyTotal) * (maxHeight - 1)); With a maximum of 20 contributions and a height limit of 14, a week with three contributions gets seven window rows. The busiest week gets 14, and a week with zero contributions keeps one row. Quiet periods therefore retain their position in the calendar. An antenna marks the busiest displayed week; when totals are tied, the most recent week receives it. The two columns of windows are filled by cycling through the days of the week. This repeats the daily pattern in taller buildings, so window count is a visual texture. The weekly total controls height, while the window colors carry the daily contribution levels. Together, they give you a view of the pattern across your calendar. You can change weeks to 12 for a shorter view. The tallest building is then calculated from those recent weeks, which lets the selected period use the available height. Keep this in mind when comparing images: the same week can appear taller in a shorter view. The displayed contribution total still covers the full calendar returned by GitHub. Follow the Data Through Java The released project uses Quarkus with the Quarkus GitHub Action extension. That extension connects an @Action method to workflow inputs, outputs, and the job summary. Inside the method, the drawing process comes down to three steps: ContributionCalendar calendar = contributionClient.fetch( configuration.username(), inputs.getGitHubToken().orElseThrow(), context.getGithubGraphQLUrl()); Scene scene = cityBuilder.build(calendar, configuration.options()); String svg = svgRenderer.render(scene, configuration.options().theme()); The client reads the contribution calendar with one GraphQL request and converts the response into Java records for the calendar, weeks, and days. CityBuilder uses these records to place roofs, windows, and the street in a grid. SvgRenderer turns that grid into the final SVG. Changes to the skyline can therefore be made in plain Java classes, with the GitHub connection kept in the client. The SVG contains the shapes and text needed to display the city. Adjacent characters with the same color are grouped during rendering, and their horizontal positions are set explicitly to keep the columns aligned. Identical data and options produce identical file contents, which allows the publisher to compare images before creating a commit. The action sets up Java and JBang, builds the selected source revision, and runs the packaged application. Change the Layout Locally To try a change, install JDK 21 and JBang, with Git and Bash available. The repository includes a Maven wrapper. JBang is also used by the integration tests to run the packaged action through the same launcher used in GitHub Actions. git clone --branch v1.0.0 https://github.com/myfear/quarkus-contribution-city.git cd quarkus-contribution-city ./mvnw verify The tests use local test data and a local GraphQL server, so you can run them without GitHub credentials. They also generate SVG previews under target/previews. Open github-dark.svg, recent-12.svg, and empty.svg in a browser to compare a full calendar, a shorter period, and a calendar with zero contributions. Start with the height calculation in CityBuilder, then run the same command again and compare the previews. The saved data stays the same between runs, which makes the effect of your change easier to see. The project is available under the Apache License 2.0, and the v1.0.0 release gives you a fixed starting point. Give the City a DZone Landmark Version 1.0.0 exposes weeks, height, and the two built-in themes through workflow inputs. For a custom shape, you extend the Java code in your fork. The Scene record gives you a grid of cells to change before the SVG is rendered, and that is where we'll add the DZone lettering. Each cell has a character, a role such as roof or window, and the color information used by the renderer. A character mask is enough to describe a new shape: # marks a filled block, and spaces leave the background visible. By arranging those blocks, you can draw letters, an arch, or a stepped tower. For this example, the lettering is placed above the weekly buildings. It has its own fixed shape, while the buildings below continue to show the contribution calendar. The image uses a snapshot of my real account data, with 534 contributions returned by GitHub on October 5, 2026, when I wrote this article. A DZone-inspired palette and decorative lettering added to my contribution city. The weekly building heights and daily window patterns come from the same calendar used by the regular view. Add this class to the existing city package in your fork, at src/main/java/com/themainthread/contributioncity/city/DZoneLandmark.java: package com.themainthread.contributioncity.city; import java.util.ArrayList; import java.util.Collections; import java.util.List; import com.themainthread.contributioncity.render.Theme; public final class DZoneLandmark { public static final Theme THEME = new Theme("dzone-example", "#161b22", "#242c36", "#303947", "#f58a42", "#9ba9ba", "#edf2f7", List.of("#536070", "#86431f", "#b85b25", "#e57d36", "#ffb365")); private static final List<String> MASK = List.of( "#### ##### ### # # #####", "# # # # # ## # # ", "# # # # # ## # # ", "# # # # # # # # #### ", "# # # # # # ## # ", "# # # # # # ## # ", "#### ##### ### # # #####"); private DZoneLandmark() { } public static Scene addTo(Scene city) { int landmarkWidth = MASK.getFirst().length() * 2; int width = Math.max(city.width(), landmarkWidth); int left = (width - landmarkWidth) / 2; List<List<Cell>> rows = new ArrayList<>(); Cell block = new Cell('█', CellKind.ROOF, 0, -1); for (String line : MASK) { List<Cell> row = new ArrayList<>(Collections.nCopies(width, Cell.SKY)); for (int column = 0; column < line.length(); column++) { if (line.charAt(column) == '#') { row.set(left + column * 2, block); row.set(left + column * 2 + 1, block); } } rows.add(row); } rows.add(Collections.nCopies(width, Cell.SKY)); for (List<Cell> sourceRow : city.rows()) { List<Cell> row = new ArrayList<>(sourceRow); row.addAll(Collections.nCopies(width - city.width(), Cell.SKY)); rows.add(row); } return new Scene(city.username(), city.totalContributions(), rows); } } The theme defines the background, two building shades, roof, street, and label colors, followed by the five window colors. Those window colors follow GitHub's contribution levels from zero through four. Here, the roof color is also used for the lettering, so the decoration has a consistent orange color across all seven rows. Inside addTo, each marked position is expanded to two cells because the renderer's cells are taller than they are wide. The lettering is centered, a blank row is added beneath it, and the original city rows are copied below. Short calendars receive extra background space on the right to make room for the lettering. The dates, counts, and order of the weekly buildings are preserved. To use the decoration in your fork, import DZoneLandmark from the city package in ContributionCityAction. Replace the existing scene-building and rendering lines with: Scene scene = cityBuilder.build(calendar, configuration.options()); Scene decorated = DZoneLandmark.addTo(scene); String svg = svgRenderer.render(decorated, DZoneLandmark.THEME); This selects the example's palette in Java. The published @v1 action continues to accept github-dark and mono; your fork contains the additional drawing code. For a local preview, add the following after the existing Files.writeString call in PreviewTest.write, with the same import for DZoneLandmark: Scene decorated = DZoneLandmark.addTo(builder.build(calendar, options)); Files.writeString(directory.resolve(name.replace(".svg", "-dzone.svg")), renderer.render(decorated, DZoneLandmark.THEME)); Run ./mvnw verify again and open target/previews/github-dark-dzone.svg. The build also writes the shorter and empty versions with the -dzone.svg suffix, so you can check how the decoration fits several calendars. When you're ready to use your fork on GitHub, update the second checkout in the publishing workflow to your fork's repository and a commit containing your changes. The generation step then runs the customized action from that checkout. Try It With Your Own Contributions Try Quarkus Contribution City with your own GitHub account and add the generated SVG to your profile. Start with the released action, then fork the project and change the character mask to give your city a landmark of its own. You could draw a stepped tower from your hometown or add your team's initials above the street, using the DZone version as a starting point. Share your version in a GitHub issue, or open a pull request if you'd like to contribute a new shape.

By Markus Eisele

Coding

Functions of Coding

Frameworks

Java

JavaScript

Languages

Tools

DZone's Featured Coding Resources

The Latest Coding Topics