Languages

Embedded and IoT devices operate in environments where every kilobyte of memory and every millisecond of response time matters. Yet most of them still rely on C or C++, languages powerful enough for low-level control but notorious for memory-safety flaws that can crash systems or expose them to security vulnerabilities. The issue is so widespread that around 70% of Microsoft’s and Google’s reported vulnerabilities stem from memory-safety issues. For connected devices running critical operations, such instability isn’t an option. However, a structured C++ to Rust migration addresses the root cause by replacing unsafe memory handling with compile-time guarantees and predictable real-time performance. Embedded environments leave little room for trial and error; therefore, experience and a clear roadmap make a big difference. For this purpose, companies often work with teams skilled in Rust development services and familiar with embedded software development. With the right expertise, you can introduce new Rust components gradually and safely without interruptions to the existing IoT ecosystem. Now, let’s discover how to migrate from C/C++ to Rust, review the migration challenges and best practices to mitigate them, and analyze a success story of practical Rust migration for embedded systems. Why Consider Migrating From C/C++ to Rust? Modern embedded systems, especially IoT-based medical devices, put heavy pressure on safety, energy use, and long-term maintenance. Rust handles these demands better than traditional C or C++ thanks to its built-in memory safety, reliable concurrency model, and predictable real-time behavior. When organizations consider a C++ to Rust migration, these advantages quickly become deciding factors for modernizing connected devices. They also translate into more stable products and lower life cycle costs: Reduced security risks: Rust’s memory safety drastically lowers vulnerabilities, meaning fewer patches and less downtime for IoT systems.Lower long-term maintenance costs: Rust reduces bugs that emerge later in production, leading to fewer expensive fixes after deployment.Improved reliability: With Rust, IoT devices crash less often, which means more uptime and fewer disruptions for end-users.Better performance: Rust’s efficiency can lead to lower energy consumption, extending the life of battery-powered IoT devices and reducing energy costs.Faster time-to-market for updates: Rust’s safe concurrency allows quicker development of new features without compromising stability, which means you can roll out improvements to your IoT solutions faster.Future-proof technology: Rust’s growing popularity means a sustainable ecosystem, which ensures that your IoT product stays modern and well-supported for years to come.Increased development velocity: Rust reduces debugging time and makes the code easier to reason about, speeding up onboarding and increasing overall developer productivity. Three Common Myths About Rust Migration Many organizations hesitate to begin migrating embedded systems from С or C++ to Rust because they expect major risks or disruptions. In reality, most of these concerns stem from common misconceptions rather than technical limitations in embedded software development. Understanding what Rust migration actually involves helps teams make confident, well-informed decisions. Let’s review the three most common misconceptions. Myth 1: “We’ll Have to Rewrite Everything” You don’t need a full rewrite to begin a C++ to Rust migration. Many teams start small, moving one or two isolated modules that bring the most maintenance pain. Using the Foreign Function Interface (FFI), Rust can work alongside C or C++ within the same firmware. This feature makes it possible to modernize gradually while the rest of the system stays live. Myth 2: “Switching to Rust Will Interrupt Delivery” Migration can happen without stopping production. With a structured approach, Rust modules are added one by one while existing CI/CD pipelines continue to run. Each new piece is tested, deployed, and benchmarked before the next one is introduced. The development cycle continues as usual; only the language inside changes. Myth 3: “We’ll Have to Replace the Entire Engineering Team” Most C or C++ developers adapt to Rust faster than expected. The languages share similar low-level logic, which makes the learning curve manageable with proper mentoring and review. Once trained, the same team can handle both Rust and C++, maintaining delivery speed while building long-term expertise. Three Ways of C/C++ to Rust Migration There’s no one-size-fits-all approach to C++ to Rust migration. The right strategy depends on how your system works today, how critical uptime is, and what level of modernization you’re aiming for. When adopting Rust for IoT and embedded systems, companies typically choose one of three paths: gradual integration, full rewrite, or hybrid migration. 1. Gradual Integration This path is ideal for safety-critical or continuously running devices. Instead of rewriting everything, the developers introduce Rust gradually — one module at a time. Typical starting points include telemetry handlers, utility modules, or background services that can be tested in isolation. The existing C/C++ software keeps running while new Rust components are integrated and validated. Over time, as more modules are proven stable, Rust takes over a larger share of functionality. 2. Full Rewrite If you’re launching a new product or phasing out outdated firmware, a full rewrite may be the most efficient route. The development starts with Rust from day one. It covers firmware logic, communication layers, and testing infrastructure. This scenario removes legacy constraints entirely and lets your team build clean architectures without legacy dependencies, design for memory safety from the start, and use Rust’s concurrency model to its full advantage. A full rewrite requires a skilled team, proven toolchains, and time for onboarding, but it also gives you complete control over the system’s design. Yalantis supports such transformations through end-to-end Internet of Things solutions, helping businesses develop secure, scalable, and future-ready connected products. 3. Hybrid Approach Many embedded systems can’t be fully rewritten due to vendor SDKs, licensed components, or hardware dependencies. In those cases, Rust and C/C++ coexist. Engineers connect both through FFI, so each language can do what it does best. C++ handles hardware-level operations, while Rust manages business logic, safety-critical processes, or communication layers. This model works particularly well for medical devices, industrial equipment, and long-lifecycle IoT platforms, where reliability and compliance matter just as much as modernization speed. Challenges and Best Practices by Yalantis for C/C++ to Rust Migration Before starting a full-scale migration, it’s important to understand the technical and architectural risks involved. Each challenge that appears during the embedded C++ to Rust migration can be handled through structured engineering practices. Below is an overview of key challenges in migrating embedded firmware from C++ to Rust and the Rust migration best practices Yalantis uses to keep modernization predictable and low-risk. Challenge 1: Maintaining Compatibility With Existing C/C++ Code Legacy firmware often has C++ modules woven tightly together. You can’t rewrite everything at once without risking device instability or unexpected regressions. Challenge 2: Operating Under Hardware and Microcontroller Limitations Most embedded devices run on minimal RAM, limited storage, and low CPU frequencies. A rushed migration can overload the hardware or increase latency. Challenge 3: Ensuring Deterministic Behavior in Real-Time Systems IoT and embedded applications often have strict timing and reliability requirements. Even slight delays in execution can affect safety, user experience, or device coordination. Challenge 4: Integrating With External Libraries and APIs Embedded firmware relies on vendor drivers, communication stacks, and low-level APIs. Poor integration planning can cause instability or break device compatibility. Conclusion Modernizing embedded and IoT systems requires predictable performance, strong security guarantees, and code that stays maintainable as the product evolves. Traditional C/C++ firmware often struggles with these demands. Rust solves many of these challenges upfront with built-in memory safety, reliable concurrency, deterministic execution, and a modern development ecosystem that reduces defects and speeds up delivery. A well-planned C/C++ to Rust migration strengthens product stability, lowers lifecycle costs, and gives you a foundation that scales with new device architectures and connectivity needs.

Traditionally, developers use different query optimization techniques, such as complex conditional looping and procedure branching, and query hints like OPTION (RECOMPILE). The query hint option sometimes leads to high CPU utilization. SQL Server 2022 (Compatibility Level 160) introduced a new feature called Parameter Sensitive Plan (PSP) Optimization as part of the core capabilities of the Intelligent Query Processing (IQP) feature family. PSP optimization addresses parameter sniffing at the engine level by allowing a single parameterized query to generate and maintain multiple active execution plans in the plan cache simultaneously, tailored to different data volume boundaries. How It Works: Mechanics, Dispatchers, and Variants Instead of forcing a single execution plan into cache for all parameter invocations, the SQL Server 2022 Query Optimizer uses data distribution statistics to recognize parameter sensitivity automatically during initial compilation. How to Detect Parameter Sensitivity During compilation, the optimizer evaluates predicates referencing parameter variables. It checks the associated column statistics histogram. If the statistics indicate significant data skew across histogram steps, the query is flagged as PSP-eligible. When a query qualifies, the engine creates a Dispatcher Plan. The dispatcher plan contains the logic needed to evaluate incoming runtime parameter values against pre-calculated boundary ranges. It acts as an internal router rather than a direct execution plan. The optimizer divides the parameter data distribution spectrum into distinct boundary buckets (typically low, medium, and high cardinality ranges) based on the histogram: Boundary calculation: The engine calculates threshold values using histogram step densitiesVariant generation: As different parameter values are executed, the dispatcher maps the runtime value to its corresponding boundary range and compiles or reuses a specific Variant Plan associated with that range. When reviewing execution plans in SQL Server Management Studio (SSMS), PSP-optimized queries display distinct Query Plan XML attributes under the Dispatcher and Statement Plan nodes, referencing variant identifiers (e.g., PlanVariantID="1" vs. PlanVariantID="2") Demonstrating PSP Optimization in Action The following example uses customers from the WideWorldImporters database. In the sample data, CustomerID = 90 has 150 orders, while CustomerID = 1060 has only 4. To make the execution-plan difference easier to observe, we create a small demonstration table that preserves this skew at a larger scale. SQL USE WideWorldImporters; GO DROP TABLE IF EXISTS dbo.PSP_OrderDemo; GO CREATE TABLE dbo.PSP_OrderDemo ( OrderID BIGINT IDENTITY PRIMARY KEY, CustomerID INT NOT NULL, OrderDate DATE NOT NULL, OrderAmount MONEY NOT NULL, Padding CHAR(200) NOT NULL ); GO CREATE INDEX IX_PSP_OrderDemo_CustomerID ON dbo.PSP_OrderDemo(CustomerID); GO Populate one customer with approximately 500,000 rows and the rare customer with only a few rows: -- Common customer INSERT dbo.PSP_OrderDemo (CustomerID, OrderDate, OrderAmount, Padding) SELECT TOP (500000) 90, DATEADD(DAY, -(ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) % 365), GETDATE()), 100.00, REPLICATE('X',200) FROM sys.all_objects a CROSS JOIN sys.all_objects b; GO -- Rare customer INSERT dbo.PSP_OrderDemo (CustomerID, OrderDate, OrderAmount, Padding) SELECT TOP (4) 1060, GETDATE(), 100.00, REPLICATE('X',200) FROM sys.all_objects; GO UPDATE STATISTICS dbo.PSP_OrderDemo WITH FULLSCAN; GO Create a parameterized stored procedure: CREATE OR ALTER PROCEDURE dbo.GetOrdersByCustomer @CustomerID INT AS BEGIN SELECT OrderID, CustomerID, OrderDate, OrderAmount, Padding FROM dbo.PSP_OrderDemo WHERE CustomerID = @CustomerID; END; GO Enable Actual Execution Plan in SSMS with Ctrl+M, then first test with PSP disabled. Because the rare value is compiled first, SQL Server may cache a plan optimized for four rows and reuse it when the same procedure returns approximately 500,000 rows. This illustrates the traditional parameter-sniffing problem. SQL ALTER DATABASE WideWorldImporters SET COMPATIBILITY_LEVEL = 160; GO ALTER DATABASE SCOPED CONFIGURATION SET PARAMETER_SENSITIVE_PLAN_OPTIMIZATION = OFF; GO ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE; GO SET STATISTICS IO, TIME ON; EXEC dbo.GetOrdersByCustomer @CustomerID = 1060; -- Rare EXEC dbo.GetOrdersByCustomer @CustomerID = 90; -- Common GO Because the rare value is compiled first, SQL Server may cache a plan optimized for four rows and reuse it when the same procedure returns approximately 500,000 rows. This illustrates the traditional parameter-sniffing problem. Now enable PSP and repeat the workload: SQL ALTER DATABASE SCOPED CONFIGURATION SET PARAMETER_SENSITIVE_PLAN_OPTIMIZATION = ON; GO ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE; GO EXEC dbo.GetOrdersByCustomer @CustomerID = 1060; EXEC dbo.GetOrdersByCustomer @CustomerID = 90; GO In the PSP-enabled plans, opening the execution plan XML should also reveal QueryVariantID and PLAN PER VALUE, confirming that SQL Server is maintaining separate plans for different cardinality ranges. The Parameter Sniffing Decision Tree When diagnosing and resolving parameter sniffing issues in production, follow this step-by-step decision framework: Summary of Best Practices Avoid blanket global fixes: Resist the urge to disable parameter sniffing database-wide via ALTER DATABASE SCOPED CONFIGURATION SET PARAMETER_SNIFFING = OFF. Disabling it globally starves steady-state OLTP queries of optimal compilation behavior. Prioritize targeted query hints: Apply OPTION (RECOMPILE) for complex reporting queries run infrequently, and OPTION (OPTIMIZE FOR ...) for high-throughput transactional procedures with well-understood parameters.Establish baseline guardrails in Query Store: Leverage Query Store runtime statistics to track standard deviation spikes in avg_cpu_time and avg_logical_io_reads. Use plan forcing as an immediate temporary fix during active site incidents.Upgrade to compatibility level 160 where feasible: Capitalize on IQP features in SQL Server 2022 to handle non-uniform data distributions natively without littering codebases with query hints. Conclusion Parameter sniffing is not an engine defect. SQL Server’s default mechanism for maximizing plan reuse and minimizing compilation overhead. While skewed data distributions meet volatile execution requirements, captured plans can severely compromise application performance. To conclude, SQL Server 2022’s Parameter Sensitive Plan (PSP) optimization introduces welcome automation; enterprise database systems, at the same time, still require a deliberate, structured approach to query tuning.

Database optimizers don’t see the future. They make decisions based on statistics, estimates, and cost models — and most of the time, those decisions work remarkably well. But what if the optimizer says: “The Sequential Scan is cheaper.” … and reality says: “The Index Only Scan is almost twice as fast.” That is exactly what I wanted to investigate. For this experiment, I created a deliberately skewed PostgreSQL dataset containing 100 million rows, where almost every row has the same customer_type and only a single row represents a rare value. This creates the perfect environment to challenge the optimizer’s assumptions and compare estimated cost with actual execution time. The question is simple: Does PostgreSQL’s estimated cost really tell us which execution plan will be faster? The results were surprising. Not always. And the difference between what PostgreSQL estimated and what actually happened reveals an important lesson about reading and troubleshooting execution plans. 1. Creating the Test Table SQL postgres=# CREATE TABLE tbl_test ( id BIGINT, customer_type VARCHAR(20), city VARCHAR(30), status VARCHAR(20) ); CREATE TABLE The important column for this experiment is customer_type. I wanted to create an extremely skewed data distribution: 99,999,999 rows → REGULAR1 row → VIP The other columns were added to make the table more representative of a real-world table. 2. Generating 100 Million Rows I used PostgreSQL’s generate_series() to generate 100 million rows: SQL postgres=#INSERT INTO tbl_test (id, customer_type, city, status) SELECT gs, CASE WHEN gs = 100000000 THEN 'VIP' ELSE 'REGULAR' END, CASE (gs % 10) WHEN 0 THEN 'Babol' WHEN 1 THEN 'Amol' WHEN 2 THEN 'Behshahr' WHEN 3 THEN 'MarziKola' WHEN 4 THEN 'Bandpay' WHEN 5 THEN 'DerazKesh' WHEN 6 THEN 'Sari' WHEN 7 THEN 'Rasht' WHEN 8 THEN 'Zanjan' ELSE 'Tel Aviv' END, CASE WHEN gs % 100 < 95 THEN 'ACTIVE' WHEN gs % 100 < 99 THEN 'INACTIVE' ELSE 'SUSPENDED' END FROM generate_series(1, 100000000) AS gs; INSERT 0 100000000 This is useful because it allows us to compare two very different situations. 3. Creating the Index I created a B-tree index on customer_type: SQL postgres=# CREATE INDEX idx_customer_type ON tbl_test(customer_type); CREATE INDEX Then I checked the size of both objects: SQL postgres=# SELECT 'TABLE' AS object_type, 'tbl_test' AS object_name, pg_relation_size('tbl_test') / 1024 / 1024 AS size_mb UNION ALL SELECT 'INDEX', 'idx_customer_type', pg_relation_size('idx_customer_type') / 1024 / 1024; object_type | object_name | size_mb -------------+-------------------+--------- TABLE | tbl_test | 5918 INDEX | idx_customer_type | 660 (2 rows) The index is significantly smaller than the table. This will become important later. 4. Collecting PostgreSQL Statistics Before asking the optimizer to make decisions, I collected statistics: SQL postgres=# ANALYZE VERBOSE tbl_test; INFO: analyzing "public.tbl_test" INFO: "tbl_test": scanned 30000 of 757576 pages, containing 3959968 live rows and 0 dead rows; 30000 rows in sample, 99999224 estimated total rows INFO: finished analyzing table "postgres.public.tbl_test" avg read rate: 96.315 MB/s, avg write rate: 0.010 MB/s buffer usage: 874 hits, 29255 reads, 3 dirtied WAL usage: 10 records, 3 full page images, 10583 bytes, 0 buffers full system usage: CPU: user: 0.59 s, system: 0.07 s, elapsed: 2.37 s ANALYZE 5. Querying the Common Value Now let’s ask PostgreSQL to count the REGULAR customers: SQL postgres=# explain SELECT COUNT(*) FROM tbl_test WHERE customer_type = 'REGULAR'; QUERY PLAN ------------------------------------------------------------------------------------------------ Finalize Aggregate (cost=1383572.41..1383572.42 rows=1 width=8) -> Gather (cost=1383572.20..1383572.41 rows=2 width=8) Workers Planned: 2 -> Partial Aggregate (cost=1382572.20..1382572.21 rows=1 width=8) -> Parallel Seq Scan on tbl_test (cost=0.00..1278406.17 rows=41666413 width=0) Filter: ((customer_type)::text = 'REGULAR'::text) (6 rows) PostgreSQL chooses a sequential scan instead of the index because it estimates that reading most of the table directly will be cheaper than using the index. Now, let’s execute the query and see how long PostgreSQL actually takes: SQL postgres=# SELECT COUNT(*) FROM tbl_test WHERE customer_type = 'REGULAR'; count ---------- 99999999 (1 row) Time: 15362.798 ms (00:15.363) Execution time: 15.36 seconds to count nearly 100 million rows. 6. Forcing PostgreSQL to Use the Index So far, PostgreSQL had a clear preference: sequential scan. But I wanted to see what would happen if I took that option away. For investigation purposes, I disabled sequential scans: SQL postgres=# SET enable_seqscan = off; SET Then I asked PostgreSQL to show me the plan again: SQL postgres=# explain SELECT COUNT(*) FROM tbl_test WHERE customer_type = 'REGULAR'; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------- Finalize Aggregate (cost=1610210.39..1610210.40 rows=1 width=8) -> Gather (cost=1610210.17..1610210.38 rows=2 width=8) Workers Planned: 2 -> Partial Aggregate (cost=1609210.17..1609210.18 rows=1 width=8) -> Parallel Index Only Scan using idx_customer_type on tbl_test (cost=0.57..1505044.14 rows=41666413 width=0) Index Cond: (customer_type = 'REGULAR'::text) (6 rows) PostgreSQL switched to a Parallel Index Only Scan, and now we see something very interesting. The estimated cost increased: SQL Sequential Scan ≈ 1,383,572 Index Only Scan ≈ 1,610,210 In other words, PostgreSQL’s own cost model still considers the Index Only Scan to be the more expensive option. But I wasn’t interested in what PostgreSQL thought would happen. I wanted to know what would actually happen. So I executed the query: SQL --Forcing PostgreSQL to Use the Index postgres=# SET enable_seqscan = off; SET postgres=# SELECT COUNT(*) FROM tbl_test WHERE customer_type = 'REGULAR'; count ---------- 99999999 (1 row) Time: 7883.189 ms (00:07.883) Execution time: ~7.88 seconds And this is where the experiment gets interesting. PostgreSQL’s cost model says: Index Only Scan → More expensive But the real execution says: Index Only Scan → 7.88 seconds This is the exact gap I wanted to investigate: Estimated cost and actual execution time are not the same thing. Plain Text Metric Sequential Scan Index Only Scan Estimated Cost 1.38M 1.61M Execution Time 15 sec 7.88 sec Actual Result Slower 1.9× faster 7. What Happens With the Rare Value? Now let’s ask for the opposite value: customer_type = ‘VIP’. SQL postgres=# explain SELECT COUNT(*) FROM tbl_test WHERE customer_type = 'VIP'; QUERY PLAN --------------------------------------------------------------------------------------------- Aggregate (cost=4.59..4.60 rows=1 width=8) -> Index Only Scan using idx_customer_type on tbl_test (cost=0.57..4.58 rows=1 width=0) Index Cond: (customer_type = 'VIP'::text) (3 rows) This is exactly what we would expect. There is only one VIP row in 100 million rows. Using the index allows PostgreSQL to locate that row without scanning the entire table. So we have two extreme cases: Plain Text REGULAR → 99,999,999 rows → Sequential Scan VIP → 1 row → Index Only Scan This demonstrates the importance of selectivity. 8. Removing the Aggregate To make the comparison more realistic, I removed the COUNT(*) aggregate and selected the indexed column directly. First, with sequential scans enabled: SQL postgres=# SET enable_seqscan = on; SET postgres=# EXPLAIN (ANALYZE, BUFFERS)SELECT customer_type FROM tbl_test WHERE customer_type = 'REGULAR'; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------ Seq Scan on tbl_test (cost=0.00..2007576.40 rows=100000032 width=8) (actual time=2.577..22438.130 rows=99999999.00 loops=1) Filter: ((customer_type)::text = 'REGULAR'::text) Rows Removed by Filter: 1 Buffers: shared hit=1192 read=756384 Planning Time: 0.169 ms Execution Time: 26854.922 ms (6 rows) PostgreSQL chose a sequential scan. Execution time: 26.9 seconds. Now I forced PostgreSQL to use the index: SQL postgres=# SET enable_seqscan = off; SET EXPLAIN (ANALYZE, BUFFERS)SELECT customer_type FROM tbl_test WHERE customer_type = 'REGULAR'; SET Time: 0.302 ms QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------ Index Only Scan using idx_customer_type on tbl_test (cost=0.57..2088385.13 rows=100000032 width=8) (actual time=0.163..9471.775 rows=99999999.00 loops=1) Index Cond: (customer_type = 'REGULAR'::text) Heap Fetches: 0 Index Searches: 1 Buffers: shared read=84203 Planning Time: 0.170 ms Execution Time: 13451.658 ms (7 rows) Execution time: 13.45 seconds. Plain Text Sequential Scan Index Only Scan Execution Time 26.9 sec 13.45 sec Buffers Read 756,384 84,203 The Index Only Scan was roughly 2× faster, despite PostgreSQL estimating its cost as higher. This is where the difference between planner cost and real-world execution becomes impossible to ignore. Oracle Database: A Different Optimizer Decision I repeated a similar experiment in Oracle using the same logical data distribution. SQL SQL> CREATE TABLE tbl_test ( 2 id NUMBER(19), 3 customer_type VARCHAR2(20), 4 city VARCHAR2(30), 5 status VARCHAR2(20) 6 ); Table created BEGIN FOR i IN 0..99 LOOP INSERT /*+ APPEND */ INTO tbl_test (id, customer_type, city, status) SELECT i * 1000000 + LEVEL, CASE WHEN i = 99 AND LEVEL = 1000000 THEN 'VIP' ELSE 'REGULAR' END, CASE MOD(i * 1000000 + LEVEL, 10) WHEN 0 THEN 'Babol' WHEN 1 THEN 'Amol' WHEN 2 THEN 'Behshahr' WHEN 3 THEN 'MarziKola' WHEN 4 THEN 'Bandpay' WHEN 5 THEN 'DerazKesh' WHEN 6 THEN 'Sari' WHEN 7 THEN 'Rasht' WHEN 8 THEN 'Zanjan' ELSE 'Tel Aviv' END, CASE WHEN MOD(i * 1000000 + LEVEL, 100) < 95 THEN 'ACTIVE' WHEN MOD(i * 1000000 + LEVEL, 100) < 99 THEN 'INACTIVE' ELSE 'SUSPENDED' END FROM dual CONNECT BY LEVEL <= 1000000; COMMIT; END LOOP; END; / SQL> CREATE INDEX idx_customer_type ON tbl_test(customer_type); Index created The Oracle object sizes were: SQL SQL> select segment_name,segment_type ,bytes/1024/1024 from dba_segments where owner='VAHID'; SEGMENT_NAME SEGMENT_TYPE BYTES/1024/1024 -------------------- ------------------ --------------- TBL_TEST TABLE 3744 IDX_CUSTOMER_TYPE INDEX 2112 Now let’s run the same test in Oracle: SQL set autotrace traceonly explain SQL> SELECT COUNT(*) FROM vahid.tbl_test WHERE customer_type = 'REGULAR'; Execution Plan ---------------------------------------------------------- Plan hash value: 1445291348 ------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 12 | 72420 (1)| 00:00:03 | | 1 | SORT AGGREGATE | | 1 | 12 | | | |* 2 | INDEX FAST FULL SCAN| IDX_CUSTOMER_TYPE | 107M| 1225M| 72420 (1)| 00:00:03 | ------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 2 - filter("CUSTOMER_TYPE"='REGULAR') Note ----- - dynamic statistics used: dynamic sampling (level=2) Oracle chooses an INDEX FAST FULL SCAN! Despite almost every row matching REGULAR, Oracle scans the index because the query only needs COUNT(*). In the actual test, this plan completed in approximately 10 seconds. SQL SQL> SELECT COUNT(*) FROM vahid.tbl_test WHERE customer_type = 'REGULAR'; COUNT(*) ---------- 99999999 Elapsed: 00:00:10.11

Originally, SQL was SEQUEL — Structured English Query Language. Couchbase supports NL2SQL natively, but the WHERE clause only supports language-defined filters: range, semantic search, and text search. Exploiting Typesafe.ai's Jev models, we now show how you can easily add plain old English filters into the SQL++ WHERE clause. This is a step towards the original vision of making data processing natural and human. DuckDB showed Jev() functions, and Databricks added ai_decide() functions. For Couchbase, this article will show that two function families, jev() and cb_decide(), can be easily added to Couchbase and let a Couchbase query judge documents and return typed answers with the Couchbase Server! This method shows you how to use any model, not just Jev, in Couchbase SQL++. This article explains the sample UDFs and SQL++ published in the cb-jev-functions. In fact, Couchbase users have been doing it since 2018 to interact with external systems. Ask a database for the products under $500, and the answer arrives before you finish the sentence. Ask it for the ones that would make a good gift for a frequent flyer, and you have a project. Someone labels a few thousand examples. Someone trains a classifier, builds an embedding pipeline, or writes a prompt and a parser for the output. Weeks later, it works until the catalog changes. Nothing about the question is hard. A person could answer it by reading each description. The indexes support range predicates, text, or semantic searches. Here is the same question as a Couchbase query with the jev() function: SQL SELECT name, price FROM products WHERE price < 500 AND jev(products, 'a good gift for a frequent flyer'); Results: [ { "name": "Noise-Cancelling Headphones", "price": 299 }, { "name": "Leather Weekender Bag", "price": 450 } ] On the sample catalog in the project described below, you should expect the weekender bag, the noise-canceling headphones and the USB-C cable, and not the cookware set. Two details matter more than the result. The condition is a sentence, not a column or an index. And the function returns a boolean, not a paragraph. That second detail is the whole story. A model that talks cannot sit in a WHERE clause. A model that returns types can. This article explains how that works, how to run it in Couchbase today with two function families, jev() and cb_decide(), and what architects should ask before this gets near production. A Model That Answers in Types On September 15, 2026, TypeSafe AI released Jev, which it calls a System One model. You send it a state (text or JSON) and a set of typed questions. It returns typed answers with calibrated probabilities and never writes a sentence. Each question type lines up with something a database already has: Question typeAnswerSQL analoguenoulProbability that a statement is true, 0 to 1A BOOLEAN predicate in WHEREchoiceOne option from a list you define (up to 255)An enum you can GROUP BYscoreA position on an ordered rubricAn ordinal you can ORDER BY Noul is TypeSafe's name for the yes/no type. Its answer is a number such as 0.93, the probability that the statement is true. It is not a degree. A 0.9 for "a good gift for a frequent flyer" means the model is 90% sure, not that the product is 90% giftable. For degrees, ask a score question. All questions in one request run on the qualified documents in the post-scan filter. In the query above, the functions jev() run on all the documents that qualify for the (price < 500) predicate. TypeSafe quotes 70 to 500 ms end-to-end and $0.042 per million input tokens, with output tokens free. Treat those as vendor figures: the headline 190x speedup claims come from TypeSafe's own evaluations. I've previously written on calling a chat model from Couchbase SQL++. What you got back was prose, which a query cannot sort, threshold, or join on. A number can be sorted, thresholded, and audited, and nothing needs parsing. That is the change. Prior to that, I've also written about Couchbase's ability to call an external REST endpoint. We now exploit the same mechanism. Databricks, DuckDB With Jev. Databricks released ai_decide in Beta on September 30. It takes text and one or more questions and returns a probability, a choice from named criteria, or a score. You call it from SQL for batch work or from a REST API for real-time use, and its launch examples are model routing, tagging reviews, and evaluating agents. Databricks has said the motivation was privacy: many enterprise customers could not send data to Jev, so ai_decide runs on governed data inside the platform. Keep that in mind. It is the first question an architect should ask. The community moved too. pg-jev, a Postgres extension, introduced a jev() function that turns a plain-English condition into a WHERE clause, and several DuckDB ports kept its names: jev, jev_prob, jev_choice, and jev_score. The DuckDB team's write-up covers them. Notice that the two camps chose different shapes. The pg-jev and DuckDB family asks one question per call and returns one value: a predicate, a probability, a label, or a score. ai_decide asks several questions in one call and returns a structured result. Both fit SQL. Couchbase can offer both, because the plumbing underneath is general. Because the infrastructure is open, you can do more with it. Right now! The Plumbing Couchbase Already Had CURL() CURL() has been part of SQL++ since Couchbase 5.0. It calls an HTTP or HTTPS endpoint that returns JSON and gives you the response as data, usable in projections, WHERE clauses, or the FROM clause. In 2018, I used it to run Google's Vision, Translation, and Natural Language APIs against the travel-sample hotel reviews (Using Google Artificial Intelligence Services in Couchbase N1QL). The CURL() documentation sets the rules that matter for model calls: The endpoint must return JSON, the method must be GET or POST, and redirects are not allowed. Responses are capped by the result-cap setting, 20 MB by default.A Full Administrator must put a URL on the CURL() access list before anyone can call it, and matching is by prefix. Other users also need the QUERY_EXTERNAL_ACCESS role. A JavaScript Function Around It JavaScript user-defined functions arrived in Couchbase 7.0. You upload JavaScript to a library through the Query Functions REST API, expose a function from it with CREATE FUNCTION, and call it like a built-in. Inside the function, N1QL() runs a SQL++ statement, including SELECT CURL(...). In 2023, I used exactly this pattern to wrap OpenAI as a function called chatGPT3(). One practical rule: Couchbase JavaScript UDFs cannot rely on global variables, so each function declares its own constants. Here is the core of cb_decide(). It is short because the work is in the question object you pass: JavaScript function cb_decide(state, questions) { var KEY = "your-typesafe-key"; var URL = "https://api.typesafe.ai/v1/systemone"; var opts = { "request": "POST", "header": ["Authorization: Bearer " + KEY, "Content-Type: application/json"], "data": JSON.stringify({ model: "jev-latest", state: state, questions: questions }) }; var q = N1QL("SELECT CURL($1, $2) AS r", [URL, opts]); var resp = null; for (const row of q) { resp = row.r; } q.close(); if (!resp || !resp.answers) { throw new Error("cb_decide: " + JSON.stringify(resp)); } return resp.answers; } Three steps install it. The project's install.sh does the first two and injects your key at install time, so it never lands in source control: Shell # 1. Upload the library to the Query service curl -u Administrator:password -X POST \ http://localhost:8093/evaluator/v1/libraries/decide_lib \ --data-binary @decide_lib.js # 2. Allow the endpoint for CURL() curl -u Administrator:password -X POST \ http://localhost:8091/settings/querySettings/curlWhitelist \ -d '{"all_access": false, "allowed_urls": ["https://api.typesafe.ai"]}' -- 3. Expose the function to SQL++ CREATE OR REPLACE FUNCTION cb_decide(state, questions) LANGUAGE JAVASCRIPT AS "cb_decide" AT "decide_lib"; The access list can also be set in the Query Workbench under Settings. The REST endpoint is named curlWhitelist in the documentation. Two Function Families, One Engine FamilyFunctionsShapeModeled onjevjev, jev_prob, jev_choice, jev_scoreOne question, one value, readable inside WHERE, ORDER BY, and GROUP BYpg-jev and the DuckDB extensionscb_decidecb_decide, cb_decide_prob, cb_decide_choice, cb_decide_scoreMany questions in one request, returned as an object you can storeDatabricks ai_decide The signature of cb_decide() is my own design. It follows the idea of ai_decide, not its exact syntax, and the project is not affiliated with Databricks, TypeSafe AI, or DuckDB. jev(): one question, one value Each function in this family bakes one question type into the wrapper, so a query reads like plain SQL. The sample collections are products and person. SQL -- Filter: a boolean predicate SELECT name, price FROM products WHERE jev(products, 'a good gift for a frequent flyer'); [ { "name": "Leather Weekender Bag", "price": 450 }, { "name": "Cashmere Overcoat", "price": 1800 }, { "name": "Noise-Cancelling Headphones", "price": 299 } ] -- Rank: a probability you can sort on SELECT p.name, jev_prob(p, 'a good gift for a frequent flyer') AS prob FROM products AS p ORDER BY prob DESC; [ { "name": "Noise-Cancelling Headphones", "prob": 0.82 }, { "name": "Leather Weekender Bag", "prob": 0.75 }, { "name": "Cashmere Overcoat", "prob": 0.54 }, { "name": "Swiss Automatic Watch", "prob": 0.48 }, ... Results trucated for brevity here. -- Classify, then group (the subquery computes each label once) SELECT c.category, COUNT(*) AS n FROM ( SELECT jev_choice(p, 'which category fits best?', ['travel gear', 'audio', 'apparel', 'kitchen']) AS category FROM products AS p ) AS c GROUP BY c.category; [ { "category": "apparel", "n": 3 }, { "category": "audio", "n": 1 }, { "category": "kitchen", "n": 2 }, { "category": "travel gear", "n": 2 } ] -- Score: a number on an ordered rubric SELECT p.name, jev_score(p, 'how luxurious is this product?', ['budget', 'mid-range', 'premium', 'luxury']) AS luxury FROM products AS p ORDER BY luxury DESC; [ { "name": "Cashmere Overcoat", "luxury": 2.91 }, { "name": "Swiss Automatic Watch", "luxury": 2.72 }, { "name": "Leather Weekender Bag", "luxury": 2.02 }, { "name": "Noise-Cancelling Headphones", "luxury": 1.77 }, ...Results trucated for brevity The score comes back as a number and can be fractional, so check one real response to confirm the scale before you rely on specific values. cb_decide(): Many Questions, One Trip Sometimes you want several judgments about the same document. Asking them one at a time means one round trip each. cb_decide() sends them together and returns an object with one answer per question name: SQL SELECT s.name, s.d.gift.noul AS gift, s.d.category.choice AS category, s.d.luxury.score AS luxury FROM ( SELECT p.name, cb_decide(p, { "gift": {"type": "noul", "instructions": "Would this make a good gift for a frequent flyer?"}, "category": {"type": "choice", "instructions": "Which category fits best?", "criteria": {"travel gear": "Bags and trip accessories", "audio": "Headphones and speakers", "apparel": "Clothing and accessories", "kitchen": "Cookware and kitchen items"}, "luxury": {"type": "score", "instructions": "How luxurious is this product?", "criteria": ["budget", "mid-range", "premium", "luxury"]} }) AS d FROM products AS p ) AS s ORDER BY gift DESC; [ { "category": "audio", "gift": 0.88, "luxury": 1.75, "name": "Noise-Cancelling Headphones" }, { "category": "travel gear", "gift": 0.79, "luxury": 2.03, "name": "Leather Weekender Bag" }, { "category": "apparel", "gift": 0.63, "luxury": 2.9, "name": "Cashmere Overcoat" }, { "category": "apparel", "gift": 0.62, "luxury": 2.76, "name": "Swiss Automatic Watch" }, { "category": "travel gear", "gift": 0.52, "luxury": 0.01, "name": "USB-C Cable" }, { "name": "Everyday Cotton T-Shirt", "gift": 0.36, "category": "apparel", "luxury": 0.02 }, { "category": "kitchen", "gift": 0.31, "luxury": 0, "name": "Plastic Water Bottle" }, { "category": "kitchen", "gift": 0.17, "luxury": 0.92, "name": "Stainless Steel Cookware Set" } ] A question object has a type, instructions, and, for choice and score, criteria. For a choice, criteria is an object of label to description. For a score, it is an ordered array of 2 to 10 levels. This is also where the Databricks examples translate directly. Model routing, for instance, becomes one call that judges each prompt twice, then a CASE expression picks the tier: SQL SELECT s.prompt_id, s.d.reasoning.noul AS needs_reasoning, s.d.difficulty.score AS difficulty, CASE WHEN s.d.reasoning.noul > 0.5 THEN "large-model" ELSE "small-model" END AS route FROM ( SELECT p.prompt_id, cb_decide(p.prompt, { "reasoning": {"type": "noul", "instructions": "Does answering this require multi-step reasoning?"}, "difficulty": {"type": "score", "instructions": "How difficult is this request?", "criteria": ["trivial", "easy", "moderate", "hard"]} }) AS d FROM prompts AS p ) AS s; The project's samples also include an agent-evaluation query that asks whether each answer follows a policy and how completely it answers the question. The strongest use of cb_decide() is to decide once and keep the result. Write the whole decision object back to the document, then index the part you filter on: SQL UPDATE products AS p SET p.decision = cb_decide(p, { "gift": {"type": "noul", "instructions": "Would this make a good gift for a frequent flyer?"} }) WHERE p.decision IS MISSING LIMIT 50; CREATE INDEX idx_gift ON products(decision.gift.noul); SELECT name FROM products WHERE decision.gift.noul > 0.8; You pay for each document once. Later queries filter on an ordinary field that an index can serve. Run the UPDATE on a small batch first, and rerun it to continue, since the WHERE clause skips documents that already have a decision. Which One Should I Use? You needUseA condition inside WHEREjev(doc, 'condition')A number to sort or thresholdjev_probOne label to group onjev_choice, or cb_decide_choiceAn ordered ratingjev_score, or cb_decide_scoreSeveral judgments about one document in a single requestcb_decideDecisions you will store, index and reusecb_decide with an UPDATE Can I Use a Different Model? Yes, and that is the point of the design. A model is a function from JSON to JSON, and CURL() calls such functions over HTTPS. Changing the model means changing the URL and the request body inside one wrapper. The same pattern is called OpenAI in Select ChatGPT From SQL? You Bet! and Google's Vision, Translation and Natural Language APIs in the 2018 article above. Try It The cb-jev-functions project in couchbaselabs/agentic-apps holds two folders, cb-jev and cb-decide. Each has the library, an install script, sample data, runnable examples, and an offline test that needs no cluster. In either folder: SQL export TYPESAFE_API_KEY="your-typesafe-key" export CB_PASSWORD="your-couchbase-admin-password" ./install.sh # uploads the library, allowlists the endpoint # then run sql/01_register_functions.sql, sql/02_sample_data.sql and sql/03_examples.sql Start with the sample data, not your own. Watch how many calls each query makes. Compare the labels against your own before you trust them. Jev was in early access behind a waitlist when the DuckDB team wrote about it, so check availability first. Where This Is Heading In 2024 I argued in Why Should Databases Go Natural? that databases need natural queries, and that translating English into SQL is a start, not the end state. Capella iQ takes that first step: you describe what you want, and it writes the SQL++ (iQ Interactive). jev() and cb_decide() take another step towards making the query natural and human. The English stays in the query as a typed predicate or a typed question, next to joins, GROUP BY, and window functions. The result is a boolean, an enum, or an ordinal, which SQL++ already knows how to handle. References CURL Function (Couchbase documentation)JavaScript Functions for Query Reference (Couchbase documentation)Query Functions REST API (Couchbase documentation)Using Google Artificial Intelligence Services in Couchbase N1QL (ODBMS.org, 2018)Select ChatGPT From SQL? You Bet! (DZone, 2023)iQ Interactive: Cool Things for Developers on Couchbase Capella iQ (DZone, 2023)Why Should Databases Go Natural? (DZone, 2024)Introducing System One Models and Jev (TypeSafe AI)Introducing ai_decide (Databricks)Jev and DuckDB: Plain-English Conditions in SQL (DuckDB) Images of Couchbase SQL++ With Jev functions

By Keshav Murthy CORE

Introduction In my previous comparison of Genkit Go and ADK Go 2.0, I covered the everyday tasks. This article is about the agent layer, and instead of comparing features one by one, I did what you would actually do at work: I built something complex in one framework and then tried to build the same thing in the other. Same rules as always: every program here compiled and ran. Genkit Go v1.11.0 and ADK Go v2.1.0, gemini-3-flash-preview plus gpt-5-mini, 2026-08-05. Where I cite internals, they come from the released module sources. One caveat upfront so it frames everything: Genkit's agents API sits behind genkit.WithExperimental() and can change between minor releases, while ADK's agent stack is GA. That is a real ADK advantage. Everything else in this article is about what you can build. The Build: A Research Team You Can Fire and Forget Here is the spec. It is deliberately the kind of thing a real product asks for: An orchestrator agent that coordinates a researcher and a writer.The sub-agents save their work as files (findings.md, draft.md) that end up in one place.The system survives a model outage: if the orchestrator's primary model fails, it falls back to a different vendor.Dangerous tools need human approval before they run.The client fires the task and disconnects; the result waits server-side. In Genkit, this is one program of 90 lines, and it ran on the first try. The interesting part is that requirements 1, 2, and 3 are not code at all. They are three entries in a middleware list: Go orchestrator := genkitx.DefineAgent(g, "orchestrator", aix.InlinePrompt{ // The primary model is BROKEN on purpose. ai.WithModelName("googleai/gemini-nonexistent-model"), ai.WithSystem("You coordinate a team. First delegate research to the researcher, then delegate a short draft to the writer. Finally reply with a one-sentence status."), ai.WithUse( &middleware.Fallback{Models: []ai.ModelRef{ ai.NewModelRef("openai/gpt-5-mini", nil), // req 3: cross-vendor healing }, &middlewarex.Agents{ // req 1: the team Agents: []aix.AgentRef{researcher.Ref(), writer.Ref()}, MaxDelegations: 4, ArtifactStrategy: middlewarex.ArtifactStrategySession, }, &middlewarex.Artifacts{Readonly: true}, // req 2: read the team's files ), }, aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()), ) The researcher and writer are each ~8 lines: an InlinePrompt with a Gemini model, a system prompt, and &middlewarex.Artifacts{} so they get write_artifact tools. Requirement 5 is the client side: Go conn.Send(&aix.AgentInput{ Message: ai.NewUserTextMessage("Research why Go suits AI agents, then have the writer produce a short draft."), Detach: true, }) out, _ := conn.Output() // returns immediately // later, from anywhere: snap, _ := orchestrator.GetSnapshot(ctx, out.SnapshotID) And the run, verbatim: Plain Text detached after 1ms, snapshot fc70253d-55f8-4890-a587-a6aa74d69438 settled: completed after 58s artifact "researcher_1/findings.md" (5041 chars) from researcher artifact "writer_2/draft.md" (3840 chars) from writer orchestrator: I delegated the research task to the researcher (findings.md saved) and the short draft to the writer (draft.md saved). Status: research and draft are complete and saved as researcher_1/findings.md and writer_2/draft.md. Read that output again, because a lot happened in those 58 seconds while no client was connected: the orchestrator's broken Gemini model failed on every turn and healed onto OpenAI every time; it delegated to two Gemini-powered sub-agents through auto-generated delegate_to_* tools; their artifacts were merged into the orchestrator's session with source metadata, and the whole thing settled into a snapshot I could read later from another process. A multi-vendor agent team, running unattended, in 90 lines. Why This Is Easy: Middleware Composes With Agents The capstone works because of one architectural decision: in Genkit, a middleware is a unit that can inject tools, wrap the model call, and wrap tool execution, and the same unit plugs into a plain Generate call or an agent's prompt with the same ai.WithUse(...) line. Capabilities become list entries. The shelf in v1.11.0's plugins/middleware: Retry, Fallback, ToolApproval, Filesystem (sandboxed file tools), Skills, plus the experimental Agents (delegation) and Artifacts. Two of them deserve their own demos, both of which I ran. Cross-Vendor Fallback Is Configuration The plugin ecosystem and the middleware system meet here. Any model from any plugin can be a fallback target for any other, so surviving a vendor outage is this, and nothing else: Go g := genkit.Init(ctx, genkit.WithPlugins( &googlegenai.GoogleAI{}, &openai.OpenAI{}, )) text, err := genkit.GenerateText(ctx, g, ai.WithModelName("googleai/gemini-nonexistent-model"), // broken ai.WithPrompt("Say hi in three words."), ai.WithUse( &middleware.Retry{MaxRetries: 1}, &middleware.Fallback{Models: []ai.ModelRef{ ai.NewModelRef("openai/gpt-5-mini", nil), }, ), ) My run: without the middleware, Error 404, Message: models/gemini-nonexistent-model is not found. With it: Hello there, friend. from OpenAI. 37 lines total, and the same two lines work inside an agent, which is exactly how the capstone's orchestrator heals itself. Human Approval Is Configuration, Too ToolApproval wraps tool execution and interrupts any tool not on its allowlist. The tool itself knows nothing about approvals: Go resp, _ := genkit.Generate(ctx, g, ai.WithPrompt("Delete the files in /tmp/scratch please."), ai.WithTools(deleteFiles), ai.WithUse(&middleware.ToolApproval{}), // nothing pre-approved ) // resp.FinishReason == "interrupted" part, _ := deleteFiles.RestartWith(p, ai.WithResumedMetadata[DeleteInput](map[string]any{"toolApproved": true})) final, _ := genkit.Generate(ctx, g, ai.WithMessages(resp.History()...), ai.WithToolRestarts(part), /* ... */) My run: finish reason: interrupted, then after approval, OK. I've deleted the files in /tmp/scratch. Governance became a deployment decision: you can add approval to an existing agent without touching a single tool. The Same Build in ADK 2.0, Piece by Piece Now the other side, requirement by requirement, with everything verified against the v2.1.0 sources or actually built. The team (req 1): ADK has this, GA. SubAgents with the auto-injected transfer_to_agent tool, or agenttool for function-call delegation. No complaints; this is ADK's home turf. Approval (req 4): ADK has this too, GA, and the declaration is even simpler. functiontool.Config{RequireConfirmation: true} is one flag. I ran the full round-trip in both frameworks and they land within one line of each other (102 vs 101 lines). The difference is the shape: ADK's wire flow is a synthetic adk_request_confirmation function call answered with a map[string]any{"confirmed": true} payload, while Genkit types the pause payload and the resume payload as Go structs and validates every resume against history. Pick your poison: a flag with an untyped protocol, or a typed protocol you wire yourself. Artifacts (req 2): partial. ADK has an artifact service (in-memory or GCS) and a loadartifactstool. What it does not have is the merging story: sub-agent artifacts flowing into the parent session with source attribution is what ArtifactStrategySession gave me for free. Cross-vendor fallback (req 3): you build it yourself, and it bites. ADK has no middleware and its callbacks cannot wrap the model call, so the seam is writing a custom model.LLM decorator. I built it: buffer the primary's output, and on failure call the backup. There is a trap here, and my first version hit it. The backup vendor rejected the call with: Plain Text "The requested model 'gemini-nonexistent-model' does not exist." ADK's LLMRequest carries the model name inside the request, so my OpenAI backup was asked to run the Gemini model. The decorator also has to rewrite the request: Go backupReq := *req backupReq.Model = f.backup.Name() After that fix, it worked (Hi there, friend.). Final tally: 86 lines of decorator and ceremony versus two lines of configuration, and the 86-line version is per-agent, static, one-level-deep, and has to handle mid-stream failure buffering itself. This is what "no middleware" costs in practice. Fire and forget (req 5): not possible. The v2.1.0 source is unambiguous. Runner.Run is a pull-based iterator: nothing executes unless the client consumes events; breaking the loop unwinds the stack, canceling the context aborts the work, and the REST server passes the request context straight into the run. There is no context.WithoutCancel anywhere in the runner or agent paths. ADK can park at an interrupt and survive a process restart by replaying the session event log, which is genuinely robust, but pausing at a boundary is not continuing to work after the client leaves. So the capstone scorecard: two requirements ADK meets as well or better (team, approval flag), one it meets partially (artifacts), one it makes you hand-build with sharp edges (fallback), and one it cannot do (detach). And each of the pieces it does have arrives through a different mechanism: a config field here, a toolset there, a plugin bundle, a hand-written decorator. Genkit's version is one composition model for all of them. The Pieces in Detail The capstone leaned on agent features that deserve their own numbers. All of these ran in both frameworks where both have them. Typed Session State A Genkit agent's state is a Go struct. A tool mutates it; every change streams to the client as a JSON-Patch, and the client reads it back typed: Go type GameState struct { Score int `json:"score"` Moves []string `json:"moves"` } // inside a tool if s := aix.SessionFromContext[GameState](ctx); s != nil { s.UpdateCustom(func(st GameState) GameState { st.Score += in.Points st.Moves = append(st.Moves, in.Move) return st }) } My run streamed [{"op":"replace","path":"","value":{"moves":["e4"],"score":10}] and conn.Custom() returned score=20 moves=[e4 Nf3] as a struct. Prompts read the same state with {{@state.score}. ADK's session state is Get(string) (any, error) / Set(string, any): a map, type assertions at every read, no schema. It works (my run: score=20 (int)), and instruction strings can template state with {score}, but nothing checks anything at compile time. The 2.0 graph engine does have typed parameter binding, but only for workflow function nodes, not for session state in a chat agent. Custom Agent Loops Both frameworks let you write a fully custom agent. In ADK you cannot implement the agent.Agent interface directly, since it is sealed and the compiler rejects any attempt with missing method internal; the sanctioned path is agent.New(agent.Config{Run: ...}), a full execution override, and it ran in my tests. The difference is what you write. Genkit's SessionRunner hands you a turn loop with history, per-turn tracing, snapshot writes, and panic recovery built in; my draft-then-self-critique agent is 57 lines. The ADK version of the same agent is 96 lines, because a Run override means producing raw session.Events: dig the question out of the event log, call the raw model layer, construct and yield events yourself. Agents Defined in .prompt Files DefinePromptAgent binds an agent to a .prompt file by name; the model, config and persona are data, not Go: Go pirate := genkitx.DefinePromptAgent(g, "pirate", aix.WithSessionStore[any](localstore.NewInMemorySessionStore[any]()), ) 26 lines of Go, and my run answered: Rewrite 'em in Python only if ye want yer agents movin' as slow as a barnacle-covered scow in a dead calm. Non-engineers edit the persona without touching code, and the same dotprompt format works across Genkit's other language SDKs. ADK's equivalent is an Instruction string or an InstructionProvider where you load files yourself; its YAML agent-config system exists but lives in an internal/ package only ADK's own CLI can reach. Where ADK's Agent Stack Is Ahead The list I would actually weigh on the other side, verified in the same source dive: GA stability, against everything above being on Genkit's experimental track.The graph engine as the runtime: every ADK agent runs as a node in the workflow engine, and the node kit (typed function nodes, routing, per-node schema validation, retries, nested graphs) is a public API.Schema-validated human input in workflows: a graph node can declare a JSON schema for the human's answer, validated on resume. For form-like HITL inside a pipeline, that is ahead of anything Genkit ships.Task isolation: ModeTask sub-agents get their own conversation scope with an auto-injected finish_task tool and scope-filtered history.Live audio: RunLive with modalities, speech config and transcription has no Genkit Go equivalent. The Scorecard Non-blank lines including imports. Every program ran. ExperimentGenkit GoADK Go 2.0The capstone: team + artifacts + fallback + detach90not buildable (fallback DIY, detach impossible)Cross-vendor fallback37 (configuration)86 (hand-written decorator, request-rewrite gotcha)Approval on an existing tool45 (middleware config)flag exists (GA), untyped wire protocolTyped state, 2 turns, live patches80, typed96, map + castsCustom draft-critique agent5796Agent from a .prompt file26no equivalentUndo the last exchange41no API: append-only event logContext caching40, one method calllifecycle DIY with a raw genai client Conclusion The agent layer is where Genkit's two architectural bets pay off together. The plugin ecosystem means models from different vendors, local runtimes, and vector stores are interchangeable references. The middleware system means capabilities (delegation, artifacts, fallback, retries, approval, sandboxed files, skills) are composable list entries that work identically in a one-shot generate and inside an agent. Stack those two on top of typed sessions, snapshots, and detach, and a fire-and-forget multi-vendor agent team is a 90-line program. I know because I ran it, and I also know what the same spec costs in ADK because I built the pieces: one requirement hand-written with real sharp edges, and one that cannot be built at all. ADK's agent stack remains the right pick when its strengths are your requirements: GA guarantees, graph workflows with schema validation, task isolation, live audio, and the Google Cloud platform around it. But if the question is "how hard is it to build something complex", the answer I measured is: in Genkit, complexity composes; in ADK, each capability is its own project. Further reading: Genkit Go vs ADK Go 2.0: A Hands-On Look at Boilerplate and Low-Level ControlGenkit Go basic-agents sampleGenkit middleware documentationADK 2.0 release notesGenkit GitHub repositoryADK Go GitHub repository

By Xavier Portilla Edo CORE

Why Put a WAF in Front of the Application? Web application firewalls are often introduced as a list of security features: SQL injection protection, cross-site scripting detection, bot mitigation, rate limiting, and access control. Those capabilities matter, but they are only part of the operational problem. The more difficult questions usually appear after deployment: Where should the WAF sit in the traffic path?How do we avoid exposing the origin server?How do we tune blocking rules without interrupting legitimate users?How should security events be investigated?What happens when the application has unusual request formats?How much control do we retain when using a self-hosted solution? This article describes how SafeLine can be used as a self-hosted Web Application Firewall and reverse proxy. It focuses on the deployment model, request-processing flow, practical configuration, and operational lessons that matter when protecting real web applications. The WAF's Position in the Architecture A WAF operates at the HTTP layer. It can inspect requests before they reach the application and evaluate information such as: HTTP methodsURLs and query parametersRequest bodiesHeaders and cookiesSource IP addressesRequest frequency and behavioral patterns This position gives the WAF visibility that a network firewall does not have. A network firewall can allow or deny traffic based on addresses and ports, but it usually cannot determine whether a request contains an SQL injection payload or whether a client is repeatedly probing an administrative endpoint. The WAF also provides a safety boundary around applications that may contain unknown vulnerabilities. It does not replace secure coding, dependency updates, authentication, or database protections. It adds another enforcement point between an untrusted client and the application. The origin application should accept traffic only from the SafeLine host or from a trusted upstream load balancer. If users can bypass the WAF and connect directly to the origin, an attacker can simply avoid the protection layer. Figure 1. SafeLine sits between internet clients and the protected application. The origin should accept traffic only from a trusted SafeLine host. SafeLine's Deployment Model SafeLine is designed to run in a self-hosted environment. It acts as a reverse proxy, receives traffic for configured applications, analyzes the requests, and forwards acceptable requests to the configured upstream service. This deployment model is useful for teams that need: Control over where traffic and logs are processedA WAF that can run on their own infrastructureA relatively simple installation processProtection for applications deployed on virtual machines, bare-metal servers, or private infrastructureA management interface for applications, policies, and security events SafeLine can be installed on a Linux server using Docker-based deployment. Installation can be completed with a single command: Shell bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en Full guide to install SafeLine: https://ly.cyberserval.tech/b3QLLSJ. After installation, the administrator opens the management console, completes the initial setup, and adds the application that needs protection. Figure 2. Add the application you want to protect in the SafeLine management console. For a production deployment, treat the installation host as a security-sensitive system. At minimum, the host should have: A supported Linux distributionDocker installed and maintainedPersistent storage for configuration and logsRestricted access to the management interfaceA backup strategy for configurationMonitoring for CPU, memory, disk, and network usage The management interface should not be exposed broadly to the public internet. Restricting it by firewall policy, VPN, or an administrative allowlist reduces the attack surface. How a Request Is Processed When a request reaches SafeLine, it passes through several conceptual stages. First, SafeLine identifies the application and routing configuration. It determines which upstream service should receive the request. Next, it inspects the request structure. Depending on the request, this may include the URL, parameters, headers, cookies, and body content. The detection engine then evaluates whether the request resembles an attack. Common categories include: SQL injectionCross-site scriptingCommand injectionPath traversalFile inclusionMalicious scannersAutomated abuseExcessive request rates Finally, SafeLine applies the configured action. The request may be allowed and forwarded, logged for observation, challenged, rate limited, or blocked. Figure 3. HTTP Request Processing Flow in SafeLine A request can be observed before enforcement is enabled, allowing the team to identify legitimate traffic that needs a narrow exception. A team should not begin by blocking every suspicious request in a production environment without understanding the application's normal traffic. A safer rollout sequence is: Put the application behind the WAF.Observe requests and security events.Identify legitimate traffic that may look unusual.Adjust application-specific rules.Enable blocking for high-confidence attack categories.Continue reviewing events after enforcement begins. Deployment Example Assume the application listens on port 8080 and the public domain is app.example.com. The basic workflow is: Install SafeLine on a Linux host.Point the DNS record for app.example.com to the SafeLine host.Add the domain as a protected application.Configure HTTPS and certificates.Configure the upstream as http://application-host:8080.Restrict direct access to the application host.Verify that normal requests reach the application.Send controlled test requests and confirm that events appear in the dashboard. The most important network change is often the origin restriction. The application server should no longer accept arbitrary internet traffic. For example, its firewall can permit the application port only from the SafeLine server's private IP address. This is easy to overlook because the WAF may appear to be working even while the origin remains publicly reachable. Tuning Without Creating an Availability Problem A WAF cannot understand every business rule automatically. Some applications intentionally accept content that resembles an attack payload. Examples include: Security testing platformsDeveloper tools that accept code as inputSearch features that process special charactersAPIs that receive structured expressionsCMS systems with complex administrative requests The correct response to a false positive is usually a narrow exception. Avoid disabling protection for the entire application when only one endpoint or parameter requires special handling. Figure 4. SafeLine’s Custom Allow Rule and Deny Rule Configuration Interface False-positive handling should move from an observed event to the narrowest possible exception. For example, an exception for /api/search and one request parameter is easier to review than a global allowlist for the source IP. Narrow rules also reduce the chance that an attacker can reuse the exception against other endpoints. The application team should be involved in this process. Security events are often difficult to interpret without knowing what the endpoint is supposed to accept. Rate Limiting and Anti-Bot Challenge Application-layer attacks are not limited to SQL injection or cross-site scripting. Automated traffic can also consume resources, abuse login forms, scrape content, and repeatedly call expensive API endpoints. SafeLine provides two complementary controls for this type of traffic: Rate Limiting and Anti-bot Challenge. Rate limiting restricts how frequently a client can access an endpoint. It is useful for protecting login pages, password reset flows, search APIs, file upload endpoints, report-generation APIs, public APIs, and administrative paths. Anti-bot challenges address a different problem. Instead of relying only on request frequency, they challenge clients that exhibit suspicious automated behavior. Legitimate users can usually complete the challenge, while basic scripts, crawlers, and other automated tools may be unable to proceed. These controls are most effective when applied according to the behavior and sensitivity of each endpoint. A global request limit may block legitimate users, especially when many users share the same corporate or mobile network address. A challenge may also be inappropriate for machine-to-machine APIs that cannot interact with a browser-based verification flow. When configuring rate limits and anti-bot protection, consider: Requests per IP addressRequests per account or sessionRequests per endpointBurst behaviorResponse codesLegitimate client retry behaviorWhether the endpoint is accessed by browsers, mobile clients, or APIsGeographic and network distributionWhether the client can complete an interactive challenge A useful operational metric is not simply the number of blocked requests or triggered challenges. It is the relationship between automated traffic and application health. If an increase in blocked requests and challenged sessions coincides with lower application latency, fewer authentication failures, and reduced load on expensive endpoints, the controls may be addressing a real operational problem. The recommended approach is to begin in observation mode, identify the endpoints receiving suspicious automated traffic, and then apply the narrowest suitable control. Rate limiting is usually appropriate for high-frequency access, while anti-bot challenges are more suitable when the goal is to distinguish normal users from automated clients. Investigating Security Events A WAF is most useful when its events can support an investigation. For each event, the operations team should be able to answer: Which application was targeted?Which endpoint received the request?What type of attack was detected?When did the activity begin?Was the request blocked, challenged, or only logged?Did the source continue after the first response?Did the application generate errors?Are other endpoints or applications seeing the same pattern? SafeLine's management interface provides a central place to review protection events and traffic activity. It can help teams distinguish isolated scans from persistent automated activity. Figure 5. SafeLine’s Security Posture Dashboard However, WAF logs should be correlated with application and infrastructure logs. A blocked request may be useful evidence, but it does not prove that the application was compromised. Likewise, an allowed request does not prove that the application is safe. For a complete incident review, correlate WAF events with web server logs, application logs, authentication logs, and host and network telemetry. SafeLine live demo: https://ly.cyberserval.tech/i61INy7. SafeLine Pro Annual License Promotion SafeLine has reached more than 700,000 installations. That number is useful because it reflects broad adoption across different environments, application types, and operational constraints. To mark this milestone, the SafeLine team is offering a customer feedback discount for a Pro annual license. Figure 6. SafeLine 700K Installs Promotion Campaign The promotion details, eligibility requirements, validity period, and redemption process should be confirmed on the campaign page: https://ly.cyberserval.tech/DZAGbHo. Final Takeaways A WAF is not valuable simply because it has a long feature list. It is valuable when it can be placed correctly in the request path, tuned for the application, monitored by the operations team, and maintained as part of the broader security architecture. SafeLine's self-hosted reverse-proxy model makes it practical to introduce application-layer protection without redesigning the application itself. The most important implementation lessons are: Ensure all public traffic passes through the WAF.Restrict direct access to the origin server.Start with observation before enabling aggressive blocking.Handle false positives with narrow, reviewable exceptions.Apply rate limits to specific high-risk endpoints.Correlate WAF events with application and infrastructure logs.Continue using secure development practices alongside the WAF.Evaluate commercial licensing based on operational requirements rather than feature count alone.

This is the first of eight articles about how web apps work on iOS, and it has less code than the rest. Here I look at where the oddities come from, the ones everyone who ships a PWA to the iPhone ends up living with. Open the <head> of almost any PWA built in 2026 and next to the manifest link you'll most likely find the apple-mobile-web-app-capable meta tag. Safari's old reference says it plainly: "Available in iOS 2.1 and later." Version 2.1 came out on September 12, 2008, when the App Store had already been running for two months. So full-screen web apps on the iPhone showed up after Apple had turned to the native SDK, even though the story is usually told the other way round. PWA behavior on the iPhone is full of small details like this. Behind the icon on your Home Screen are twenty-five years of attempts to turn a web page into a program, and plenty of the decisions made along the way are still sitting in your code. When Windows Trusted HTML as It Trusted Itself In 1999, together with Internet Explorer 5, Microsoft released HTML Applications, or HTA. For its time, the idea was very clear: HTML, CSS, and scripts can already draw an interface, so why not launch such a page in its own window like an ordinary Windows program, with no address bar and no browser buttons? A file with the .hta extension was opened by a system utility, mshta.exe, which took the IE engine but left its security model outside the door. Microsoft wrote about this in its documentation without a trace of embarrassment. HTAs are called "full-fledged applications" that "are trusted," and the summary reads: "an HTA runs like any executable (.exe) file written in C++ or Visual Basic." The user was asked once, before the download, whether to save the file or run it. After that, the app could do everything an ordinary program can: write to disk, start processes, read the registry. It looked roughly like this: HTML <html> <head> <title>Inventory</title> <hta:application applicationname="Inventory" border="thin" showintaskbar="yes" /> <script language="JScript"> function openLog() { var shell = new ActiveXObject("WScript.Shell"); shell.Run("notepad.exe C:\\inventory\\log.txt"); } </script> </head> <body> <button onclick="openLog()">Open log</button> </body> </html> No permission dialog, no origin, no sandbox: the WScript.Shell object runs anything with the current user's rights. For an internal admin tool in the late nineties that was convenient, sysadmins wrote setup wizards, support dashboards, and utilities in HTA that otherwise would have had to be built in Visual Basic. All of it ran only on Windows and only on the IE engine, so HTA couldn't boast the cross-platform reach that people usually pick the web for. Microsoft even patented the approach: the application "Method and apparatus for writing a Windows application in HTML" was filed on May 20, 1999, and patent US 6,662,341 was granted on December 9, 2003. Why .hta Is Still Stripped from Email Attachments The model of "download some markup, get the rights of a program" aged very badly. In MITRE ATT&CK, mshta.exe has a technique of its own, T1218.005, which describes how attackers run malicious .hta files, JavaScript and VBScript through the utility as early as the initial-access stage. A trusted system binary is handy because it slips past policies that only watch for unfamiliar executables. The LOLBAS project, which catalogs legitimate Windows programs that can be abused in attacks, lists mshta.exe with paths in System32 and SysWOW64 all the way up to Windows 11. The loudest episode happened in April 2017. FireEye described attacks through RTF documents for Microsoft Office: inside sat an OLE object with a link; Word went to the attacker's server on opening, got an HTA from there, and handed it to a handler that executed the file. The vulnerability was assigned CVE-2017-0199, and Microsoft fixed it in the update of April 11, 2017. In November 2021, CISA added it to its catalog of vulnerabilities exploited in real attacks. HTA itself wasn't the bug in that chain. It was just the full trust the attackers were after. Today Outlook blocks .hta attachments by default, on the same list as .exe and .bat, and I can't think of a better epitaph for the technology. What do PWAs have to do with it? Well, they look like apps, open from an icon and work offline, but they stay inside the browser security model, with its origins, HTTPS, sandbox, and permissions the user grants one at a time. When, in the next parts, a Service Worker refuses to register over plain HTTP, or iOS won't let a web app near Safari's data, remember mshta.exe: that's exactly the kind of neighbor browsers don't want to see next door anymore. June 2007: No SDK Needed On January 9, 2007, at Macworld, Steve Jobs showed the iPhone and said nothing about third-party apps. Today that looks odd, but back then the phone was sold as a finished product with mail, a browser, maps and an iPod inside, and the question of a platform simply never came up on stage. Developers got their answer on June 11 at WWDC. According to a transcript that 9to5Mac published later, Jobs said the iPhone has "the full Safari engine" inside, so you can write Web 2.0 and Ajax apps that "look exactly and behave exactly like apps on the iPhone," and then added: "There's no SDK that you need!" Apple's press release from the same day says the same thing in a calmer voice. Web apps would look and behave like built-in ones, and they could make a call, send an email, and drop a pin in Google Maps. They would extend what the iPhone can do "without compromising its reliability or security." And they could be updated by "simply changing the code on the developers' own servers." It sounds almost like a definition of a PWA, but in June 2007 there wasn't much behind those words. The app lived in a Safari tab, you reached it through a link or from your bookmarks, and all the integration with the phone came down to tel: and mailto: links plus addresses that opened in Maps. There was no camera API, no background work, and no storage in today's sense. There was no Home Screen icon at all: the Home Screen itself was fixed, and without a jailbreak you couldn't rearrange anything on it. Developers took to this model anyway, and with real enthusiasm. At iPhoneDevCamp in July 2007, Wired reported, the person everybody was talking about was Joe Hewitt, one of the creators of Firefox and Firebug. He had put together iUI, a set of templates that made a site on the iPhone look and act almost like a built-in app. Many developers had already adopted the templates as a base, and Wired wrote that they were clearly becoming the default look of the iPhone web. That August, Hewitt, by then at Facebook, built a version of the social network specifically for the iPhone, and it was a web app too: the native client didn't come out until the summer of 2008. To me, this model still feels more honest than the way it was later retold. The web didn't pretend to be deeply integrated with the phone. It asked the system to perform a system action, and that habit survived all the way to Web Share, and to the way an installed web app on iOS now asks for permission to send notifications. An Icon in January, an SDK in March The button this whole series exists for arrived with iPhone OS 1.1.3 on January 15, 2008. TidBITS described the novelty in detail at the time: Safari got a plus button, its menu had an Add to Home Screen item, and an icon appeared on the Home Screen that Apple called a Web Clip. The same release let you edit the Home Screen for the first time, icons started to wiggle under your finger, and the number of screens went up to nine. Apple's documentation has said the same thing ever since: "The Web Clip feature is available in iOS 1.1.3 and later." The apple-touch-icon was born along with the icon. A site could drop a PNG in the root or point to one in a <link>, and if it did neither, iOS went looking in the root for a file called apple-touch-icon.png. Later versions also tried variants like apple-touch-icon-57x57-precomposed.png. The -precomposed suffix, which has worked since iOS 2.0, was simply a request: Safari, please don't add your own effects to this icon. The era of glossy highlights is long gone, yet I still see requests for these files in server logs, on servers where nobody ever put them. Still, an icon added this way opened an ordinary Safari tab with all the toolbars. The first Web Clip was a bookmark moved to the Home Screen, and nothing more. Apple, meanwhile, had already changed its mind. Back on October 17, 2007, a short letter from Jobs appeared on the company's site. It began with the words "Let me just say it: We want native third-party applications on the iPhone" and promised an SDK in developers' hands in February. He blamed the delay on security: Apple was trying to give developers an open platform and protect users from viruses and malware at the same time, and he pointed to Nokia, which on some of its new phones allowed only apps carrying a digital signature from a known developer. Four months passed between the June "no SDK needed" and this letter. The SDK missed February. It was shown on March 6, 2008, together with the App Store, and the press release from that day contains a phrase that set the course for the next fifteen years: third-party apps "must be approved by Apple and will be available exclusively through the App Store." The store itself opened on July 10, 2008, through iTunes, and by the time the iPhone 3G went on sale, Apple was promising more than 500 native apps in it. iPhone OS 2.0, with the App Store built in, came out the next day, July 11. Full-Screen Mode Came After the App Store Now the detail I started with. The apple-mobile-web-app-capable meta tag hides Safari's interface and opens a site from its icon in full screen. Its neighbor, apple-mobile-web-app-status-bar-style, controls the strip with the clock. Apple marked both as available since iOS 2.1, and that version came out on September 12, 2008. The same reference describes the navigator.standalone property, which lets a page tell that it was launched from the Home Screen. The apple-touch-startup-image splash picture was added only in iPhone OS 3.0, and icons for different screen resolutions in 4.2. So Apple was still finishing the web app mode when the native platform was already on sale. There is no "first they bet on the web, then dropped it" in this timeline. Web apps got modest, minimal support and lived on it for almost ten years. What iOS kept from that period is a set you can still find in old projects: a separate meta tag for every capability, no general description of the app, and installation only by hand, through the Safari menu. A lot of projects have dragged the launch-mode check along since those days, and it usually looks like this: JavaScript const standalone = window.navigator.standalone === true || window.matchMedia('(display-mode: standalone)').matches; if (standalone) { document.documentElement.classList.add('launched-from-home'); } The first half of the condition is a legacy of iPhone OS 2.1, and the second arrived with the web manifest. On the iPhone, this check usually fails not because of the code but because of expectations: navigator.standalone exists only in WebKit on iOS, while the value of display-mode depends on exactly how the site was added to the Home Screen and what was in the manifest at install time. We'll break down how this works in Part 4 of this series. Ten Years on Meta Tags and AppCache Between 2008 and 2018, web apps on the iPhone stood on two things: those same Apple tags and the Application Cache. The tags were simple and tedious. For the splash screen, you had to draw a separate image for almost every iPhone and iPad screen size, in the right orientation and with the right media query. I've come across repositories where the folder of splash images weighed more than the app itself, with a comment sitting next to it along the lines of "don't delete, white screen on iPad otherwise." Offline was worse, and its roots, by the way, aren't Apple's. On May 31, 2007, less than two weeks before WWDC, Google released Gears, a browser plugin that gave pages a local SQLite database, background workers, and a LocalServer module that could serve saved HTML, scripts, and images without a network. Gears was what powered offline access in Gmail. On February 19, 2010, the Gears team wrote on its blog that its effort was shifting toward getting all of Gears' capabilities into web standards like HTML5, and that the plugin itself would get no new features. For the history of PWAs this is a telling moment: a browser vendor retired its own proprietary technology in favor of a standard, and Service Worker would later travel roughly the same road. The standard heir to LocalServer was AppCache. You declared a text file listing your resources in <html manifest="...">, and the browser promised to open the page without a network. Safari on the iPhone supported it, and for web apps launched from the Home Screen it was for a long time the only standard way to open without a connection instead of showing a "You are not connected to the Internet" error page. A typical manifest looked harmless: HTML CACHE MANIFEST # v12 CACHE: /index.html /app.js /styles.css NETWORK: /api/ FALLBACK: / /offline.html The catch was in the update rules. Apple's documentation warned outright that changing the HTML file does nothing until the manifest itself changes, so everybody kept a comment with a version number in it and remembered to bump it. If the manifest did change, the new cache was built in the background and applied only on the next visit, which meant the user saw the old version first and the new one only after a restart. If a single file from the list came back with an error, the update wasn't applied at all. Some users could get stuck on an old bundle for a long time without even knowing it. In May 2012, Jake Archibald, who was then building Lanyrd's mobile site on AppCache, published an article in A List Apart with a famously rude title: "Application Cache is a Douchebag." He went through all of these traps, and if I put it in my own words, AppCache looked like an offline mode but behaved like a hidden deploy that you don't control. Funnily enough, Archibald later ended up among the editors of the Service Worker specification, along with Alex Russell, who comes up below. Service Worker grew straight out of that grievance. Instead of a declarative list, you got a programmable fetch handler, an explicit lifecycle, and a Cache API where you decide for yourself what to store and when. If a legacy project still has <html manifest="cache.appcache"> sitting in it, don't try to migrate gradually by adding a worker alongside. Two offline mechanisms on one origin fight over the same page, and the bug then looks like "some users have the old version" even though git is perfectly clean. Remove AppCache, ship the worker, and check a cold start on a real device. How Apple Campaigned for HTML5 On April 29, 2010, Apple published Jobs's open letter "Thoughts on Flash," explaining why Flash wouldn't be coming to the iPhone, iPod and iPad. Most of the text is about battery life, stability, and the touch interface, but the most interesting part is how it talks about the web. "Rather than use Flash, Apple has adopted HTML5, CSS and JavaScript," Jobs writes, meaning open standards that its devices support quickly and efficiently. The letter ends with a forecast: "New open standards created in the mobile era, such as HTML5, will win on mobile devices (and PCs too)." There's also a line that's hard to read today with a straight face. Adobe, according to Jobs, claimed that Apple's decision was driven by business and a wish to protect the App Store, "but in reality it is based on technology issues." On Flash, I think, he turned out to be right in the end. But the forecast of HTML5's victory came from a company whose Home Screen web apps couldn't send notifications and relied on that very AppCache for offline. Safari was one of the best mobile browsers at the time, and WebKit inside it really did push standards forward, while the app mode stayed roughly where it had been left in 2008. Developers saw the contradiction. In November 2009, that same Joe Hewitt, by then the author, by TechCrunch's estimate, of the most popular iPhone app ever, handed the native Facebook client over to another engineer. He told the publication that the decision was entirely about Apple's policies and that he was fundamentally opposed to the review process. His explanation ended like this: "The web is still unrestricted and free, and so I am returning to my roots as a web developer." I think this contrast explains a whole decade better than any conspiracy theory. Apple was happy to promote the web as a technology for pages and much more careful about the web as an app platform on its own phone. At least that's how it looks from the outside, since the company doesn't publish its motives for each missing API. The most popular explanation for the pause comes down to the App Store commission. You can't prove that kind of causality from public sources, and I wouldn't build technical decisions on it. The engineering explanation is duller, but it can be checked: background execution, persistent storage, and notifications for websites sit badly with a phone's battery, memory, and permission model, and WebKit on iOS has to keep compatibility for years with everything it has already shipped. The commercial incentives don't go away because of that. They just don't explain why, for example, cache.addAll fails entirely because of one broken URL. Who Else Tried to Turn the Web into a Platform? Apple wasn't the only one in those years to look at HTML as a language for apps, and almost all of these attempts ended badly. They're worth remembering, if only so you don't repeat old arguments in PWA debates. Palm showed the Pre smartphone at CES in January 2009, and on June 6 it went on sale through Sprint. The webOS operating system was daring by design: the interface was built on WebKit, and third-party apps were written in HTML, CSS, and JavaScript with the Mojo framework. The Verge, in a long piece on Palm's history, quoted one of its sources explaining the choice: instead of keeping a person around whose job is writing code to center text, you can "stand on the shoulders of giants and let WebKit do it." According to the same piece, Palm ported the V8 engine for speed and was the first to ship it on a mobile device. The rest of the story is short. In 2010, HP bought Palm; on August 18, 2011, it announced it would stop making webOS devices, and in February 2013 it sold the system to LG, where it still lives on in TVs. Google came at it from a different side. The Chrome Web Store opened on December 7, 2010, and besides extensions, it distributed Chrome Apps, packaged and hosted apps that ran in Chrome almost like desktop programs. On August 19, 2016, the Chromium blog published "From Chrome Apps to the Web": packaged apps on Windows, Mac, and Linux were actively used by about 1% of users. Google announced that it would remove them from those platforms and told developers to move to Progressive Web Apps with Service Workers and Web Push. By mid-December 2017, Chrome Apps had vanished from search and the Chrome Web Store directory, although installed apps on those systems kept launching for a while. The final shutdown dates were pushed back several times after that, but the direction never changed again. Mozilla went further than anyone and built a whole operating system in which an app is a web page. The first commercial Firefox OS phone, the ZTE Open, went on sale through Telefónica in Spain on July 2, 2013, for about 90 dollars, aimed at emerging markets. On December 8, 2015, Mozilla announced it was ending development and sales of Firefox OS smartphones, and gave a fairly honest reason: "we weren't able to offer the best user experience possible." In the same statement, Mozilla wrote, not without pride, that Firefox OS had proved the flexibility of the web on devices ranging from cheap smartphones to HD TVs. In September 2016, commercial development stopped altogether, but the code didn't disappear. A fork grew into KaiOS, which ran feature phones such as India's JioPhone. Microsoft, in Windows 8, which went on sale on October 26, 2012, made HTML5 and JavaScript first-class languages for Windows Store apps, right alongside C# and C++. For that, there was the WinJS library, which was open-sourced under the Apache license in April 2014. The last commit to its GitHub repository is dated May 2020. Of the four, webOS survived most visibly, and even that moved into TVs, while Firefox OS carried on in someone else's fork for feature phones. Put these stories together, and the conclusion is an unpleasant one for people who love grand plans. The web as an app language lost every time it needed its own operating system, its own store, or an ecosystem it had to build from scratch. PWAs survived because they ask for none of that and ride inside a browser the user already has. That is exactly what makes iOS a special case. The browser on the iPhone belongs to the same company as the platform with the store, and every browser there has to run on WebKit (apart from regulated exceptions in the EU), so the pace at which PWAs gain new capabilities is set by Apple. WebKit and the DMA get a full treatment in Part 3 of this series. A Name for Something That Already Existed On June 15, 2015, Alex Russell of the Chrome team published "Progressive Web Apps: Escaping Tabs Without Losing Our Soul." He writes that the evening before, over dinner, he and the designer Frances Berriman had listed the properties of a new class of apps: they adapt to any screen, work offline thanks to Service Worker, are always fresh, are served only over TLS, can be found by search engines, can be put on the home screen, and open through an ordinary link. Berriman called them Progressive Open Web Apps. In the body of the post, they shrink to Progressive Apps, while the headline carries the combination that ended up sticking. My favorite line from it says that such apps aren't packaged or distributed through stores because "they're just websites that took all the right vitamins." The same post has a detail worth reading twice if you care about iOS. Russell describes a scenario using the Chrome Dev Summit site: at first it's an ordinary tab, and on the second, third, or fourth visit, once the browser is confident the site is used regularly, it shows an install prompt on its own, filled in with data from the manifest. Safari still has no such prompt, and that's exactly why so many PWAs for the iPhone have a homemade hint somewhere in the interface, with an arrow pointing at the Share button. The word "progressive" comes from the term progressive enhancement. A basic site works in any browser, and wherever Service Workers, a manifest, and push are available, offline support, an icon, and notifications get layered on top. Installation stays optional. If your app shows a blank screen without JavaScript and counts as broken without a registered Service Worker, you've built the exact opposite of what the authors of the term had in mind. That will show up first on an old corporate iPad sitting behind a temperamental proxy. The first big examples appeared quickly, and nearly all of them were about Android. On November 10, 2015, Flipkart engineer Aditya Punjani described how Flipkart Lite was built, the mobile site of India's largest online store: a single-page React app, HTML shells served from the Service Worker cache, offline browsing of recently viewed products, and notifications through Web Push. On April 6, 2017, Twitter announced Twitter Lite at mobile.twitter.com and promised it would take up less than a megabyte on the device. The same announcement has a caveat that deserves a careful read: notifications and offline worked "in Google Chrome and other modern browsers on Android devices." On the iPhone in 2017, both of them were just a fast website, because Service Worker didn't exist in Safari yet. Cupertino, We Have a Problem! iOS 11.3 came out on March 29, 2018, and Apple's press release about it talks about new ARKit features, Animoji, health records in the Health app, and the battery health screen. Not a word about Service Workers. It turned up in the release notes for Safari 11.1, which shipped with that iOS version: support for Service Workers with debugging in Web Inspector, the Cache API, the Web App Manifest for sites added through Add to Home Screen, and Payment Request for paying with Apple Pay. The WebKit overview separately notes that Service Workers are partitioned by the origin of the top-level document, so they can't be used to track a user across sites, and the implementation details are described in the post "Workers at Your Service." The word "support" has to be read very narrowly here. Maximiliano Firtman, who had already written a piece during the beta under the headline "Cupertino, we have a problem," listed right after the release what had been left out. There was no install prompt, so the user had to open the Share menu themselves and find the Add to Home Screen item. There were no push notifications. iOS ignored the icons from the manifest and took apple-touch-icon, and without one it built an icon from a screenshot of the page. There was no splash screen, most of the colors from the manifest were dropped too, and when you left the app its state wasn't preserved: you stepped out to check an SMS code, came back, and the PWA started from scratch. Even Uber's careful PWA threw the user out into Safari at the login step, after which nothing was left of the app mode but the name. Apple's decision makes engineering sense to me. The company switched on the standards it knew how to support inside its own process and under its own battery limits, and it didn't copy Chrome's behavior on Android. For people who had already shipped a PWA on Android and expected identical behavior on both platforms, it looked like mockery, and I can see why they felt that way. What Came Later After that, the story moved in fits and starts. Push for web apps added to the Home Screen arrived only in iOS and iPadOS 16.4, in March 2023, and WebKit's announcement immediately spelled out the conditions: notifications are available only after installation, and permission is requested in response to a user action. The same release brought the Badging API and the ability to add sites to the Home Screen from third-party browsers. Declarative Web Push, which no longer needs an installed Service Worker to show a notification, was added in iOS 18.4 in March 2025. In iOS 26, a site added to the Home Screen opens as a web app by default, and a toggle called "Open as Web App" controls that. I'll say more about storage, because that's where the most outdated advice has piled up. The figure of 50 MB came from write-ups in 2018 and stopped describing reality long ago. The "seven-day rule" appeared in March 2020, when Intelligent Tracking Prevention in Safari started wiping script-writable storage for sites the user hadn't interacted with in seven days of browser use. In the same post, WebKit clarified that web apps on the Home Screen have their own counter of days of use, and their own data shouldn't be deleted in normal operation. The system can still evict it if the device runs out of space, and Apple doesn't name a single public quota figure that covers every case. I draw one practical conclusion from all this. The server stays the source of truth, local storage is a cache that may vanish one day, and a cold start after such a wipe needs to be tested on a real phone. Five years between Service Worker and push meant one simple thing for a product: if your app needed notifications, until spring 2023 native was your only option on the iPhone, however good the PWA turned out. Why an Engineer Needs All This in 2026 The history is here because it's useful for the job. It helps you avoid mistaking an incompatibility for a bug in your own app, and the typical case goes like this: a team writes a correct manifest, checks it in Chrome DevTools, and on the iPhone sees an old icon or a screenshot in its place. Someone who knows only the manifest spec starts hunting for a mistake in the JSON. Someone who remembers the Web Clip looks first at apple-touch-icon, at the way the app was installed, and at the fact that the icon was saved on the device at the moment it was added and most likely won't update by itself. The minimal <head> that I put even into internal admin tools grows out of that same past: HTML <link rel="manifest" href="/manifest.webmanifest"> <meta name="apple-mobile-web-app-capable" content="yes"> <meta name="apple-mobile-web-app-status-bar-style" content="default"> <link rel="apple-touch-icon" href="/icons/apple-touch-icon.png"> On recent iOS versions, the apple-mobile-web-app-capable tag simply repeats display: standalone from the manifest, but on a line of tablets that go years without updates, that duplicate is cheaper than working out why the app opened in a tab. Make the icon an opaque 180×180 PNG in a separate file: the system isn't obliged to use a tab favicon for the Home Screen. Another benefit is that it pays to break the word PWA into parts before you even size up the task. On the iPhone, it hides three separate capabilities, and they arrived in different years. The icon and launching without a browser UI came in 2008 along with the meta tags; solid offline support became possible in 2018 with Service Workers, and notifications showed up in 2023, only for installed apps. An internal CRM often gets by with the first. A warehouse app needs the second, plus a queue of operations: iOS still has no Background Sync, so data will only go to the server the next time the app is opened. A messenger needs the third, and this is exactly where the web on the iPhone faces its tightest limits. This is also where the habit of naming a minimum iOS version comes from. "We support iOS" isn't a requirement: iOS 15, 16.4 and 18.4 are very different from one another. In 15, web apps have no push at all; 16.4 brought standard Web Push, and 18.4 added Declarative Web Push on top. Besides, in corporate environments old iPads stay in service much longer than the product team assumes. So I check not a list of APIs but scenarios on a real device: API names say little about how things actually behave in practice. The app should install with the right name and icon and open without the browser bar. After a full close, it should launch with no network; after sign-out, it shouldn't leave the previous user's data in the cache; and it should ask for notification permission only after installation and on a button tap. Lighthouse can do what it's good at, but it won't walk through the Web Clip install path for you (Apple calls that Home Screen icon a Web Clip). Another source of confusion comes from the 2016-2018 Android articles with their beforeinstallprompt, install banners and Google Play packaging. There is no such event on the iPhone, so a snippet that triggers a "native install dialog" is useless for Safari. I'd keep recipes like that in a wiki marked "Chromium only". Otherwise, sooner or later someone will carry them into the shared bundle and spend a week figuring out why the event doesn't fire on the iPhone, when it isn't supposed to. In requirements, instead of "make a PWA," I write three lines: the minimum iOS version, whether push is needed, and what has to work offline. If "sync in the background" shows up in the last line, it's better to end the "a PWA will do" conversation before anyone estimates in story points. And if everything adds up, the actual engineering begins. In Part 2 of this series, we'll put together a minimal setup of HTTPS, a manifest, and a Service Worker, and look at why updating without App Store review works in both directions: you can ship a broken sw.js in a minute, and in that same minute break the app for all your users at once.

Genkit and ADK (Agent Development Kit) are both Google-backed Gen AI frameworks with first-class Go SDKs, and I keep getting the same question: what is actually different when you sit down and write the code? This is not a "which one is better" article. They point at different problems: ADK is an agent orchestration framework (agents, sessions, multi-agent transfer, and since 2.0 a graph-based workflow engine), while Genkit is a GenAI application framework (model calls, flows, structured output, middleware). But they overlap on the everyday tasks every Go developer does (call a model, get typed output, stream, add tools, intercept requests), and on those tasks the developer experience diverges a lot. So instead of comparing docs, I wrote the same eight programs twice and ran every single one against a live Gemini backend: One prompt → one answerA raw model call with no framework machineryStructured output into a Go structStreamingTool callingIntercepting/wrapping the model callA custom model backendMulti-turn conversation and HTTP serving Everything below compiled and ran on Genkit Go v1.11.0 and ADK Go v2.1.0 (the google.golang.org/adk/v2 module, GA since June 30, 2026) with gemini-3-flash-preview, on 2026-08-05. Where I quote framework internals, they come from those released versions, not from the main branch. Nothing in this article is speculated from documentation. TL;DR TaskGenkit GoADK Go 2.0One prompt → textgenkit.GenerateText(...): 21 lines, 2 error pathsmodel + agent + runner + session + event loop: 49 lines, 8 error pathsStructured outputgenkit.GenerateData[Recipe](...), schema inferred from struct tags, typed resulthand-written genai.Schema tree, raw JSON text back, unmarshal it yourselfStreamingcallback receives deltas, final response is ready-assembledSSE partial events; the final event repeats the full text, you deduplicate by handToolstyped Go func, loop runs inside Generatetyped Go func (genuinely elegant), but agent/runner/session around itWrap the model callWithUse middleware with a real next(), applied per callfixed-signature callbacks (no wrapping) or decorate the model.LLM interface, per agentLocal models (Ollama)official plugin, first-class providerno native support; OpenAI-compatible adapter with BaseURLCustom model backendone function, works instantly with Generatetwo-method interface (the smallest contract in either SDK), but invoking it needs the full stackMulti-turnautomatic via agents + session stores (experimental track); resp.History() for plain generate callsautomatic via the GA session servicesMulti-agentdelegation via the Agents middleware (experimental track)first-class: sub-agents, transfer, workflow graph engineServe over HTTPgenkit.Handler(flow) on plain net/httpfull REST platform API (sessions, SSE, events) + embedded Web UIEcosystem16 Go plugins: multi-vendor models, vector stores, telemetry, MCPGoogle Cloud integrations (Vertex, GCS, BigQuery, Apigee, Cloud Run) + OpenAI-compatible + MCP The pattern that emerged: Genkit's primitive is the model call; ADK's primitive is the agent. When the thing you need is a model call, Genkit hands it to you directly, and ADK makes you rent the whole agent apparatus. When the thing you need is a long-lived, multi-agent system, ADK's apparatus stops being boilerplate and starts being the product. The Mental Model Genkit: The Model Call Is the Unit Genkit Go has exactly one setup object (*genkit.Genkit) and everything else is a package-level function call: Go g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{})) text, err := genkit.GenerateText(ctx, g, ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."), ) That is the complete program (plus imports and an error check). Init doesn't even return an error: configuration problems panic at startup, and from there on there is a single err to handle per generation. My runnable version is 21 non-blank lines with 2 error-handling lines. ADK: The Agent Is the Unit ADK has no "just generate" entry point at the framework level. The unit of execution is an agent, and an agent runs inside a runner, and a runner requires a session service. This is the minimal equivalent program: Go model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{ APIKey: os.Getenv("GOOGLE_API_KEY"), }) // if err != nil ... a, err := llmagent.New(llmagent.Config{ Name: "assistant", Model: model, Instruction: "You are a helpful assistant.", }) // if err != nil ... r, err := runner.NewInMemory("hello-app", a) // in-memory session/artifact/memory services // if err != nil ... msg := genai.NewContentFromText("Why is Go a great language for building AI agents? One short sentence.", genai.RoleUser) var text string for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) { if err != nil { /* ... */ } if event.LLMResponse.Content == nil { continue } if event.IsFinalResponse() { for _, part := range event.LLMResponse.Content.Parts { text += part.Text } } } 49 non-blank lines, 8 error-handling lines for the same output. And to be fair, runner.NewInMemory is already ADK being nice: the explicit form makes you wire runner.Config{AppName, Agent, SessionService, AutoCreateSession} yourself. Count the concepts you must hold for one prompt: a model, an agent with a Name (mandatory), an app name, a user ID, a session ID (both meaningless for a one-shot call, both mandatory positional arguments), a RunConfig, an event iterator, an IsFinalResponse() predicate, and manual concatenation of Content.Parts. There is no .Text() helper on events. Even ADK's own console launcher spends ~45 lines turning events into printable text (cmd/launcher/console/console.go), because every consumer re-implements this loop. None of this is bad design: sessions, users and apps are exactly the right vocabulary for a deployed multi-agent platform. It is simply the wrong altitude when what you wanted was an answer to a prompt, and there is no lower altitude offered (almost; see the escape hatch below). Structured Output: Where the Gap Is Widest This was the experiment with the biggest difference, and the one where I initially got ADK wrong in an instructive way. Genkit You describe the shape as a Go type. The JSON Schema is inferred from the struct (via reflection over json/jsonschema tags), constrained decoding is enabled by default, and you get the struct back, parsed and typed: Go type Recipe struct { Name string `json:"name"` PrepMinutes int `json:"prep_minutes"` Ingredients []string `json:"ingredients"` } recipe, _, err := genkit.GenerateData[Recipe](ctx, g, ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithSystem("You generate recipes."), ai.WithPrompt("A simple tortilla de patatas recipe"), ) fmt.Println(recipe.Name, recipe.PrepMinutes, recipe.Ingredients) Output from my run: Plain Text Classic Spanish Tortilla de Patatas (15 min): 5 ingredients 28 lines total. The schema definition is the Go type. ADK ADK's llmagent.Config.OutputSchema takes a *genai.Schema: a hand-built tree, field by field, with a hand-maintained Required list. There is no "infer from struct" path for agent output (curiously, ADK does have schema inference, but only for tool inputs): Go recipeSchema := &genai.Schema{ Type: genai.TypeObject, Properties: map[string]*genai.Schema{ "name": {Type: genai.TypeString}, "prep_minutes": {Type: genai.TypeInteger}, "ingredients": {Type: genai.TypeArray, Items: &genai.Schema{Type: genai.TypeString}, }, Required: []string{"name", "prep_minutes", "ingredients"}, } a, err := llmagent.New(llmagent.Config{ Name: "chef", Model: model, Instruction: "You generate recipes.", OutputSchema: recipeSchema, }) The schema enforcement worked perfectly in my runs: Gemini returned valid JSON every time. The interesting part is what happens next. My first attempt read the event's Output field, which looks exactly like what you want. It was always nil. Digging into the released v2.1.0 source: the LlmAgent does parse the structured output internally, and does attach it to the event, but the runner deliberately strips it before the event reaches you. In runner/runner.go, every non-partial agent reply gets cloned with clone.Output = nil. The parsed value exists to feed ADK 2.0's graph workflow engine (typed routing between nodes), not for you. As a consumer of Runner.Run, you receive raw JSON as text and parse it yourself: Go var jsonText string for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) { // ... error handling, nil checks, IsFinalResponse, Parts loop ... jsonText += part.Text } var recipe Recipe if err := json.Unmarshal([]byte(jsonText), &recipe); err != nil { /* ... */ } (If you set OutputKey, the same raw string is also copied into session state, still not parsed for you.) Two more sharp edges I hit in the source while verifying this: The OutputSchema doc comment in v2.1.0 still warns: "when this is set, agent can only reply and cannot use any tools, such as function tools, RAGs, agent transfer, etc." The implementation has grown workarounds (on some backends it injects a synthetic set_model_response tool; on Vertex AI with Gemini ≥2.0 it can use native constrained output alongside tools), which means combining tools and output schema behaves differently depending on backend and model name, silently.Final tally: 68 lines vs Genkit's 28, and the typed half of "typed output" is entirely yours to write. Streaming: Deltas vs. Partial Events Both frameworks stream. The difference is who assembles the result. Genkit gives you a callback that receives deltas, and the final response arrives already assembled: Go resp, err := genkit.Generate(ctx, g, ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithPrompt("Count from one to ten in words, one per line."), ai.WithStreaming(func(ctx context.Context, chunk *ai.ModelResponseChunk) error { fmt.Print(chunk.Text()) return nil }), ) My run printed streamed chars: 48, final response chars: 48. Chunks and final response account for the same text exactly once each. ADK streams by setting agent.RunConfig{StreamingMode: agent.StreamingModeSSE} and then flags chunk events with LLMResponse.Partial. The gotcha: after all the partial events, ADK emits a final non-partial event that repeats the entire text. My run: partial (streamed) chars: 48, final event chars: 48. Print events naively and you output everything twice. Every ADK consumer must carry this filter: Go for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{ StreamingMode: agent.StreamingModeSSE, }) { // ... if event.LLMResponse.Partial { fmt.Print(part.Text) // the streamed deltas } // the final event repeats the full text: skip it, or you print it twice } This isn't hypothetical: ADK's own example code and console launcher both contain hand-rolled deduplication (if text != prevText in cmd/launcher/console/console.go). It's a small thing, but it's a small thing every single ADK streaming consumer writes, and no Genkit consumer does. As a bonus, Genkit's GenerateDataStream[T] even streams partially-parsed typed structs (your Recipe filling in field by field), which has no ADK equivalent. Tool Calling: Closer Than I Expected Here I expected a bigger gap and didn't find one: ADK's function tools are genuinely well designed. Both frameworks let a tool be an ordinary typed Go function, with the JSON Schema inferred from the input struct: Go // Genkit weather := genkit.DefineTool(g, "getWeather", "Gets the current weather for a city", func(ctx *ai.ToolContext, in WeatherInput) (string, error) { return fmt.Sprintf("22C and sunny in %s", in.City), nil }) resp, err := genkit.Generate(ctx, g, ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithPrompt("What is the weather in Valencia right now?"), ai.WithTools(weather), ) Go // ADK: also nice! Generics, schema inferred, type params never spelled out. weatherTool, err := functiontool.New(functiontool.Config{ Name: "get_weather", Description: "Gets the current weather for a city", }, func(_ agent.Context, in WeatherInput) (WeatherOutput, error) { return WeatherOutput{Report: fmt.Sprintf("22C and sunny in %s", in.City)}, nil }) Both ran the tool and produced the right answer. The differences are around the edges: Ceremony: the Genkit program is 29 lines; the ADK one is 66, because the tool still needs the agent/runner/session stack around it.Where the loop lives: Genkit's tool loop runs inside Generate (with parallel tool execution and a WithMaxTurns bound), so tools work in a plain one-shot call. In ADK the loop belongs to the agent flow, which is consistent with its philosophy, but it means there is no way to use ADK tools without an agent.A public-API gotcha: ADK's public tool.Tool interface is only Name()/Description()/IsLongRunning(). The executable contract (Declaration(), Run()) lives in an internal/ package you cannot reference, so you can't hand-implement a runnable tool against a public interface; you go through functiontool.New. Genkit's equivalent (ai.ToolFunc + DefineTool) is fully public.Composability limit: ADK documents (in its own examples/tools/multipletools) that you cannot mix built-in Gemini tools like geminitool.GoogleSearch{} with custom function tools on one agent; the workaround is wrapping each in a sub-agent.Genkit also has tool interrupts/resume as a stable API (pause generation from inside a tool, get an Interrupts() list, resume with RespondWith/RestartWith). ADK's equivalents are tool confirmation flows wired through its HITL machinery. Getting Low-Level You can go deeper in Genkit than in ADK, and its middleware system is the reason: Genkit exposes the model call, the tool executions, and the whole generation loop as wrappable layers, while ADK exposes fixed callback points and one wrappable interface. Here is what that means in code. Wrapping the Model Call Genkit middleware is a real onion: you get the request and a next() you control. Timing, retry, caching, and request rewriting are all one closure, applied per call: Go timing := ai.MiddlewareFunc(func(ctx context.Context) (*ai.Hooks, error) { return &ai.Hooks{ WrapModel: func(ctx context.Context, params *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) { start := time.Now() resp, err := next(ctx, params) log.Printf("model call: %d messages in, took %s", len(params.Request.Messages), time.Since(start)) return resp, err }, }, nil }) resp, err := genkit.Generate(ctx, g, /* ... */, ai.WithUse(timing)) Ran, printed model call: 1 messages in, took 3.323s. There are WrapGenerate and WrapTool hooks at the other two altitudes (the whole tool loop, and each tool execution), middleware can inject tools, and the built-ins (Retry, Fallback, tool approval, filesystem sandbox) compose the same way. ADK has callbacks, not middleware: BeforeModelCallback and AfterModelCallback with fixed signatures. They can short-circuit (return a response from Before and the model is never called, the documented caching hook) and they can replace the response, but they cannot wrap. There is no next(). To time a call, I had to smuggle state between two separate functions: Go var start time.Time before := func(ctx agent.Context, req *model.LLMRequest) (*model.LLMResponse, error) { start = time.Now() return nil, nil // nil means "continue to the model" } after := func(ctx agent.Context, resp *model.LLMResponse, respErr error) (*model.LLMResponse, error) { log.Printf("took %s", time.Since(start)) return nil, nil } This works (ran, took 1.631s) but it's a workaround; it's attached per-agent rather than per-call, and anything genuinely wrap-shaped (hedged requests, per-attempt timeouts, fallbacks across models) has to move to a different mechanism entirely: decorating the model. Which brings me to the thing ADK gets really right: Custom Model Backends: Local Gemma on Ollama Nobody plugs in an echo model, so no toy examples here. The realistic scenario is a backend the framework didn't bless for you: a local model server, an internal gateway, a provider without an official plugin. I tested two levels of this: running Google's Gemma locally on Ollama, and writing a backend from scratch. Genkit ships an official Ollama plugin, so the local model is a first-class provider: Go oll := &ollama.Ollama{ServerAddress: "http://localhost:11434"} g := genkit.Init(ctx, genkit.WithPlugins(oll)) gemma := oll.DefineModel(g, ollama.ModelDefinition{Name: "gemma3n:e4b", Type: "chat"}, nil) resp, err := genkit.Generate(ctx, g, ai.WithModel(gemma), ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."), ) 23 lines, ran on the first try against gemma3n:e4b on my Mac, and the local model gets the same Generate, flows, middleware, and structured output as any cloud model. The plugin covers embedders too. ADK has no Ollama integration of its own. The path that works is the OpenAI-compatible adapter pointed at Ollama's OpenAI endpoint: Go gemma, err := openaimodel.NewModel(ctx, "gemma3n:e4b", &openaimodel.ClientConfig{ BaseURL: "http://localhost:11434/v1", APIKey: "ollama", // any non-empty value; Ollama ignores it }) This also ran (47 lines with the usual agent, runner, and session around it), and it is a documented path: the package README lists Ollama, LM Studio, and vLLM. The difference is the shape of the support. In Genkit, the local model is a provider with its own plugin; in ADK, it rides on OpenAI compatibility, so anything Ollama offers outside that API surface is out of reach, and then you are writing the backend yourself, which is the next scenario. And When There Is No Plugin at All When the backend has no plugin and no OpenAI-compatible endpoint (an internal LLM gateway, a niche provider), you write the integration yourself. I did, in both frameworks: the same minimal OpenAI chat/completions client, roughly 45 lines of identical plain-net/http glue, run against the real OpenAI API. ADK's model.LLM interface is the smallest custom-backend contract I've seen in any framework: Go type LLM interface { Name() string GenerateContent(ctx context.Context, req *LLMRequest, stream bool) iter.Seq2[*LLMResponse, error] } Two methods, and this interface is also ADK's genuine wrapping seam: ADK's own examples ship a resilientModel decorator doing retries and per-attempt timeouts this way. My OpenAI-backed model.LLM worked, with two costs on top of the shared HTTP glue. First, the vocabulary: requests and responses are google.golang.org/genai types, so you translate both ways, and some of it is non-obvious. The agent's Instruction arrives in Config.SystemInstruction, not in Contents; miss that and your backend silently drops the system prompt. (ADK's own OpenAI adapter is four files of translation code, which tells you how deep that rabbit hole goes. Also, parts of ADK sniff behavior from llm.Name(), so a renaming wrapper can silently change how output schemas are handled.) Second, the invocation: those two beautiful methods still need the agent, runner, and session stack around them to run. Total: 116 lines. The Genkit version of the same backend is the HTTP glue, a capability declaration, and nothing else: Go func callOpenAI(ctx context.Context, req *ai.ModelRequest, cb func(context.Context, *ai.ModelResponseChunk) error) (*ai.ModelResponse, error) { body := oaRequest{Model: "gpt-5-mini"} for _, m := range req.Messages { role := string(m.Role) // "system" | "user" | "model" if role == "model" { role = "assistant" } body.Messages = append(body.Messages, oaMessage{Role: role, Content: m.Text()}) } // ... plain net/http POST to api.openai.com/v1/chat/completions ... return &ai.ModelResponse{ FinishReason: ai.FinishReasonStop, Message: ai.NewModelTextMessage(out.Choices[0].Message.Content), }, nil } gpt := genkit.DefineModel(g, "myco/gpt-5-mini", &ai.ModelOptions{ Supports: &ai.ModelSupports{Multiturn: true, SystemRole: true}, }, callOpenAI) resp, err := genkit.Generate(ctx, g, ai.WithModel(gpt), ai.WithSystem("You are a helpful assistant."), ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."), ) Total: 79 lines, and Generate calls it directly; the model immediately works with structured output, streaming, middleware, and flows. The messages arrive as provider-neutral structs (ai.Message, with system prompt as a regular system-role message), so the translation is one role-mapping loop instead of a genai round-trip. One honest note: my first attempt failed at runtime because I passed nil options, since a Genkit model must declare what it Supports. That cost me one extra struct literal, and it's the flip side of a real feature: the framework validates every request against the declared capabilities and even simulates a system prompt for models that don't support one. The Raw Call Escape Hatch I didn't want to write "you can't go low-level in ADK" without trying it myself, so I tried. model.LLM is public, and this compiles and runs (33 lines in my experiment): Go llm, _ := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{APIKey: key}) for resp, err := range llm.GenerateContent(ctx, &model.LLMRequest{ Contents: []*genai.Content{genai.NewContentFromText(prompt, genai.RoleUser)}, }, false) { // resp.Content.Parts ... } But notice what you're holding: this is the bare genai transport with an ADK-shaped envelope. No instruction templating, no tool loop, no callbacks, no structured output processing, no telemetry: all of that lives inside the agent flow you just bypassed. There are no helpers (Text(), message builders) at this layer, and no example in the entire ADK repo uses it; only tests do. It's an escape hatch, not a supported altitude. That's the crux of the low-level story: in Genkit, dropping down doesn't cost you the framework. Generate is the low-level call: raw messages in, with middleware, tools, schemas, and tracing still active. In ADK, the abstraction is a cliff: you're either at agent altitude with everything, or at transport altitude with nothing. Where ADK's Ceremony Pays Off I also ran the experiments where I expected ADK's extra structure to earn its keep. It mostly did. Multi-turn conversation. Both frameworks carry the conversation for you; neither makes you thread history by hand. In ADK, I called r.Run twice with the same session ID and the second turn knew my name and city. The session service is the conversation, and swapping InMemoryService for the database or Vertex AI implementation brings persistence with it. In Genkit, the agents API does the same job with a session store. I ran the equivalent two turns on v1.11.0: Go g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}), genkit.WithExperimental(), ) assistant := genkitx.DefineAgent(g, "assistant", aix.InlinePrompt{ ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithSystem("You are a helpful assistant."), }, aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()), ) first, err := assistant.RunText(ctx, "My name is Xavi and I live in Valencia. Say hi in one sentence.") // ... second, err := assistant.RunText(ctx, "Where do I live? Answer with just the city name.", aix.WithSessionID[any](first.SessionID)) Both frameworks answered turn 2: Valencia in my runs, with zero history plumbing on either side. And Genkit's session layer goes further than a message log: session snapshots, detach/resume (the client disconnects, the server keeps working, you resume from a snapshot) and artifacts, with in-memory and file stores shipping today plus a SessionStore interface for your own persistence. For conversation state, the two frameworks are at parity. (If you are outside an agent, composing plain Generate calls, ai.WithMessages(resp.History()...) threads history in one line, but that is a choice, not a limitation.) The one difference I would plan around is the maturity label. ADK's session services are GA and are the default path; Genkit's agents API is on the experimental track behind genkit.WithExperimental() and can change between minor releases, which is also why the rest of this article compares the stable surfaces. If you are comfortable riding the experimental track, Genkit gives you all of it today; if you need the stability guarantee, ADK has it. Multi-agent. ADK's sub-agents get an auto-injected transfer_to_agent tool, there are sequential/parallel/loop workflow agents, and 2.0's headline feature is a full graph workflow engine with typed routing between nodes, retries, and human-in-the-loop pauses. Genkit delegates too. The Agents middleware injects one delegate_to_<name> tool per sub-agent, discovers each sub-agent's description automatically for the orchestrator's prompt, forwards recent conversation history, and can merge sub-agent artifacts back into the orchestrator's session. I ran an orchestrator delegating to a poet sub-agent: Go orchestrator := genkitx.DefineAgent(g, "orchestrator", aix.InlinePrompt{ ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithSystem("You are a coordinator. Delegate writing tasks to the right sub-agent using its delegation tool, then return its result verbatim."), ai.WithUse(&middlewarex.Agents{ Agents: []aix.AgentRef{poet.Ref()}, MaxDelegations: 2, }), }, aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()), ) 43 lines, worked on the first run (I got my haiku back through the orchestrator), and the full researcher-plus-engineer version is in the official basic-agents sample. Same maturity note as sessions: this is the experimental track. And ADK's workflow layer, the sequential/parallel/loop agents and the 2.0 graph engine? Genkit's answer is that the orchestration language is Go itself. Flows compose: a flow calls other flows, sequence is code order, parallel is a goroutine group, a loop is a loop, and every flow stays traced and individually servable. I ran a brief-writing flow that fans out to two sub-flows in parallel and then synthesizes: Go brief := genkit.DefineFlow(g, "briefFlow", func(ctx context.Context, topic string) (string, error) { // Parallel: plain Go concurrency. var facts, structure string eg, egCtx := errgroup.WithContext(ctx) eg.Go(func() error { var err error facts, err = research.Run(egCtx, topic) return err }) eg.Go(func() error { var err error structure, err = outline.Run(egCtx, topic) return err }) if err := eg.Wait(); err != nil { return "", err } // Sequential: the next line of code. return genkit.GenerateText(ctx, g, ai.WithModelName(model), ai.WithPrompt("Write a two-sentence brief about %s using these facts:\n%s\nand this outline:\n%s", topic, facts, structure)) }) 53 lines including both sub-flows, ran on stable APIs, and genkit.Handler(brief) serves it over HTTP like any other flow. So the difference between the two workflow stories is representation, not capability. ADK reifies the pipeline as a data structure (edges, typed routes) executed by an engine that owns retries, pauses, and resumability at the graph level; Genkit keeps the pipeline as ordinary Go and covers those concerns with its own pieces (tool interrupts for human-in-the-loop, the retry middleware, and durable execution through integrations like Temporal). Pick by taste: an inspectable graph object, or code you read top to bottom. The serving platform. Genkit's HTTP story is its philosophy in miniature. A flow becomes a handler on plain net/http: Go mux := http.NewServeMux() mux.HandleFunc("POST /greetingFlow", genkit.Handler(greetingFlow)) server.Start(ctx, "127.0.0.1:9090", mux) Shell $ curl -X POST localhost:9090/greetingFlow -d '{"data":"Xavi"}' {"result":"Hello, Xavi!"} ADK's adkrest.NewServer is also a plain http.Handler (nice), but what it mounts is a platform: app listing, session CRUD, SSE runtime, artifacts, debug/trace endpoints. My curl transcript needed two calls (create the session, then run) and got back the full event envelope with usage metadata, state deltas, and node info. On top of that, ADK's launcher embeds the entire ADK Web UI into your binary via go:embed: go run . web webui gives you a chat UI with no npm and no separate process, which is honestly great. (Genkit's Developer UI is a separate CLI process, though it's also more than a chat UI: it traces every generate call, middleware hop, and tool execution.) These aren't contradictions of the boilerplate findings; they're the same design measured on its own terrain. ADK's five constructors are the price of a system where conversations persist, agents transfer to each other, and a REST platform falls out of a config struct. The Plugin Ecosystems One more thing I checked in the released modules, because it changes the decision for real projects: what each framework integrates with out of the box. Genkit Go v1.11.0 ships sixteen plugins in the module: model providers for Google (Gemini API and Vertex AI), Anthropic, OpenAI plus any OpenAI-compatible endpoint, and Ollama; vector stores for AlloyDB, PostgreSQL/pgvector, Pinecone, Weaviate and a local dev store; Firebase and Google Cloud telemetry; MCP; evaluators; and the middleware pack. And that is the Go module alone, before you count the wider JS ecosystem the framework shares its design with. ADK Go v2.1.0's integrations are a coherent Google Cloud story: Gemini and Vertex models, the OpenAI-compatible adapter, Apigee; sessions in memory, in a SQL database or on Vertex AI; GCS artifacts; Vertex AI memory; BigQuery agent analytics; MCP toolsets; and deploy tooling for Cloud Run and Agent Engine. What it does not have is a provider ecosystem beyond that: no Anthropic plugin, no vector store integrations, nothing aimed at other clouds. If your stack is pure Google Cloud, ADK's set covers it end to end, deploy included. The moment you need Anthropic next to Gemini, a Postgres vector store, or observability outside Google Cloud, Genkit has the plugin, and ADK expects you to build it. The Scorecard Numbers from my runnable programs (non-blank lines including imports; error-handling lines in parentheses): ExperimentGenkit GoADK Go 2.0Hello world21 (2)49 (8)Raw model callsame as hello33 (4), undocumented pathStructured output → typed struct28 (2)68 (9)Streaming27 (2)54 (8)One tool29 (2)66 (10)Time the model call34 (2)61 (8)Local Gemma via Ollama, invoked23 (2)47 (8)Custom model backend (real OpenAI), invoked79 (5)116 (13)Multi-turn (2 turns)37 (4) with agents + sessions; 30 (4) with plain Generate51 (8) Line counts are a blunt instrument; the interesting part is what the extra lines are. In every ADK program, they are the same lines: construct model, construct agent, construct runner, invent user/session IDs, iterate events, filter partials, concatenate parts. It's a fixed tax, roughly 25-30 lines, charged on every program regardless of whether sessions and events are relevant to it. A Concrete Decision Guide I would reach for Genkit Go when: The unit of my product is a model call or a flow: an API endpoint, a pipeline step, a CLI, a backend feature.I want typed structured output without maintaining schema trees by hand.I want to wrap, retry, fall back, cache, or rewrite model calls with composable per-call middleware.I'm integrating a custom or non-Google model backend and want the framework features to keep working on top of it.I value the Dev UI's per-call tracing during development.I want sessions and agents too, and I'm fine with the experimental track while that API stabilizes.My stack mixes vendors: Anthropic or OpenAI next to Gemini, Ollama locally, or Postgres/Pinecone/Weaviate for retrieval. The plugin shelf already covers them. I would reach for ADK Go 2.0 when: The unit of my product is a persistent, session-based agent: a chat product, a support agent, anything where conversation state is the point, and I want it on GA APIs today.I need multi-agent coordination: sub-agent transfer, workflow agents, or 2.0's graph engine with typed routing and HITL pauses.I want the serving platform for free: session REST API, SSE, the embedded Web UI, A2A, Cloud Run / Agent Engine deploy tooling.My stack is Gemini/Vertex-first and the genai-typed vocabulary is a feature, not a translation burden. And these compose better than you'd think: nothing stops you from building your model-centric services with Genkit flows and standing up an ADK agent where you genuinely need the agent platform. They're both idiomatic modern Go (both lean on iter.Seq2 and generics nicely), and they're solving different layers of the same problem. Conclusion After running everything, the facts are these: Genkit lets you get lower-level, and ADK charges more boilerplate. In detail: The boilerplate is real and remarkably consistent: ADK charges a fixed ~25-30 line agent tax on every program, because there is no supported altitude between "full agent" and "bare transport". Genkit's Generate occupies exactly that missing middle: raw request access with the framework still on.The low-level gap is sharpest in structured output (typed structs vs hand-built schema trees plus DIY parsing, and a runner that deliberately strips the parsed value it already has) and in middleware (a real next() vs fixed-signature callbacks).But ADK's ceremony is not waste: it buys GA session persistence, multi-agent transfer, a graph engine and an embedded serving platform, and some of its individual designs (the two-method model.LLM, generic function tools) are excellent Go. Pick the altitude that matches your product, and don't confuse "more setup" with "more capable", or "less setup" with "less serious." They're different tools that happen to share a language and a model provider. Further reading: Genkit Go Get Started guideADK 2.0 release notesGenkit GitHub repositoryADK Go GitHub repositoryGenkit Go basic-agents sample (orchestrator, delegation, artifacts)Genkit Go flows documentationStop Using Python for Gen AI: Genkit GoVercel AI SDK Middleware vs. Genkit Middleware

By Xavier Portilla Edo CORE

Teams considering or committed to adopting Kotlin often plan a migration as a series of rewrites, swapping Java services for Kotlin counterparts one microservice at a time. In practice, that kind of plan tends to stall before it produces any shipped code. Kotlin's role in the JVM ecosystem is expanding because it doesn't require that kind of commitment. Companies such as Netflix, Adobe, and DoorDash have adopted it incrementally, alongside existing JVM systems, rather than through a full rewrite. The lowest-risk starting point is tests. They give you Kotlin's language benefits, including null-safety, data classes, and expressive lambdas, without touching anything that runs in production. This guide walks through adding Kotlin to an existing Java + Spring Boot project for test sources only. You'll configure Gradle, write JUnit 5 tests in Kotlin, use MockK for Kotlin-native mocking, and apply Kotest assertions to improve test readability. Your production services don't change, and your team starts experiencing Kotlin in a bounded, no-risk context. Why Testing Is the Safest Place to Introduce Kotlin Kotlin test sources are isolated from the production application artifact and don't require Kotlin in the production runtime. They can still affect CI and test infrastructure, but leave production services untouched. That separation from production makes tests the right entry point for a language your team hasn't yet committed to. Introducing Kotlin in test sources means that your existing Java services remain completely unchanged, with no refactoring, no migration risk, and no regressions in production paths. It also allows your team to experiment with real Kotlin syntax on tests they'd be writing anyway, rather than on contrived examples. Developer buy-in builds faster when engineers are writing tests they actually want to write. Kotlin tests tend to be shorter, more readable, and easier to scan at code review. Leadership and platform teams are skeptical of language changes for good reasons. The proposal "we want to use Kotlin for tests" carries almost no risk to evaluate, since it doesn't touch production or your deployment infrastructure. It's a clean, bounded experiment, and the pattern has been validated by engineering organizations across the industry. Adding Kotlin to an Existing Java + Spring Boot Project You don't need to migrate your entire build to use Kotlin in tests. If you're running a standard Spring Boot + Gradle project, you add the Kotlin JVM plugin, pull in kotlin("stdlib") and io.mockk:mockk at test scope, and IntelliJ IDEA recognizes src/test/kotlin as a source set automatically. Open the project in IntelliJ IDEA and update your build.gradle.kts, adding the Kotlin JVM plugin in the plugins block and the test-scoped dependencies: Kotlin plugins { java id("org.springframework.boot") version "4.1.0" // example version, check for latest id("io.spring.dependency-management") version "1.1.5" // example version, check for latest // Add the Kotlin JVM plugin alongside your existing Java plugin kotlin("jvm") version "2.4.10" // example version, check kotlinlang.org for latest stable } dependencies { // Your existing production dependencies remain untouched implementation("org.springframework.boot:spring-boot-starter-web") // Existing test dependency testImplementation("org.springframework.boot:spring-boot-starter-test") // Required for coroutines testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.11.0") // Kotlin stdlib (test scope only) testImplementation(kotlin("stdlib")) // MockK: Kotlin-native mocking library (check GitHub for latest stable) testImplementation("io.mockk:mockk:1.14.11") // SpringMockK: @MockkBean and @SpykBean for Spring context tests (check GitHub for latest stable) testImplementation("com.ninja-squad:springmockk:5.0.1") // Kotest assertions, no need for the full test runner (check kotest.io for latest stable) testImplementation("io.kotest:kotest-assertions-core:6.2.3") } tasks.withType<Test> { useJUnitPlatform() } Always check the Kotlin and library documentation for the latest stable releases before pinning versions in a new project. If you’re using Maven, you need to add the kotlin-maven-plugin: XML <properties> <kotlin.version>2.4.10</kotlin.version> </properties> <dependencies> <!-- Existing Spring Boot test dependencies --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <!-- Kotlin standard library, test scope --> <dependency> <groupId>org.jetbrains.kotlin</groupId> <artifactId>kotlin-stdlib</artifactId> <version>${kotlin.version}</version> <scope>test</scope> </dependency> <!-- MockK --> <dependency> <groupId>io.mockk</groupId> <artifactId>mockk</artifactId> <version>1.14.11</version> <scope>test</scope> </dependency> <!-- Kotest assertions --> <dependency> <groupId>io.kotest</groupId> <artifactId>kotest-assertions-core</artifactId> <version>6.2.3</version> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.jetbrains.kotlin</groupId> <artifactId>kotlin-maven-plugin</artifactId> <version>${kotlin.version}</version> <executions> <execution> <id>test-compile</id> <phase>test-compile</phase> <goals> <goal>test-compile</goal> </goals> <configuration> <sourceDirs> <source>src/test/kotlin</source> <source>src/test/java</source> </sourceDirs> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> </plugin> </plugins> </build> After saving, IntelliJ IDEA displays a notification to reload the Gradle project. Use Shift+Cmd+I on macOS or Ctrl+Shift+O on Windows and Linux to trigger the sync. Once complete, IntelliJ IDEA automatically recognizes src/test/kotlin as a source set, allowing you to use Kotlin alongside the existing Java test infrastructure, like base classes, testcontainer setup, test data factories, etc. Writing JUnit 6 Tests in Kotlin Writing JUnit 6 tests in Kotlin removes a lot of the friction of test setup. The language's expressiveness lets you focus on the scenario and expected behavior instead. Consider a Spring service that manages orders. Test code often spends more time constructing test data than testing behavior. In Java, creating several variations of the same object can require repeatedly calling setters: Java Order order = new Order(); order.setId(1L); order.setCustomerId(42L); order.setTotal(new BigDecimal("99.99")); order.setStatus(OrderStatus.PENDING); Order cancelledOrder = new Order(); cancelledOrder.setId(1L); cancelledOrder.setCustomerId(42L); cancelledOrder.setTotal(new BigDecimal("99.99")); cancelledOrder.setStatus(OrderStatus.CANCELLED); The setup becomes even more repetitive as a test suite introduces additional scenarios. A test for a large order, for example, would need to repeat the same boilerplate just to change the total. Kotlin's named and default arguments can already reduce that repetition, but you can go further by creating a small test-data DSL that describes orders in terms of the scenarios your tests care about: Kotlin fun orderScenario(block: OrderScenario.() -> Unit): Order = OrderScenario().apply(block).build() class OrderScenario { var id: Long = 1L var customerId: Long = 42L var total: BigDecimal = BigDecimal("99.99") var status: OrderStatus = OrderStatus.PENDING fun pending() { status = OrderStatus.PENDING } fun confirmed() { status = OrderStatus.CONFIRMED } fun cancelled() { status = OrderStatus.CANCELLED } fun forCustomer(id: Long) { customerId = id } fun withTotal(amount: String) { total = BigDecimal(amount) } fun build() = Order(id, customerId, total, status) } Tests can now describe the data they need as scenarios: Kotlin val pendingOrder = orderScenario { forCustomer(42L) withTotal("99.99") pending() } val cancelledOrder = orderScenario { forCustomer(42L) cancelled() } val largeOrder = orderScenario { withTotal("999.99") pending() } The DSL uses Kotlin's lambdas with receivers to make OrderScenario's properties and functions available directly inside the orderScenario { ... } block. Default values provide sensible test data, while individual tests override only the attributes relevant to their scenario. Most importantly, the production Order class can remain completely unchanged in Java. This approach becomes particularly useful as a test suite grows. Instead of accumulating specialized factory methods such as cancelledOrder(), largePendingOrder(), or confirmedOrderForCustomer(), the DSL gives tests a small vocabulary for composing those scenarios themselves. The test describes *what state it needs* without repeating the mechanics of constructing the Java object. A complete JUnit 6 test can then combine the scenario DSL with MockK and Kotest assertions: Kotlin import io.kotest.matchers.shouldBe import io.mockk.every import io.mockk.mockk import io.mockk.verify import org.junit.jupiter.api.Test internal class OrderServiceTest { private val repository = mockk<OrderRepository>() private val orderService = OrderService(repository) @Test fun `returns order when found by id`() { val order = orderScenario { forCustomer(42L) withTotal("99.99") pending() } every { repository.findById(order.id) } returns order val result = orderService.getOrder(order.id) result shouldBe order verify(exactly = 1) { repository.findById(order.id) } } @Test fun `throws when order is not found`() { every { repository.findById(any()) } returns null shouldThrow<NoSuchElementException> { orderService.getOrder(99L) } } } Backtick-delimited test names (`returns order when found by id`) let you write descriptive test names directly in the function declaration, so the name appears naturally in test runner output. This can also save you from using JUnit's `@Named` annotation when you want a readable test name. When a Java repository method returns a nullable `Order?`, the Kotlin compiler forces the test code to account for the null case before using the result. That can surface mismatches between the test and the Java API contract at compile time rather than as unexpected `NullPointerException`s during test execution. A full guide is available in the docs. MockK for Kotlin-Idiomatic Mocking Mockito remains a viable choice when adding Kotlin tests to a Java application, but MockK provides an API designed around Kotlin's syntax. Its concise every { ... } returns ... and verify { ... } constructs make mocked interactions easier to read, while allowing Kotlin tests to mock existing Java classes without changing the production code. This can be particularly useful when testing Java Spring applications that use Reactor. The production application can remain entirely in Java and continue to return Mono or Flux. Kotlin is only introduced in the test suite. MockK handles the mocked dependencies, while Reactor's StepVerifier verifies the resulting publishers. Consider a Java service that returns an order reactively: Java public Mono<Order> findOrder(Long id) { return repository.findById(id); } The service can be tested from Kotlin without changing its implementation: Kotlin import io.kotest.matchers.shouldBe import io.mockk.every import io.mockk.mockk import io.mockk.verify import org.junit.jupiter.api.Test import reactor.core.publisher.Mono import reactor.test.StepVerifier internal class OrderServiceTest { private val repository = mockk<OrderRepository>() private val orderService = OrderService(repository) @Test fun `returns order when found`() { val order = Order( id = 1L, customerId = 42L, total = BigDecimal("99.99"), status = OrderStatus.CONFIRMED ) every { repository.findById(1L) } returns Mono.just(order) val result = orderService.findOrder(1L) StepVerifier.create(result) .expectNext(order) .verifyComplete() verify(exactly = 1) { repository.findById(1L) } } } The production service and repository can remain in Java. The Kotlin test uses MockK's Kotlin-friendly syntax to define the mock behavior, while StepVerifier checks that the returned Mono emits the expected value and completes successfully. The same pattern applies to Flux-returning APIs. For example, a test can verify multiple emitted values without manually subscribing to the publisher: Kotlin every { repository.findAll() } returns Flux.just(order1, order2) StepVerifier.create(orderService.findAll()) .expectNext(order1, order2) .verifyComplete() MockK doesn't replace Reactor's testing tools. Each library handles a different part of the test. MockK provides concise mocking, while StepVerifier provides assertions over Reactor publishers. Kotlin brings those APIs together in a compact test that's focused on the behavior being verified. In practice, that means you can adopt Kotlin for the test source set, use MockK to mock your existing Java dependencies, and test reactive Java code without introducing Kotlin or coroutines into the production application. Kotest Assertions for Cleaner Test Intent You don't need Kotest's full test runner to benefit from its assertion library. The kotest-assertions-core dependency adds infix matchers (shouldBe, shouldNotBe, shouldThrow, shouldContain) that layer on top of your standard JUnit 5 tests. Compare JUnit's assertion style with Kotest's: Kotlin // JUnit 5 assertions Assertions.assertEquals(OrderStatus.CONFIRMED, result.status) Assertions.assertNotNull(result.id) // Kotest matchers (same assertions, different shape) result.status shouldBe OrderStatus.CONFIRMED result.id shouldNotBe null The Kotest versions read in the same order as a sentence describing what you expect. That ordering makes the intent visible without parsing the argument order of assertEquals. shouldThrow is particularly useful for testing Spring service exception paths: Kotlin val exception = shouldThrow<NoSuchElementException> { orderService.getOrder(99L) } exception.message shouldBe "Order 99 not found" The matcher returns the thrown exception, so you can chain further assertions on its properties without additional try/catch blocks. Spring Boot Integration Tests With @MockkBean For Spring slice tests (@WebMvcTest, @DataJpaTest, or a partial @SpringBootTest), you swap Spring's @MockBean for @MockkBean from the springmockk library. It integrates with Spring's test context the same way @MockBean does, but creates MockK instances: Kotlin package com.example.demo.controller import com.example.demo.model.Order import com.example.demo.model.OrderStatus import com.example.demo.service.OrderService import io.mockk.every import org.junit.jupiter.api.Test import org.springframework.beans.factory.annotation.Autowired import org.springframework.boot.webmvc.test.autoconfigure.WebMvcTest import org.springframework.test.web.servlet.MockMvc import org.springframework.test.web.servlet.get import com.ninjasquad.springmockk.MockkBean import java.math.BigDecimal import java.util.NoSuchElementException @WebMvcTest(OrderController::class) internal class OrderControllerTest { @Autowired private lateinit var mockMvc: MockMvc @MockkBean private lateinit var orderService: OrderService @Test fun `GET order returns 200 with order data`() { val order = Order( 1L, 42L, BigDecimal("99.99"), OrderStatus.CONFIRMED ) every { orderService.getOrder(1L) } returns order mockMvc.get("/orders/1") .andExpect { status { isOk() } jsonPath("$.id") { value(1) } jsonPath("$.customerId") { value(42) } jsonPath("$.total") { value(99.99) } jsonPath("$.status") { value("CONFIRMED") } } } @Test fun `GET missing order returns 500`() { every { orderService.getOrder(99L) } throws NoSuchElementException("Order 99 not found") mockMvc.get("/orders/99") .andExpect { status { is5xxServerError() } } } } The @MockkBean annotation replaces the service in the Spring context for the duration of the test. IntelliJ IDEA resolves the Kotlin DSL for MockMvc (mockMvc.get(...)) with full autocomplete support, removing the friction of looking up chained method names. Getting Started With Kotlin in Tests Introducing Kotlin through test sources, before any production code changes, is a validated incremental strategy. The tooling is mature, the Gradle setup is a handful of lines, and the first Kotlin test can ship the same day you configure the dependency. Over time, writing Kotlin tests can change how your team thinks about the language. Developers who spend a few months writing Kotlin tests start asking why they can't use data classes in production. They miss null safety in Java service code. Gradual adoption happens organically, driven by genuine preference rather than a top-down mandate. JetBrains and the Spring team have an official strategic partnership focused on improving the developer experience for backend engineers. If you want to explore what Kotlin can do for your team beyond the test suite, kotlinlang.org is the right place to start.

Microservices do not eliminate transactions. They force architects to decide where a transaction should end. That decision is harder than it sounds. A business operation that was once handled inside a single database transaction may cross several services after decomposition. An order can involve inventory, payment, pricing, customer limits, audit history, and notification. The architecture may divide these responsibilities cleanly, but the business still sees one operation: place the order. This is where many distributed systems become fragile. Teams draw service boundaries first and discuss consistency later. By then, business rules already cross databases, synchronous calls have formed long dependency chains, and eventual consistency has become a general excuse for states nobody can clearly explain. The better approach is to begin with business invariants. First decide which facts must remain true together. Then place transaction and service boundaries around those facts. Technology comes after that decision. Start With the Invariant, Not the Service An invariant is a business rule that must remain true whenever an operation is considered complete. It is stronger than ordinary validation. For example: A confirmed reservation must own the seat assigned to it.A financial transfer must preserve the accounting relationship between debit and credit entries.An accepted command must not create the same business effect twice.An order must not be marked as confirmed if the required payment authorization failed. These rules define the real consistency boundary. If two pieces of state must change together for an invariant to remain true, the safest design is usually to keep them inside one local transaction. Splitting them across services does not remove the rule. It converts the rule into a distributed coordination problem. That conversion may sometimes be necessary, but it should be a conscious tradeoff. A service boundary is not free merely because it looks clean on an architecture diagram. Three Different Consistency Problems Architects often discuss consistency as though the system has only two choices: strong consistency or eventual consistency. In practice, a distributed workflow usually contains three different boundaries. Figure 1. Three consistency boundaries: local atomicity, durable event publication, and eventual convergence across independently committed services. 1. The Atomic Core The atomic core contains state changes that must succeed or fail together. A local ACID transaction is still the simplest and strongest tool for protecting this boundary. Consider an order service that accepts a command, creates the order, records its initial status, and saves a durable event describing the accepted order. If losing that event would leave downstream systems permanently unaware of the order, the order record and the event must be committed together. This does not require a distributed transaction. Both records can live in the same database and participate in the same local transaction. 2. The Durable Handoff Once the atomic state is committed, the system must communicate the result to other components. This is a durability problem, not just a messaging problem. Publishing directly to a broker inside application logic creates a dangerous gap. The database commit may succeed while the publish fails. Publishing first creates the opposite problem: another service may receive an event for state that never committed. The transactional outbox pattern closes this gap by storing the business change and the outgoing event in the same local transaction. A separate publisher later sends the event to the broker. The handoff is durable because the intent to publish survives process crashes and temporary broker failures. 3. The Convergent Workflow After the event crosses a service boundary, the workflow is no longer atomic. Each service commits its own state at a different time. The architecture must therefore define how the system reaches a valid final state. This is where eventual consistency belongs. It should describe an explicit convergence process, not an undefined period in which anything may happen. The design must answer: Which intermediate states are valid?How long may the workflow remain incomplete?What happens when a step fails permanently?Can a completed action be compensated?Which component owns the workflow state?How can support teams and auditors determine what happened? If these questions have no clear answers, the design is not eventually consistent. It is merely incomplete. A Java Boundary Using a Transactional Outbox The following simplified Spring example keeps the order and its outgoing event inside one transaction: Java @Service public class OrderApplicationService { private final OrderRepository orderRepository; private final OutboxRepository outboxRepository; public OrderApplicationService( OrderRepository orderRepository, OutboxRepository outboxRepository) { this.orderRepository = orderRepository; this.outboxRepository = outboxRepository; } @Transactional public OrderId placeOrder(PlaceOrder command) { Order order = Order.create( command.customerId(), command.items()); orderRepository.save(order); OrderPlaced payload = new OrderPlaced( order.id().value(), order.customerId().value(), order.total(), order.version()); outboxRepository.save( OutboxMessage.pending( UUID.randomUUID(), "OrderPlaced", order.id().value(), payload)); return order.id(); } } The important detail is not the framework annotation. It is the placement of the boundary. The order and the outbox message are committed in the same database transaction. Either both exist, or neither exists. An outbox publisher can read pending messages and publish them asynchronously. If it crashes after publishing but before marking the message as sent, it may publish the same event again. That is acceptable only when consumers can recognize repeated business effects. The outbox solves atomic handoff, but it does not remove the need for idempotent processing. This distinction matters. Each pattern solves a specific failure window. Combining patterns without understanding their boundaries creates complexity without correctness. Model the Workflow as State, Not as a Call Chain A common mistake is to implement a distributed transaction as a chain of synchronous calls: Plain Text Order → Inventory → Payment → Shipping → Notification The chain looks simple when every call succeeds. Under failure, however, it behaves like a distributed transaction without the guarantees of one. Suppose inventory is reserved and the payment provider completes the authorization, but its response times out before reaching the order service. The order service cannot safely assume that payment failed. Retrying may create another authorization, while immediately starting compensation may cancel an order that was already paid. These situations should be modeled as durable business states rather than hidden inside request-handling logic. The normal path moves from PENDING to INVENTORY_RESERVED, then PAYMENT_PENDING, and finally CONFIRMED. A definitive payment rejection moves the workflow to PAYMENT_FAILED, followed by inventory release and cancellation. A timeout requires a different path. The workflow moves to PAYMENT_STATUS_UNKNOWN because the outcome is uncertain. A reconciliation step must query the payment provider or process a later payment event before deciding whether to confirm the order or continue with compensation. Figure 2. A durable order workflow distinguishes definitive payment failure from an unknown payment result that requires reconciliation. Each transition should have a durable cause, a stable business identifier, an expected previous state, and an idempotency rule. The component that owns the workflow must record progress before initiating the next step. With this design, the workflow can resume after a process crash because its progress exists as data rather than only inside a thread or request stack. It also gives architects, developers, and support teams an authoritative explanation of what happened and what the system should do next. This does not mean every workflow requires a large orchestration platform. A state table and a small coordinator may be sufficient. What matters is that intermediate states, failure paths, and recovery decisions are explicit. When Eventual Consistency Is Safe Eventual consistency is a good choice when temporary disagreement is acceptable, and convergence can be demonstrated. Typical examples include: Updating a search index after the source record commitsSending confirmation notificationsRefreshing analytics or reporting projectionsUpdating recommendations or cached viewsPropagating noncritical profile changes These operations can usually lag without violating the core business transaction. The decision becomes more difficult around inventory, credit limits, payment authorization, and regulatory records. These domains often contain rules that cannot tolerate arbitrary intermediate states. They may still require distributed workflows, but the workflow must use reservation, versioning, expiration, reconciliation, or another explicit control. For example, inventory does not need to be permanently deducted at the moment an order starts. It can be reserved for a limited period. The reservation becomes a first-class domain concept with an owner, an expiration time, and a release rule. That model is more honest than pretending an asynchronous deduction behaves like a local transaction. Database per Service Is a Guideline, Not the Starting Point Database ownership is important because shared schemas create coupling. However, “database per service” should not be applied before understanding the domain. If two pieces of data participate in the same invariant and change together frequently, separating them can create more coupling rather than less. The services may become independently deployable on paper while requiring constant runtime coordination in production. Sometimes the correct first architecture is a modular monolith with strong internal boundaries and one transactional database. Services can be extracted later when their business boundaries, scaling needs, and consistency requirements are understood. This is not a rejection of microservices. It is recognition that distribution should solve a real problem. Network boundaries are useful for independent ownership, deployment, scaling, security, and fault isolation. They are costly when placed inside an invariant that demands constant coordination. A Practical Decision Test Before splitting an operation across services, architects should ask the following questions: QuestionArchitectural implicationMust these changes always be visible together?Keep them in one local transaction when possible.Can the business tolerate an incomplete state?If yes, define the allowed state and its maximum duration.Can an action be safely retried?Require a stable business identifier and an idempotency rule.Can a completed action be reversed?Define compensation as a separate business operation.Who owns progress across services?Use an explicit coordinator or clearly documented choreography.How will the system detect stalled work?Persist deadlines and support reconciliation.What evidence explains the final state?Record transitions, causes, versions, and correlation identifiers. These questions should be answered before choosing a broker, workflow engine, or framework. Infrastructure can transport messages and execute retries, but it cannot decide which temporary states the business permits. Common Boundary Mistakes Several design mistakes appear repeatedly in distributed Java systems. Splitting First and Coordinating Later Teams decompose a system around nouns or technical layers, then discover that every important operation crosses several services. The resulting architecture has many services but few independent business capabilities. Calling Everything Eventually Consistent Eventual consistency is used as a label without defining the expected final state, deadline, or recovery path. Temporary inconsistency then becomes permanent drift. Treating Broker Delivery as Business Correctness A broker can redeliver a message, but it cannot know whether the corresponding business effect already occurred. Correctness must be enforced where business state changes. Hiding Workflow State in Application Code Retries, timeouts, and compensations are implemented as nested calls without a durable state model. After a crash, the system cannot determine where to resume. Assuming Compensation Always Works The design contains a backward arrow for every forward step, even when the business action is time-sensitive or irreversible. Architecture Is the Placement of Guarantees The central question in distributed design is not whether a system uses microservices, events, sagas, or an outbox. It is where the system places its guarantees. Use a local transaction for the smallest set of state that must change together. Use a durable handoff when committed state must be communicated. Use an explicit workflow when multiple services must converge over time. Add compensation only where the business can genuinely support it. Most importantly, let business invariants shape the boundaries. A clean service diagram is valuable, but it is not more important than preserving the rules that make the system correct. Microservices make boundaries visible. Good architecture makes those boundaries honest.