Java Resources

DZone's Featured Java Resources

Two Engines, One Query: DuckDB Inside Neo4j

By Akmal Chaudhri

CORE

If you've been following the database world lately, you'll have noticed DuckDB turning up in some unexpected places. One of the more interesting examples is pg_duckdb, a Postgres extension that embeds DuckDB directly into the Postgres process, letting analytical queries run through DuckDB's engine without leaving the database. It's a clever idea: keep your data where it is, but bring a faster analytical engine. That got us wondering: could we do something similar with Neo4j? The answer is not in the same way. Postgres has a mature extension architecture that allows foreign query engines to be embedded at a low level. Neo4j doesn't expose that kind of hook. But Neo4j does let you write custom procedures in Java that run inside the Neo4j JVM and are callable directly from Cypher. Also, DuckDB ships an official JDBC driver that bundles its native binary and loads it via JNI. Put those two things together and you've got something that feels surprisingly close to the pg_duckdb experience: DuckDB running in-process, queries fired from Cypher, results flowing back into a graph. That's what we built. In this article, we'll walk through the whole thing from scratch. The full source code is available on GitHub. Prerequisites Before starting, we'll need the following: JDK 21 – check with java -versionMaven – check with mvn -versionNeo4j Desktop – with a local database instance created and ready to start Note that this approach requires a self-managed Neo4j install. AuraDB doesn't support custom plugins, so Neo4j Desktop is the right choice here. Version Summary For reference, here are all the component versions used in this article: ComponentVersionOpenJDK21.0.12.1Maven3.9.6Neo4j Desktop2.2.1Neo4j2026.07.0Neo4j Procedure API5.26.0DuckDB JDBC1.1.3JUnit Jupiter5.10.2maven-compiler-plugin3.11.0maven-shade-plugin3.5.2maven-surefire-plugin3.5.4 Getting the Code The full source code is on GitHub. To get just this project: Shell cd ~ git clone --filter=blob:none --sparse https://github.com/VeryFatBoy/neo4j.git cd neo4j git sparse-checkout set duckdb-procedure mv duckdb-procedure ../duckdb-procedure cd ../duckdb-procedure/neo4j-duckdb-procedure Or follow along from scratch using the steps below. How It Works The architecture is straightforward. We write two Java classes and package them as a fat jar: DuckDBConnection manages a singleton DuckDB connection that lives for the lifetime of the Neo4j process. It uses double-checked locking to keep things thread-safe and reconnects automatically if the connection is ever closed.DuckDBProcedure is the Neo4j procedure itself. It accepts a SQL string from Cypher, runs it against the shared DuckDB connection and streams the results back as rows, each one a Map<String, Object> that Cypher can destructure with dot notation. DuckDB runs in-memory by default, which means it starts clean every time Neo4j restarts. That's fine for analytics against external files, and we'll discuss file-backed persistence in the article Summary section. Project Setup Start by creating the project directory: Shell mkdir -p neo4j-duckdb-procedure/src/main/java/com/example/duckdb cd neo4j-duckdb-procedure Verify the structure: Shell find . -type d Now create pom.xml in the project root. XML <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>neo4j-duckdb-procedure</artifactId> <version>1.0.0</version> <packaging>jar</packaging> <properties> <java.version>21</java.version> <maven.compiler.source>21</maven.compiler.source> <maven.compiler.target>21</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <neo4j.version>5.26.0</neo4j.version> <duckdb.version>1.1.3</duckdb.version> </properties> <dependencies>  <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>${neo4j.version}</version> <scope>provided</scope> </dependency>  <dependency> <groupId>org.duckdb</groupId> <artifactId>duckdb_jdbc</artifactId> <version>${duckdb.version}</version> </dependency>  <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter</artifactId> <version>5.10.2</version> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.11.0</version> <configuration> <source>21</source> <target>21</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>3.5.4</version> </plugin>  <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.5.2</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <createDependencyReducedPom>false</createDependencyReducedPom> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> The Neo4j dependency is provided scope — it won't be bundled into the jar since Neo4j supplies it at runtime. The DuckDB JDBC driver is included at compile scope so it gets bundled along with its native binary. The JUnit dependency is test scope so it's excluded from the fat jar. The maven-shade-plugin builds the fat jar at package time. Confirm Maven can resolve the dependencies: Shell mvn dependency:resolve The Java Classes Create DuckDBConnection.java at src/main/java/com/example/duckdb/: Java package com.example.duckdb; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; public class DuckDBConnection { private static volatile Connection instance; private DuckDBConnection() {} public static Connection getInstance() throws SQLException { if (instance == null || instance.isClosed()) { synchronized (DuckDBConnection.class) { if (instance == null || instance.isClosed()) { instance = DriverManager.getConnection("jdbc:duckdb:"); } } } return instance; } } The volatile keyword ensures the instance is visible across threads without full synchronization on every read. The jdbc:duckdb: connection string with no path gives us an in-memory DuckDB instance. Now create DuckDBProcedure.java in the same package: Java package com.example.duckdb; import org.neo4j.procedure.Description; import org.neo4j.procedure.Name; import org.neo4j.procedure.Procedure; import java.sql.Connection; import java.sql.ResultSet; import java.sql.ResultSetMetaData; import java.sql.Statement; import java.sql.Types; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.stream.Stream; public class DuckDBProcedure { public static class QueryResult { public Map<String, Object> row; public QueryResult(Map<String, Object> row) { this.row = row; } } @Procedure(name = "duckdb.query") @Description("Run a DuckDB SQL query and return results as rows") public Stream<QueryResult> query(@Name("sql") String sql) throws Exception { List<QueryResult> results = new ArrayList<>(); synchronized (DuckDBConnection.class) { Connection conn = DuckDBConnection.getInstance(); try (Statement stmt = conn.createStatement()) { boolean hasResultSet = stmt.execute(sql); if (hasResultSet) { try (ResultSet rs = stmt.getResultSet()) { ResultSetMetaData meta = rs.getMetaData(); int columnCount = meta.getColumnCount(); while (rs.next()) { Map<String, Object> row = new HashMap<>(); for (int i = 1; i <= columnCount; i++) { String colName = meta.getColumnName(i); int colType = meta.getColumnType(i); String typeName = meta.getColumnTypeName(i); Object value = mapValue(rs, i, colType, typeName); row.put(colName, value); } results.add(new QueryResult(row)); } } } else { Map<String, Object> status = new HashMap<>(); status.put("status", "OK"); status.put("updateCount", (long) stmt.getUpdateCount()); results.add(new QueryResult(status)); } } } return results.stream(); } private Object mapValue(ResultSet rs, int index, int sqlType, String typeName) throws Exception { if (typeName != null && typeName.equalsIgnoreCase("HUGEINT")) { long val = rs.getLong(index); return rs.wasNull() ? null : val; } switch (sqlType) { case Types.TINYINT: case Types.SMALLINT: case Types.INTEGER: { long val = (long) rs.getInt(index); return rs.wasNull() ? null : val; } case Types.BIGINT: { long val = rs.getLong(index); return rs.wasNull() ? null : val; } case Types.FLOAT: case Types.REAL: case Types.DOUBLE: case Types.NUMERIC: case Types.DECIMAL: { double val = rs.getDouble(index); return rs.wasNull() ? null : val; } case Types.BOOLEAN: case Types.BIT: { boolean val = rs.getBoolean(index); return rs.wasNull() ? null : val; } case Types.NULL: return null; default: return rs.getString(index); } } } A few things worth noting about DuckDBProcedure: We use stmt.execute(sql) rather than executeQuery() because DuckDB write operations like COPY don't return a ResultSet. The boolean return value tells us which path to take.The mapValue method handles type mapping explicitly. DuckDB's HUGEINT type – used for aggregated sums — has no standard java.sql.Types mapping, so we check the type name directly. Every numeric read is followed by rs.wasNull() to avoid returning 0 for a null column.We synchronize on DuckDBConnection.class to keep concurrent Cypher calls thread-safe against the shared connection. Writing the Tests We tried neo4j-harness first, which spins up a lightweight embedded Neo4j instance in JUnit — the same approach used in the Neo4j Procedure Template. In our environment, we found that the embedded server took longer than expected to become ready during the test run, which led us to a simpler alternative. The fix is to skip neo4j-harness entirely and test the DuckDB logic directly via JDBC. Since DuckDBProcedure is a thin wrapper around a JDBC connection, we can exercise the same type mapping, null handling, and file reading logic without Neo4j in the loop at all. The tests run in under 4 seconds and are fully self-contained. First, add the test directory structure and a src/test/resources folder to hold the test data files: Shell mkdir -p src/test/java/com/example/duckdb mkdir -p src/test/resources The test data files cover all the types and edge cases we want to verify: Plain Text src/test/resources/ ├── test.csv ├── test_nulls.csv ├── test_floats.csv ├── test_booleans.csv ├── test_timezones.csv └── test.json Maven copies everything in src/test/resources to target/test-classes automatically, so the test class can resolve file paths via the classloader without hardcoding any absolute paths. The test class at src/test/java/com/example/duckdb/DuckDBProcedureTest.java covers 12 cases: Range queriesCSV readingParquet write and readJSON readingAggregation returning numeric values rather than stringsNull handlingFloatsBooleansEmpty result setsBad SQL error surfacingIn-memory table persistence across callsDuckDB-side JOIN between two files Each test mirrors something we verified manually in Neo4j. File paths are resolved portably so the tests work regardless of where the project is cloned: Java private String resourcePath(String filename) { return getClass().getClassLoader().getResource(filename).getPath(); } The Parquet test is the one exception — since DuckDB generates the Parquet file rather than reading a pre-existing one, it writes to a JUnit @TempDir that's cleaned up automatically after the test run. Run the full suite with: Shell mvn test All 12 tests should pass in a few seconds. Running mvn package will also run the tests before building the fat jar, so any regression in the DuckDB logic is caught before the jar is deployed. Building and Installing Build the fat jar: Shell mvn package -DskipTests Any warning about MANIFEST.MF from the shade plugin is harmless. The jar lands at target/neo4j-duckdb-procedure-1.0.0.jar. Copy it to your Neo4j Desktop plugins folder. You can find the path in Neo4j Desktop under the three-dot menu for your database instance, then Open Folder > Plugins. Then copy the jar: Shell cp target/neo4j-duckdb-procedure-1.0.0.jar "/path/to/your/neo4j/plugins/" Open neo4j.conf in the conf folder alongside plugins and add: Plain Text dbms.security.procedures.unrestricted=duckdb.* Restart the database. Confirm the procedure is registered using Query from the left-hand navigation pane: Cypher SHOW PROCEDURES YIELD name WHERE name STARTS WITH 'duckdb' RETURN name You should see duckdb.query. If the procedure doesn't appear after restarting, also add the following in neo4j.conf: Plain Text dbms.security.procedures.allowlist=duckdb.* Some Neo4j configurations require procedures to be explicitly allowlisted before they're loaded at startup. Putting It to Work The test data files used here are also available in src/test/resources in the project repository. Let's start simple. DuckDB has a built-in range function we can use without any external data: Cypher CALL duckdb.query("SELECT range AS num FROM range(1, 6)") YIELD row RETURN row.num AS num Five rows should be returned, confirming that DuckDB is running inside Neo4j. Note that DuckDB's range() function is exclusive of the upper bound, so range(1, 6) returns 1 through 5. Now let's read a CSV file. Create a test file at /tmp/test.csv: Plain Text city,country,population London,UK,9648110 Paris,France,2161000 Berlin,Germany,3645000 Tokyo,Japan,13960000 New York,USA,8336817 Query it from Cypher: Cypher CALL duckdb.query("SELECT city, country, population FROM read_csv_auto('/tmp/test.csv') ORDER BY population DESC") YIELD row RETURN row.city AS city, row.country AS country, row.population AS population DuckDB reads and parses the CSV; Neo4j receives the result. We can also write the CSV out as a Parquet file: Cypher CALL duckdb.query("COPY (SELECT * FROM read_csv_auto('/tmp/test.csv')) TO '/tmp/test.parquet' (FORMAT PARQUET)") YIELD row RETURN row And read it back: Cypher CALL duckdb.query("SELECT city, country, population FROM read_parquet('/tmp/test.parquet') ORDER BY population DESC") YIELD row RETURN row.city AS city, row.country AS country, row.population AS population DuckDB handles JSON just as easily. Create /tmp/test.json: JSON [ {"city": "London", "country": "UK", "population": 9648110}, {"city": "Paris", "country": "France", "population": 2161000}, {"city": "Berlin", "country": "Germany", "population": 3645000}, {"city": "Tokyo", "country": "Japan", "population": 13960000}, {"city": "New York", "country": "USA", "population": 8336817} ] And read it: Cypher CALL duckdb.query("SELECT city, country, population FROM read_json_auto('/tmp/test.json') ORDER BY population DESC") YIELD row RETURN row.city AS city, row.country AS country, row.population AS population We can also join two files entirely inside DuckDB before the result reaches Cypher. Create /tmp/test_timezones.csv: Plain Text city,timezone London,GMT Paris,CET Berlin,CET Tokyo,JST New York,EST Then join it against the population data: Cypher CALL duckdb.query("SELECT p.city, p.population, t.timezone FROM read_csv_auto('/tmp/test.csv') p JOIN read_csv_auto('/tmp/test_timezones.csv') t ON p.city = t.city ORDER BY p.population DESC") YIELD row RETURN row.city AS city, row.population AS population, row.timezone AS timezone The join happens entirely inside DuckDB. Neo4j just receives the final result set. Now for the most interesting demo: combining DuckDB analytics with Neo4j graph data in a single query. First, create some graph nodes: Cypher UNWIND [ {city: "London", continent: "Europe"}, {city: "Paris", continent: "Europe"}, {city: "Berlin", continent: "Europe"}, {city: "Tokyo", continent: "Asia"}, {city: "New York", continent: "North America"} ] AS data MERGE (c:City {name: data.city}) SET c.continent = data.continent Now join DuckDB file data with Neo4j graph data by city name: Cypher CALL duckdb.query("SELECT city, population FROM read_parquet('/tmp/test.parquet')") YIELD row MATCH (c:City {name: row.city}) RETURN c.name AS city, c.continent AS continent, row.population AS population ORDER BY row.population DESC And aggregate by continent — population figures from DuckDB, continent groupings from the graph: Cypher CALL duckdb.query("SELECT city, population FROM read_parquet('/tmp/test.parquet')") YIELD row MATCH (c:City {name: row.city}) RETURN c.continent AS continent, count(c) AS city_count, sum(row.population) AS total_population ORDER BY total_population DESC That's two engines contributing to a single result: DuckDB handles the file analytics, Neo4j handles the graph traversal and aggregation. What We Learned Building this wasn't entirely smooth, and the bumps are worth documenting. XML comments in a Maven POM can't contain double dashes anywhere except at the closing -->. That's a quirk of XML parsing that's easy to miss.DuckDB's HUGEINT type – used for the result of aggregation functions like SUM – has no corresponding entry in the standard java.sql.Types constants. Our initial type mapping switch statement silently fell through to the string default, returning numeric values as strings. The fix is to check ResultSetMetaData.getColumnTypeName() by name before entering the switch.Write operations like COPY can't go through executeQuery() since they don't return a ResultSet. Switching to stmt.execute() and checking the boolean return value handles both read and write paths cleanly.Null numeric values in JDBC return 0 from getInt() and getLong(), not null. You must call rs.wasNull() immediately after each numeric read to distinguish a genuine zero from a missing value. Comparing Approaches It's also worth noting that APOC (Awesome Procedures on Cypher) Extended's apoc.load.jdbc procedure supports DuckDB via its JDBC driver and apoc.jdbc.analytics goes further by letting you export graph data into a temporary DuckDB table for analytical queries. These are complementary approaches worth exploring alongside the custom procedure built here. The following table compares the three ways to use DuckDB with Neo4j, each with a different integration style. ApproachHow It WorksDirectionPersistent StateRequiresCustom procedure (this article)DuckDB runs in-process inside the Neo4j JVM as a singleton connectionFiles and SQL into Neo4jYes, across Cypher callsCustom Java pluginAPOC apoc.load.jdbcDuckDB treated as an external JDBC data source; driver jar dropped into plugins folderFiles and SQL into Neo4jNoAPOC Extended + DuckDB JDBC jarAPOC apoc.jdbc.analyticsGraph data exported into a temporary DuckDB table; analytical SQL runs against itGraph data into DuckDB for analyticsNoAPOC Extended + DuckDB JDBC jar Summary We started with a simple question: could DuckDB be embedded in Neo4j the way pg_duckdb embeds it in Postgres? The answer is not natively, but closer than you might expect. With two Java classes, a fat jar, and a few lines of Neo4j configuration, we built a working duckdb.query procedure that runs DuckDB SQL directly from Cypher. Along the way we handled CSV, Parquet and JSON files, DuckDB-side joins, cross-engine aggregations with Neo4j graph data and a handful of type mapping edge cases that the 12-test suite now catches automatically. There's plenty of room to grow this further. File-backed DuckDB, a proper connection pool, cleaner error handling, and possibly a maintained open-source plugin are all natural next steps. But the foundation is there. What we've built here is a proof of concept — it works, and the tests confirm it, but anyone taking it further should treat robustness, connection pooling, and error handling as open items before putting it near production data. The full source code is available on GitHub. More

Why Kotlin or Java Is the Wrong Question For Enterprises

By Aniket Bhattacharyea

If you've sat through more than one architecture review, you've probably sat through a protracted Kotlin-versus-Java debate that goes nowhere. The team spends weeks comparing Kotlin's concise syntax to Java's maturity, runs a poll on developer preference, and produces a decision matrix nobody fully trusts. Then the meeting ends, and nothing changes, because nobody agreed on what problem they were actually trying to solve. Choosing between infrastructure instruments, such as Kotlin and Java, is an investment decision, and like any investment decision, it needs a thesis. Four business pressures should drive the discussion: time-to-market, operational risk, talent economics, and cross-platform consistency. The Problem With Asking Kotlin or Java When the question of Kotlin or Java arises, syntax and feature comparisons dominate the discussion, because they're concrete and measurable, and a natural place to start. You can count lines of code, benchmark compile times, and run surveys on developer satisfaction, but none of that will tell you if those metrics connect to any business outcome you actually care about. A language can be a good technical fit and still be a poor investment for a particular system. A stable backend handling 50,000 transactions a day with a three-person maintenance team faces a different set of risks than a payments service changing weekly under regulatory pressure, and both look different again from a monorepo running the same business logic across Android, backend, and edge. The JVM language strategy question comes down to where complexity actually lives in the system. Ask where Kotlin meaningfully supports your modernization strategy, and where staying with Java makes better sense right now. Modern Java May Be the Right Answer For a stable, low-change system, modernizing the existing Java stack can address the underlying problem without introducing the costs of another language, toolchain, or learning curve. If the system's main constraints don't align with Kotlin's strengths, staying with Java may offer a better return. If you're considering Kotlin, first identify where its capabilities address a problem worth solving. Start With the Business Problem Drawing on Yuri Geronimus's experience as Senior Staff Engineer and Chief Architect for EMEA Enterprise Payments Platforms at Verifone, the Kotlin Enterprise Playbook frames Kotlin adoption as an enterprise investment decision rather than a language preference. It organizes the business case around four key levers: delivery throughput and time-to-market, defect-cost avoidance, legacy modernization, and talent economics. Time-to-Market If your delivery lead time affects revenue, product launches, or certification cycles, ask whether the team can deliver faster. Kotlin's concise syntax means less boilerplate per feature, and fewer null pointer exceptions reaching production translates to less time spent on hotfixes and incident response. Whether that translates to faster delivery depends on where your bottlenecks sit. If the constraint is product clarity or test coverage rather than boilerplate or type errors, language choice won't fix it. Operational Risk Where are your expensive defects concentrated? Kotlin's type system encodes nullability explicitly, allowing some null-related errors to be caught at compile time rather than in production. For systems where a single production bug triggers a compliance review or a customer-facing outage, that compile-time safety has measurable financial value. For a read-heavy reporting service where reliability is already strong, the business case is thinner. Talent Economics The cost of a JVM talent problem can show up in longer time-to-hire, higher onboarding overhead, attrition, or difficulty moving engineers between teams. Java developers can build on their existing JVM experience when learning Kotlin, making internal mobility a potential part of the business case. The strategic partnership between JetBrains and Spring, announced in May 2025, also means that 27% of Spring developers who've already used Kotlin are a potential internal resource pool you may not have fully mapped. Cross-Platform Consistency If your organization maintains duplicated business logic across an Android app, a backend service, and an edge deployment, that duplication is an operational risk disguised as a technical decision. Kotlin Multiplatform (KMP) can let teams share domain logic, models, and validation across platforms, potentially reducing duplicated implementation and the risk of logic drift. That also reduces the compliance and maintenance burden of finding and fixing the same bug in multiple places. Keep in mind that you should evaluate the case for KMP separately, with the scope of shared code kept deliberately narrow. Kotlin doesn't automatically improve time-to-market any more than a new CI tool does. The question is whether you've identified the constraint it's positioned to address. Measure the Outcome You Actually Care About To make a solid business case for Kotlin, you need a measurable hypothesis. Each business lever maps to an observable metric. Time-to-market connects to delivery lead time and deployment frequency (both DORA metrics your platform team likely already tracks). Operational risk connects to change failure rate, mean time to recovery, and incident cost. Talent economics connects to time-to-hire, time to first commit after onboarding, and attrition rate. Cross-platform consistency connects to duplicated-logic defect rate and the volume of bugs that required fixes in more than one codebase. Those metrics support the business case, but you still have to translate them into financial terms. For an enterprise investment decision, connect the operational improvement to a financial outcome. Faster delivery can reduce the cost of delay, fewer incidents can reduce remediation and customer-impact costs, and shorter onboarding can reduce hiring and training costs. Bear in mind that productivity gains don't translate directly into headcount savings. Faster delivery can instead create capacity for new product work or reduce the cost of delay on a revenue-generating feature. Conflating the two overstates the financial case and creates a credibility problem when someone on the finance team models it out. Also, don't count productivity gains as both lower operating costs and increased revenue in the same proposal. A hypothesis that says "we expect Kotlin's null safety to reduce null-related production incidents by X% over 12 months, based on our current incident log showing Y% of incidents trace to NPEs" is defensible. A hypothesis that says "Kotlin will improve developer productivity" gives you nothing to measure against. Interoperability Makes This a Scoped Decision The Kotlin-or-Java framing misleads because it implies a binary. Kotlin compiles to JVM bytecode and is fully interoperable with Java, so Kotlin and Java files can coexist in the same project without a rewrite. Your existing Spring Boot or Hibernate configurations stay in place. Your Java libraries continue working. A single new service, a shared utilities module, or even the test layer of an existing Java codebase are all valid starting points for introducing Kotlin. IntelliJ IDEA, JetBrains' own IDE, includes a Java-to-Kotlin converter and first-class mixed-project support, so the tooling overhead of running both is lower than teams typically assume. That shifts the question from "should we adopt Kotlin?" to "where should we introduce Kotlin first?" A new, high-change service with a measurable business outcome may be a reasonable candidate. A stable order-management system with a small team and clean Java code can probably wait on Kotlin until a clearer business case emerges. Account for the Iceberg and the J-Curve An honest transition plan accounts for costs that typically don't appear in the optimistic spreadsheet. The J-curve is real. Before a team reaches full Kotlin proficiency, velocity drops. Training takes time, mixed-language code reviews add overhead (reviewers need to understand both languages to catch idiomatic issues), and CI pipelines may need reconfiguration for multi-language builds. Compliance documentation may need updating if your audit trail references language versions or toolchain components. The eventual decommissioning of legacy Java code, if that's the direction, carries its own cost that's easy to leave out of a five-year model. Scope the initial experiment carefully, and build these costs into the transition plan upfront. A proposal that acknowledges the J-curve and plans for it presents a more credible case than one that doesn't. Three Questions for Your Next Architecture Review If your team is heading into an architecture review where Kotlin is on the agenda, these three questions cut through the syntax debate and turn it into a scoped, testable proposal. Where are the expensive defects or operational problems concentrated? Pull the last 12 months of production incidents and categorize the ones with the highest cost (in engineer time, customer impact, or compliance exposure). If a significant share traces to null-related errors or type-safety failures, Kotlin's type system is directly relevant. If the distribution looks different, the investment case needs a different justification. What does your JVM talent pipeline actually look like? Map your current team's Kotlin exposure, your open Java roles, and the average time-to-hire over the past year. If you're competing for senior Java engineers in a tight market and losing candidates to organizations with more modern stacks, that's a talent economics argument worth quantifying. If your team is stable and hiring is closed, the argument doesn't apply. Which service, module, or domain is the lowest-risk place to test whether Kotlin improves the target outcome? Look for something actively changing (so you'll have delivery-cycle data to measure), with a well-defined scope, and without deep compliance dependencies that would complicate a mid-stream language shift. New functionality is often the best candidate. Adding Kotlin tests to an existing Java service can also provide a relatively low-production-risk way to gain experience with the language and toolchain. These questions convert an abstract language debate into a scoped experiment with a falsifiable hypothesis. Run the experiment, measure the result, and expand where it works. Make the Strategic Decision Your CFO and CTO won't ever see a line item for Java versus Kotlin, but they will see the costs of delivery lead time, incidents, onboarding overhead, and duplicated-logic defects. The strategic decision is where to invest to improve those outcomes, domain by domain, with measurement in place. Evaluate Kotlin where you have a number to test against, start with the lowest-risk entry point, and leave stable systems alone when the case is weak. The goal is a better outcome where the investment is justified. If you want to see how other enterprise teams have structured this transition, JetBrains has a practical case for it at kotlinlang.org/lp/enterprise-playbook. How has your team navigated this decision? Share your experience in the comments. More

Testing Java and Spring Applications With Kotlin: Low-Risk to Adoption

By Aniket Bhattacharyea

Java Virtual Threads Under Load: Three Production Limits

By A N M Bazlur Rahman

CORE

Where Should a Transaction End? Designing Consistency Boundaries in Distributed Java Systems

By Krishna Kandi

Complex Event Processing (CEP) With Apache Flink: What It Is and When (Not) to Use It

https://dzone.com/articles/apache-flink-1Most teams working with Apache Flink think about stream processing: filter events, aggregate values, enrich records, join streams. This covers the majority of real-time data problems. But there is a specific class of problems that stream processing handles poorly: detecting meaningful sequences of events across time. Did this specific chain of things happen, in this order, within this time window? Complex event processing (CEP) is the tool designed for exactly this, and it works fundamentally differently from stream processing. CEP is a built-in capability of Apache Flink, available as open source and deployable at scales the old proprietary platforms could never reach. Yet it remains underused and misunderstood. This article is a practical guide to what CEP is, when to use it, how Flink implements it, and where it fits in modern data architectures, including agentic AI, which is covered in depth in the companion article. What Is Complex Event Processing? The simplest way to understand CEP is through a contrast with regular stream processing. Stream processing handles events continuously, one at a time or in windows. It answers questions like: What is the average transaction value over the last five minutes? How many login attempts has this user made in the last hour? These are stateful, real-time computations, but they operate on individual events or aggregates. They do not look for sequences. CEP answers a different class of question: did event A happen, followed by event B within 30 seconds, but not preceded by event C in the last 10 minutes? It detects patterns across multiple events in a defined order within a defined time window. Think of it as a regular expression engine applied to event streams instead of text strings. You define the pattern once. Flink evaluates every incoming event against every active pattern continuously. The moment a complete match is found, a result is emitted. The architectural implication is significant. In a traditional database, queries run against stored data. In CEP, data runs against stored queries. Events that match no active pattern are discarded immediately. This makes CEP extremely efficient at high volumes because irrelevant data never accumulates. One distinction worth making explicit: CEP uses static, predefined patterns. You specify exactly what sequence you are looking for. This is the right approach when the business logic is known and deterministic. The alternative is dynamic anomaly detection, where stream processing with statistical or ML functions builds a baseline from historical data and flags deviations without needing explicit rules. Both approaches solve real problems, and both run on Flink. The choice depends on whether you know the pattern you are hunting for. More on this in the section on when not to use Flink CEP. The Missing Event Problem One of the most valuable and least understood CEP use cases deserves its own section: detecting events that do NOT happen. This surprises most practitioners because the instinct is to think of event processing as reacting to events that arrive. But some of the most important business signals are absences. A machine sensor that should emit a heartbeat every 60 seconds goes silent. A delivery confirmation that should follow a shipment event within four hours never arrives. A payment settlement that should complete a transaction chain is missing. In each case, the business needs to know immediately, not after a human notices the gap. The instinct when facing this problem is to reach for the CEP Pattern API or MATCH_RECOGNIZE. But for missing event detection, a LEFT JOIN in Flink SQL is often the more appropriate and more memory-efficient approach. The query asks: give me all cases where event A arrived but the expected event B never followed within the time window. This is cleaner and more predictable for state management than expressing absence as a negative pattern. Memory management matters more here than most teams expect. Flink maintains state for every active pattern it is evaluating. A pattern with a seven-day lookback window across high-cardinality keys requires far more memory than one with a 60-minute window. Always use the WITHIN clause in pattern definitions to bound the time window explicitly. Without it, state grows unbounded in long-running streaming jobs, which is one of the most common operational problems in production Flink CEP deployments. Vendor Landscape: From Legacy Proprietary Platforms to Apache Flink Before looking at the vendors, one distinction is worth clarifying: the CEP market and the stream processing market were historically separate categories with separate products. The diagram below illustrates why: stream processing outputs on every window; CEP outputs only when a complete pattern match is confirmed. Understanding this separation explains both the fragmentation of the old market and why Flink's ability to handle both in one platform represents such a significant shift. TIBCO: BusinessEvents (CEP) and StreamBase (Stream Processing) A dominant middleware vendor of that era, TIBCO had two separate products. BusinessEvents, launched in 2004, was a pure CEP engine: rule-based inference, stateful event correlation, strong particularly in telco and insurance. StreamBase, acquired in 2013, was a completely separate pure stream processing platform designed for high-throughput financial trading. Software AG: Apama (CEP and Stream Processing in One) Software AG Apama was the notable exception: a single platform combining CEP and event stream processing in one product. This was its competitive advantage against TIBCO, which needed two separate tools to cover the same ground. Apama was strong in capital markets and algorithmic trading and competed directly in the financial services segment. Software AG acquired it in 2011. IBM and Oracle: Niche Adoption IBM WebSphere Business Events and Oracle CEP rounded out the market, though neither achieved the adoption depth that TIBCO or Apama built in their core verticals. All of the above platforms are effectively legacy today. They were expensive, required deep specialist expertise, and could not scale to the event volumes that modern architectures demand. Apache Flink arrived as an open-source framework that absorbed the core capabilities of all of them into a single unified platform running at far greater scale and at a fraction of the cost. Open Source: Apache Flink The foundation of Apache Flink remains open source. Self-managed, fully featured, and the basis on which all commercial distributions are built. The right choice for teams with strong engineering capabilities who want maximum control and flexibility. IBM: Confluent Cloud and Confluent Platform Confluent delivers Flink as a fully managed serverless service on Confluent Cloud as well as a self-managed distribution via Confluent Platform, both with Apache Kafka tightly integrated. CEP via MATCH_RECOGNIZE in Flink SQL is a first-class capability across both. Confluent Platform additionally supports the DataStream API, giving teams on-premises or in private cloud environments access to the full programmatic CEP feature set. Confluent was recently acquired by IBM, a move that positions the combined portfolio as a core pillar of IBM's hybrid cloud strategy: enterprises can run Confluent Cloud as a fully managed service or deploy Confluent Platform on-premises and across private clouds, with IBM's enterprise reach and support behind both. Ververica: VERA Engine Ververica, founded by the original creators of Apache Flink, offers the VERA engine. VERA follows an open-core model: 100% compatible with open-source Apache Flink, with proprietary performance optimizations on top. Key capabilities include updating pattern matching rules in a running job without restart, the Gemini state storage engine, which replaces the standard RocksDB backend with a cloud-native alternative delivering significantly faster snapshots, compute-storage separation, and streaming join optimizations that can reach up to twice the performance of vanilla Apache Flink for demanding workloads. Ververica is available in three deployment models: self-managed on-premises via Ververica Platform, fully managed on Ververica's public cloud, and BYOC where Ververica manages the platform on the customer's own cloud infrastructure. The managed cloud offering provides pay-as-you-go pricing billed hourly per compute unit, or reserved capacity for predictable pricing. It is not fully serverless in the scale-to-zero sense, making it better suited for teams with stable, continuous workloads than for highly variable or intermittent processing patterns. Amazon: Managed Service for Apache Flink Amazon Managed Service for Apache Flink is AWS's fully managed Flink offering and a natural choice for organizations already running on AWS infrastructure. The service handles cluster provisioning, auto-scaling, checkpointing, and multi-AZ availability, and it supports Java, Python, and SQL APIs, including MATCH_RECOGNIZE for CEP use cases. One important caveat on the "fully managed" positioning: like Amazon MSK, the service is not truly serverless. Billing is based on allocated KPUs (compute units) rather than actual consumption, meaning you pay for provisioned capacity regardless of whether your jobs are actively processing data. This is the same marketing gap that caught many MSK users by surprise. Teams deeply committed to the AWS stack who want Flink without managing cluster infrastructure will find it a reasonable option. Those needing a consumption-based, fully managed platform with deeper CEP and SQL capabilities will find Confluent Cloud or Ververica the stronger choices. For a complete map of the data streaming vendor landscape including deployment models and adoption rankings, the Data Streaming Landscape 2026 is the reference. How Flink Implements CEP Flink provides two complementary approaches to CEP. The right choice depends on pattern complexity and the technical profile of the team. One important note on distribution coverage: the Pattern API via the DataStream API is available in self-managed Apache Flink, Confluent Platform, and Ververica, but not in Confluent Cloud, which exposes Flink exclusively through the SQL and Table API. For teams on Confluent Cloud, MATCH_RECOGNIZE is the CEP interface available. The diagram below shows the core concept at a glance before diving into the technical details of each approach. The Pattern API via the DataStream API The programmatic approach, available in Java and Python. You define a Pattern object specifying the event sequence to detect, chaining conditions with operators: .next() for strict contiguity where events must follow immediately, .followedBy() for non-strict contiguity where other events can appear in between, and .within() for time constraints on the entire pattern. The Pattern is applied to a DataStream to create a PatternStream, from which matches are extracted using a PatternProcessFunction. This gives the most complete access to CEP capabilities and the most flexibility for complex logic. The tradeoff is that DataStream jobs require stronger engineering skills and can become difficult to maintain as complexity grows. MATCH_RECOGNIZE via the SQL and Table API The SQL approach is based on the ISO standard from December 2016 and available in Flink since version 1.7. Under the hood, it uses the same FlinkCEP library as the programmatic API. PARTITION BY handles per-entity pattern matching, ORDER BY ensures correct event ordering, PATTERN defines the sequence using regular-expression-style syntax, and DEFINE maps events to pattern variables. Flink adds the non-standard WITHIN clause for time-bounded patterns, which is essential for production memory management. The SQL approach is accessible to a broader range of developers, data engineers, and analysts. Its limitations relative to the full Pattern API are in practice a useful constraint: they prevent patterns from becoming so complex they are impossible to debug or maintain. For the majority of business-level CEP requirements, MATCH_RECOGNIZE is sufficient. The UI/UX Gap: CEP Is Still Largely an Engineering Discipline This is an honest limitation that most Flink CEP articles skip, and it deserves a direct discussion. Pattern Authoring Is an Engineering Task Pattern authoring in Apache Flink is fundamentally an engineering task. Writing DataStream API code in Java or Python requires a developer. Writing MATCH_RECOGNIZE SQL lowers the bar significantly but still assumes familiarity with SQL syntax, event time semantics, and the specific constraints of streaming pattern matching. A fraud operations analyst, a supply chain manager, or a compliance officer cannot walk up to a Flink cluster and define their own detection rules without technical support. This was not always the accepted norm in the CEP world. TIBCO BusinessEvents shipped with a visual rule editor and decision table support that allowed business analysts to author and maintain CEP rules without engineering involvement. Apama had a graphical development studio. These visual interfaces were real strengths of the proprietary platforms, enabling the domain experts who best understood the business patterns to directly encode them without going through an engineering ticket queue. The open-source Flink ecosystem has no mature equivalent today. The tradeoff is real but not entirely negative: engineering-based CEP is more powerful, more flexible, more testable, and more maintainable at scale than visual rule editors, which tend to produce opaque logic that is hard to version-control or audit. But it does gate adoption behind developer availability and creates a bottleneck when business rules need to change quickly. GenAI Is Closing the Gap, Partially The gap is starting to close, partially, through a different route than visual tooling. Current LLMs are capable of generating correct MATCH_RECOGNIZE SQL from a natural language description of a pattern. A fraud analyst describing "flag any user who makes more than three transactions over 500 euros within 10 minutes across different merchants" can get a working SQL pattern from a code-capable LLM today. This lowers the barrier for SQL-expressible patterns and gives non-engineers a practical path to prototyping detection logic before handing it to engineering for review and hardening. The limits are equally real. Generated DataStream API code for complex stateful patterns still requires engineering review before it goes anywhere near production. LLMs can produce syntactically correct Flink code that contains subtle logical errors in time semantics, state handling, or watermark behavior that are not obvious without deep Flink expertise. Generating and deploying without human review is not production-safe for mission-critical CEP logic today. The forward-looking picture is that GenAI will reduce but not eliminate the engineering dependency. Business analysts and domain experts working with well-defined, SQL-expressible patterns will be increasingly served by natural language interfaces, whether embedded in platforms like Confluent Cloud, Ververica, or third-party tooling built on top of Flink. Complex pattern engineering involving custom DataStream operators, iterative conditions, or deeply stateful logic will remain a developer discipline for the foreseeable future. The right platform architecture accounts for both populations: an abstraction layer exposing simple pattern authoring to business users, with full engineering-level access available for the cases that require it. Flink CEP Industry Use Cases CEP is not a niche capability. The following use cases represent industries where sequential pattern detection delivers business value that general stream processing cannot replicate. Financial Services: Fraud Detection Payment fraud is the most mature CEP application in production globally. Modern fraud is almost never a single suspicious event. It is a chain: a card used in Berlin, then two minutes later a contactless payment attempt in London, followed immediately by a high-value online transaction. CEP detects that sequence the moment it completes, calculates whether the travel time between locations is physically possible, and triggers a step-up authentication or account freeze before money moves. More sophisticated patterns chain five or more events: rapid card enumeration, velocity checks across channels, account takeover sequences. The ability to update detection rules in a running job without restart is particularly valuable in this domain. Fraud patterns evolve continuously with attacker behavior, and reducing the window between identifying a new attack pattern and deploying a rule to detect it is a direct business impact measure. Manufacturing and IoT: Predictive Maintenance A machine about to fail does not fail suddenly. It emits a multi-sensor signature in the period before failure: a subtle temperature rise, then increasing vibration frequency, then a pressure deviation. A single-sensor threshold alert would either miss the pattern entirely or produce too many false positives to be actionable. CEP detects the full multi-event signature reliably and triggers a maintenance workflow while there is still time to intervene, turning reactive maintenance into predictive maintenance. Event-driven architectures built on Apache Kafka and Apache Flink are already in production across automotive and industrial manufacturing environments for real-time operational intelligence. CEP-based predictive maintenance is a natural and high-value application layer on top of that existing streaming foundation, applying sequential pattern detection to the sensor streams that are already flowing. Supply Chain and ERP: Process Monitoring Business process events flowing from SAP and other ERP systems, B2B integration platforms, and EDI networks create a continuous stream representing the state of operations across the enterprise. A production order moving through its lifecycle generates events at every stage: order created, inventory checked, production started, quality inspected, goods dispatched. In B2B scenarios, the same logic applies across trading partner boundaries: a purchase order triggers an expected sequence of EDI messages including order acknowledgment, advance ship notice, and invoice. If any step is skipped, delayed beyond an SLA, or occurs out of order, the downstream business impact is real regardless of whether the event originates inside the enterprise or from an external partner. CEP monitors the entire process flow in real time across thousands of concurrent orders and trading partner transactions simultaneously. The moment a single order or EDI exchange deviates from its expected sequence, a structured alert is published to a downstream consumer and the appropriate workflow is triggered. This is categorically more valuable than a batch report that surfaces the deviation 24 hours after it occurred, when the window to intervene has already closed. Telecommunications: Network and Fraud Monitoring Telco was one of the original CEP industries and remains one of the strongest fits. Network intrusion detection, SLA breach monitoring across millions of simultaneous connections, and sequential call detail record analysis for subscription fraud all match the CEP pattern precisely. The combination of high event volumes, strict latency requirements, and the sequential nature of the fraud and failure patterns makes this a domain where CEP consistently delivers value that simpler stream processing cannot replicate. E-Commerce: Customer Journey Detection Sequential behavioral patterns are where MATCH_RECOGNIZE in Flink SQL particularly shines, because the patterns are expressible in clean SQL logic that sits close to the business domain. A customer who held a premium subscription, renewed twice, then downgraded to basic is a churn risk signal. A user who visited the same product page four times within a week without purchasing is a conversion opportunity. These patterns are straightforward to express in MATCH_RECOGNIZE and can be authored and maintained by analysts without requiring deep DataStream API expertise. The output feeds real-time personalization engines, retention workflow triggers, or conversion campaign systems. When NOT to Use Flink CEP CEP is a powerful tool for the right problems. For the wrong problems, it adds operational complexity with no benefit. Three situations where you should reach for something else: The pattern is undefined or too dynamic to specify explicitly. If you do not know what sequence you are looking for, or if the relevant patterns change faster than rules can be written and deployed, ML-based anomaly detection running on Flink's stream processing capabilities is the better approach. Let the model learn what normal behavior looks like from historical data and flag statistical deviations, rather than attempting to enumerate every possible bad pattern upfront. CEP and ML anomaly detection are complementary: CEP for known patterns, ML for unknown ones, both running on the same Flink platform feeding the same downstream systems. The problem does not actually require sequential pattern detection. If the pattern is expressible as a windowed aggregation rather than a strict event sequence, a standard Flink windowed stream processing query is simpler, cheaper, and easier to maintain than a CEP job. More than five failed logins within a 10-minute window, for example, is a count within a time window. It does not require CEP unless the ordering and interleaving of specific event types matters. CEP adds value specifically when sequence and ordering are part of what makes the pattern significant. If they are not, a tumbling or sliding window in Flink SQL handles it cleanly without the overhead of CEP state machines. And if the business action can wait for a scheduled report or nightly batch job, do not build a streaming architecture at all. The lookback window is impractically long relative to event volume. A pattern that requires tracking state across seven days of high-cardinality event data consumes significant Flink cluster memory. If the time window is very long and both event volume and key cardinality are high, the state management cost may outweigh the benefit. Either constrain the window more aggressively using the WITHIN clause, reduce key cardinality through pre-aggregation upstream, or reconsider whether a batch or lambda architecture is more appropriate for that specific pattern. Flink CEP Reference Architecture and Best Practices The reference architecture below shows the full picture: source systems feeding an event ingestion layer, Flink CEP as the pattern detector, and downstream consumers reacting independently to the structured output. The best practices that follow are derived directly from this architecture and from production deployments across industries. Keep patterns focused and always bound them with WITHIN. Every active pattern consumes Flink state proportional to its complexity and time window. Start with the simplest pattern that captures the business intent. Add the WITHIN clause in every pattern definition without exception. State that cannot be pruned will cause operational problems in production, often in ways that are difficult to diagnose under load. Decouple detection from response via a downstream consumer. The output of a CEP job should be a structured event published to a Kafka topic, a message broker, or whatever downstream system the architecture uses, not a direct action or API call embedded in the job itself. Downstream consumers react independently: alerting systems, workflow engines, AI agents. This makes the CEP job simpler and more stable, allows multiple consumers to react to the same signal independently, and lets response logic evolve without touching detection logic. It is the most important architectural principle for production CEP deployments. Avoid monolithic Flink jobs. A single large job combining pattern detection, enrichment, response logic, and downstream routing is an anti-pattern. A fraud detection job that also calls an external risk scoring API, writes to a database, and sends notifications is doing too many things. When any one component needs to change or fails, the entire job is affected. Separate concerns into focused jobs: CEP detection in one job, enrichment in another, AI inference in a third. Each job does one thing well and can be updated, scaled, and debugged independently. CEP Is Underused, Practical, and More Relevant Than Ever The decision framework is simple. CEP is the right tool when you know the pattern and need to detect it in real time. Windowed stream processing is simpler when ordering does not matter. ML-based anomaly detection is the right tool when the pattern is too dynamic to define explicitly. All three approaches are complementary and run on the same Flink platform. The legacy CEP market is gone. Apache Flink, delivered through Confluent (IBM), Ververica, and Amazon, has absorbed everything those platforms offered at scales and economics they could never reach. The remaining gap is the UI/UX barrier, which GenAI is beginning to close for SQL-expressible patterns but has not eliminated for complex logic. Detecting the pattern is only half the problem. The other half is what happens next: which system acts, with what context, and with how much autonomy. A follow-up article covers exactly that: how Flink CEP connects to Agentic AI architectures and why CEP is the layer that makes AI agents trustworthy rather than merely powerful. Stay tuned.

By Kai Wähner

CORE

"Works on My Machine" Is an Architecture Problem

The familiar claim that software behaves correctly on a developer workstation but fails elsewhere is often treated as a debugging cliché. In practice, it describes an architectural failure where part of the system’s behavior exists outside version control. Source code may be identical while the effective execution environment differs in runtime version, CPU architecture, filesystem semantics, locale, time zone, native libraries, certificates, environment variables, dependency resolution, or build tooling. When those inputs remain implicit, the developer workstation becomes an undocumented dependency. The resulting failure is not merely environmental inconvenience, but it is evidence that the system lacks a complete, reproducible definition of the conditions under which it is expected to build and run. Reproducible-build practices are based on the same principle: a declared source and build inputs should create an independently verifiable path to the resulting artifact rather than depending on hidden machine state. The Machine Is an Input, Even When the Repository Pretends Otherwise A repository normally captures application source, configuration templates, and dependency manifests, but operating assumptions often remain outside it. A service compiled with one JDK can behave differently under another. Code that relies on the host time zone inherits machine configuration, and Java’s default time zone is derived from the environment in which the JVM runs; APIs that obtain the system default can therefore return different values across hosts. Locale has a similar characteristic because a JVM maintains a default locale that can influence locale-sensitive operations. The safer approach is to make important defaults explicit at application boundaries. A timestamp formatted for interchange, for example, should not silently inherit workstation configuration: Java String timestamp = eventTime .atZone(ZoneOffset.UTC) .format(DateTimeFormatter.ISO_OFFSET_DATE_TIME); Using an explicit UTC offset removes the host time zone from this operation. The same principle applies to number parsing, character encoding, paths, sorting, and regional formatting. Defaults are convenient only while machines happen to share them. Architecture becomes more predictable when environmental choices are represented as declared inputs or explicit invariants. Java’s time API distinguishes instants, offsets, and full time-zone rules, including daylight-saving gaps and overlaps, making this distinction important whenever local time participates in application logic. Filesystem behavior creates a subtler form of skew. Git documents core.ignoreCasebecause filesystems including APFS, HFS+, FAT, and NTFS can require case-insensitive handling. Git probes this behavior when a repository is created or cloned and configures its operation accordingly. Consequently, a reference to Config.json may appear valid in one environment while deployment fails because the repository actually contains config.json. Reproducibility Starts Before Deployment Locking application dependencies is necessary, but it is not sufficient. The package manager, compiler, operating-system libraries, runtime, and build-time tools also participate in artifact production. npm’s npm ci, for example, requires an existing lockfile and exits when package.json and the lockfile disagree rather than modifying the lockfile to accommodate the current installation. That behavior makes dependency resolution more suitable for automated, repeatable builds. Shell node --version npm --version npm ci npm test These commands become significantly more meaningful when the Node.js and npm versions are controlled by CI rather than inherited from whatever happens to be installed. A successful workstation build should not define correctness as a clean build executed from declared inputs should. Hermetic build systems take this further. Bazel defines a hermetic action as one that consumes declared inputs rather than arbitrary files from the host filesystem and produces declared outputs. Sandboxing helps detect accidental dependencies on local tools or files. The architectural benefit extends beyond caching and build speed, as an undeclared dependency can become a build failure instead of remaining hidden until another machine lacks it. That changes an important relationship: Plain Text source + declared dependencies + declared toolchain ↓ artifact The workstation should not be an additional invisible operand in that equation. Containers Reduce Machine Drift, but They Do Not Erase It Containers are useful because they package a controlled userspace, but containerization does not make host differences disappear. CPU architecture remains significant. Docker treats linux/amd64 and linux/arm64 as separate platform targets and provides multi-platform builds for producing images across operating-system and architecture combinations. Native libraries, JNI dependencies, compiled Node modules, and architecture-specific executables can expose such differences immediately. A CI pipeline can turn architecture support into an executable requirement: Shell docker buildx build \ --platform linux/amd64,linux/arm64 \ --tag registry.example.com/payments:${GIT_SHA} \ --push . This is stronger than discovering an architecture mismatch after deployment because the supported platforms become part of the build contract. Base images introduce another source of environmental movement. Container tags are references that can point to newer image content over time. Docker recommends digest pinning when exact image identity matters, as referencing a digest ensures that the same image version is selected even if the associated tag later changes. Dockerfile ARG RUNTIME_IMAGE FROM ${RUNTIME_IMAGE} ENV TZ=UTC COPY service.jar /app/service.jar ENTRYPOINT ["java", "-Duser.timezone=UTC", "-jar", "/app/service.jar"] The pipeline can provide RUNTIME_IMAGE as an approved repository plus immutable digest. Base-image updates then become explicit changes that can pass through build, test, review, and deployment rather than appearing because a floating tag moved. Containers therefore narrow the environment boundary, and they do not eliminate the requirement to define it. Development Environments Belong in Source Control Development machines do not need to reproduce every operational detail of production, but the toolchain that determines software behavior should be describable. The Development Container Specification exists to capture development-specific tools, settings, runtime configuration, and metadata in a machine-readable definition such as devcontainer.json. JSON { "name": "payments-service", "image": "ghcr.io/example/java-dev@sha256:<approved-digest>", "containerEnv": { "TZ": "UTC", "LANG": "en_US.UTF-8" } } The important property is not this particular format. The architectural goal is that a clean machine can reconstruct the required toolchain without depending on manually installed compilers, remembered shell configuration, stale global packages, or undocumented IDE state. This distinction separates documentation from reproducibility. A setup guide describes actions that are expected to produce an environment. An executable environment definition participates directly in producing that environment. The latter is easier to validate continuously and considerably harder to misinterpret. Environmental Assumptions Should Be Observable Even controlled environments can drift. Production diagnosis becomes easier when runtime identity is recorded at startup alongside application version and build commit. Java exposes runtime, architecture, time zone, and locale information directly, allowing relevant execution characteristics to be captured without exposing arbitrary environment variables. Java log.info( "runtime java={} arch={} timezone={} locale={} commit={}", Runtime.version(), System.getProperty("os.arch"), ZoneId.systemDefault().getId(), Locale.getDefault().toLanguageTag(), buildCommit ); This fingerprint turns environment comparison into evidence. A defect occurring only on aarch64, under a particular time zone, or under a different runtime version can be correlated with actual execution metadata instead of machine descriptions passed through incident discussions. Critical assumptions can go further and become startup invariants: Java if (!ZoneId.systemDefault().equals(ZoneOffset.UTC)) { throw new IllegalStateException( "Unsupported runtime timezone: " + ZoneId.systemDefault() ); } Failing immediately is often safer than allowing an unsupported environment to produce subtly incorrect state. Testing should follow the same principle. Dimensions intentionally supported by the software should vary in CI, while dimensions that must remain constant should be frozen. Cross-platform builds validate CPU targets. Clean package installation validates dependency locks. Case-sensitive environments expose path errors. Explicit time-zone tests reveal accidental dependence on host defaults. The distinction is architectural, as supported variability is tested and unsupported variability is removed. Architecture Must Define the Machine Boundary The lasting solution to environment-specific failures is not a larger setup document or a longer troubleshooting checklist. It is a tighter definition of system inputs where locked dependencies, controlled toolchains, pinned base images, explicit locale and time-zone behavior, validated CPU architectures, hermetic builds where practical, reproducible development environments, and observable runtime identity. Reproducible Builds describes reproducibility as an independently verifiable path from source to binary, and the same engineering discipline can be extended to the environment in which that binary executes. A reliable software architecture therefore includes more than source code, services, databases, and deployment topology. It also includes the assumptions connecting code to a machine. When those assumptions are declared, versioned, tested, and observable, environment-specific failures become ordinary engineering defects with traceable causes. When they remain implicit, every workstation quietly becomes a different architecture.

By Uthej Mopathi

CORE

Going All In On the JVM 25 Virtual Threads

The project AngularAndSpring has been refactored from Spring Reactor to Spring blocking APIs using JDK 25 and virtual threads. The Old Architecture The reason AngularAndSpring was built using Spring Reactor was primarily the efficient use of operating system threads and memory under high load. Spring Reactor had a limited thread pool (usually the number of available CPU cores) and used OS threads only when the CPU had work to do. It did not let the operating system threads sleep. This used CPU resources much better and required far fewer OS threads, each of which needs around a megabyte of memory. The trade-off was a reactive programming style, which can become harder to read as the complexity of the implemented functionality grows, along with stack traces that are very difficult to read. These hard-to-read stack traces make finding and fixing errors much harder and significantly more time-consuming. How Virtual Threads Change the Trade-Off Before JDK 25, the choice was between efficient hardware use with Spring Reactor and easy-to-read code with easier debugging and readable stack traces. Spring's support for Java virtual threads offers the use of very efficient virtual threads for request processing. With virtual threads, operating system threads are used only when processing is needed; virtual threads are unmounted from the OS thread, freeing the OS thread to run a different virtual thread; and virtual threads have minimal memory requirements. That means a limited number of operating threads is able to handle a very large number of virtual threads. With JDK 25 and the virtual threads update, the choice has changed: JDK 25 with virtual threads uses hardware resources efficiently, and you can keep the code easy to read and debug with readable stack traces. There are limitations for virtual threads when native code is called, but the same limitations apply for Spring Reactor too. The main virtual threads blocker: The missing synchronized support was fixed in JDK 25. That means the main advantages of Spring Reactor are now provided by the JVM with its virtual threads. The New Architecture The basic building blocks of the new architecture can be found in the backend pom.xml. It now uses these starters: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-mongodb</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-restclient</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webmvc</artifactId> </dependency> The virtual threads are switched on with this in the application.properties: Properties files spring.threads.virtual.enabled=true The spring-boot-starter-webmvc is used to provide REST endpoints with blocking APIs. The spring-boot-starter-restclient is used to provide the new REST client for calling the blocking REST endpoints of other systems. The spring-boot-starter-data-mongodb is used to provide repositories and APIs for accessing MongoDB; all of these APIs are blocking. The ‘spring.threads.virtual.enabled=true’ enables the Spring Boot support for virtual threads provided by the JVM. The REST Endpoint The blocking REST endpoints now look like this in BitstampController: Java @RestController @RequestMapping("/bitstamp") public class BitstampController { private final BitstampService bitstampService; public BitstampController(BitstampService bitstampService) { this.bitstampService = bitstampService; } @GetMapping("/{currpair}/orderbook") public String getOrderbook(@PathVariable String currpair) { return this.bitstampService.getOrderbook(currpair); } ... } These endpoints do not return the reactor wrapper types. It just returns the intended return type. The ‘BitstampService’ methods now also return the intended return types without any wrappers. These endpoints just use plain Java. The Restclient The new blocking RestClient is now the Spring RestClient and looks like this in ScheduledTask: Java @Component public class ScheduledTask { private static final Logger LOG = LoggerFactory.getLogger(ScheduledTask.class); private static final String URLBS = "https://www.bitstamp.net/api"; private static final String URLCB = "https://api.coinbase.com/v2"; private static final String URLBF = "https://api.bitfinex.com"; private final BitstampService bitstampService; private final BitfinexService bitfinexService; private final CoinbaseService coinbaseService; private final MyUserService myUserService; private final RestClient.Builder restClientBuilder; public ScheduledTask(BitstampService bitstampService, MyUserService myUserService, BitfinexService bitfinexService, CoinbaseService coinbaseService, RestClient.Builder restClientBuilder) { this.bitstampService = bitstampService; this.bitfinexService = bitfinexService; this.coinbaseService = coinbaseService; this.myUserService = myUserService; this.restClientBuilder = restClientBuilder; } ... @Async("clientTaskExecutor") @Scheduled(fixedRate = 60000, initialDelay = 3000) @SchedulerLock(name = "BitstampQuoteBTC_scheduledTask", lockAtLeastFor = "PT50S", lockAtMostFor = "PT55S") public void insertBitstampQuoteBTC() { String currPair = "btceur"; insertBsQuote(currPair); } private ClientHttpRequestFactory customRequestFactory() { HttpClientSettings settings = HttpClientSettings.defaults() .withConnectTimeout(Duration.ofSeconds(4)) .withReadTimeout(Duration.ofSeconds(5)); return ClientHttpRequestFactoryBuilder.detect().build(settings); } private void insertBsQuote(String currPair) { LocalTime start = LocalTime.now(); try { QuoteBs quote = this.restClientBuilder.requestFactory(this.customRequestFactory()).build().get() .uri(String.format("%s/v2/ticker/%s/", ScheduledTask.URLBS, currPair)) .accept(MediaType.APPLICATION_JSON).retrieve().body(QuoteBs.class); quote.setPair(currPair); quote = this.limitPrecision(quote); this.bitstampService.insertQuote(quote); this.logDuration("Bitstamp", currPair, start); } catch (Exception e) { LOG.warn(String.format("Bitstamp data import failed for: %s", currPair), e); } } ... } The RestClient is created using the RestClientBuilder to allow connect and read timeouts to be configured. The timeouts are created in the customRequestFactory() method and set as a parameter of the RestClientBuilder. The RestClient uses the .uri(...) method to set the URL to call. The .accept(...) method sets the MediaType in the HTTP header that is expected from the REST client. The .retrieve() method executes the REST call, and the .body(...) method specifies the object into which the JSON response should be mapped. For debugging purposes, it can be useful to map the response body to a String first and analyze the response to ensure that it can be mapped correctly to the appropriate DTOs. The MongoDB Client For accessing the database, Spring Data MongoDB is used. The per-minute quotes are in the QuoteBs, QuoteBf, QuoteCb entities. The per-hour quotes are in the QuoteHourBs, QuoteHourBf, QuoteHourCb entities, and the per-day quotes are in the QuoteDayBs, QuoteDayBf, QuoteDayCb entities. The different entities enable Spring Data MongoDB support for each entity with its own repository. This is an example of QuoteBsMongoRepository: Java public interface QuoteBsMongoRepository extends MongoRepository<QuoteBs, ObjectId>, QuoteBsRepository { ... } The Interface ‘MongoRepository<QuoteBs, ObjectId>’ provides standard interfaces and enables the generated CRUD methods in QuoteBsRepository. Java public interface QuoteBsRepository extends QuoteRepository<QuoteBs>, QuoteMongoRepository<QuoteBs> { ... QuoteBs insert(QuoteBs quote); List<QuoteBs> findByPairAndCreatedAtAfterOrderByCreatedAtAsc(String pair, Date date, Limit limit); @Query("{ 'createdAt': { '$gt': ?0, '$lt': ?1 } }") List<QuoteBs> findByCreatedAtGreaterThanAndCreatedAtLessThan(Date from, Date to); } This interface uses the by-method-name-generated CRUD methods that Spring Data MongoDB provides and uses the ‘@Query’ annotation for a finder that can not be generated only by a method name. The entity QuoteBs looks like this: Java @Document public class QuoteBs implements Quote,QuotePairBs { @Id private ObjectId _id; @NotBlank @Indexed @JsonProperty private String pair; @NotNull @Indexed(name = "QuoteBs-createdAt") @JsonProperty private Date createdAt = new Date(); private BigDecimal high; private BigDecimal last; private Date timestamp; private BigDecimal bid; private BigDecimal vwap; private BigDecimal volume; private BigDecimal low; private BigDecimal ask; private BigDecimal open; The ‘@Document’ annotation marks the class as a MongoDB document. The ‘QuotePairBs’ interface is used to be able to access the common properties of minute, hour, and day entities of the same source. The ‘@Indexed’ annotation makes sure an index is created for this property in MongoDB for faster access and ordering. The precision of the Java BigDecimal type is higher than the precision of the MongoDB number type. Because of that, this method in ScheduledTask is used: Java private <T> T limitPrecision(T quote) { var fields = quote.getClass().getDeclaredFields(); for(Field field : fields) { if(field.getType().isAssignableFrom(BigDecimal.class)) { field.setAccessible(true); try { var value = (BigDecimal) field.get(quote); value = value.round(new MathContext(30, RoundingMode.HALF_UP)); field.set(quote, value); } catch (IllegalAccessException e) { throw new RuntimeException(e); } } } return quote; } The precision is limited to 30 decimals for all properties with the type BigDecimal. This is enough for all use cases in this application. Thread Pools If the hour and day quotes are recalculated from a large number of minute quotes, this can take several minutes. To avoid competing with the thread pool handling the user requests, a separate thread pool is created for this task in SchedulingConfig: Java @Bean(name = "clientTaskExecutor") public Executor threadPoolTaskExecutor() { ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); executor.setVirtualThreads(true); executor.setThreadNamePrefix("clientTask-"); return executor; } This creates a separate TaskExecutor that can be used in the ‘@Async(“clientTaskExecutor”)’ annotations. That makes sure that no user requests wait in the queue for a free thread while long-running threads are at work. Conclusion With virtual threads in JVM 25 and Spring’s support for them, many reasons for using reactive programming in Spring have disappeared. The synchronized issue with virtual threads has been fixed in JDK 25, and the remaining causes of thread pinning are not relevant anymore for most use cases. The JVM now provides scalability and efficiency of reactive frameworks out of the box, while the blocking programming model is easier to implement and debug. Spring Data MongoDB support makes database access much easier with generated CRUD methods and the ‘@Query(…)’ support. It is time for projects still using reactive programming to move to an easier, faster programming model for developers.

By Sven Loesekann

Running TypeSafe AI's Jev Inside Neo4j for Confidence-Aware Sentiment Decisions

This is Part 3 of a series on running sentiment analysis inside Neo4j. Part 1, Running Sentiment Analysis Inside Neo4j With a Java Plugin, used the VADER lexicon via a pure Java UDF – rule-based, fully local, high throughput.Part 2, Wasm Inside Neo4j: Building the Example That Didn't Exist, embedded a Wasm runtime inside a Java UDF using wasmtime-java — sandboxed, still local, lower throughput. Part 3 takes a different approach. Instead of bundling a scoring model inside the plugin JAR, we call TypeSafe AI's Jev from inside a Java UDF. Jev is a System One model – it doesn't generate text. It takes a state (the text to evaluate) and a set of typed questions and returns structured decisions with explicit probability and confidence information. No prompt engineering, no JSON schema coercion, no parsing. The result is callable directly from Cypher: Cypher RETURN com.example.jev.sentiment('The movie was great!') AS result; The full source code is available on GitHub. What Jev Gives You That VADER Doesn't VADER gives you a compound score. For "The movie had some good moments but ultimately disappointed," it returns something around -0.3. That's a number, but it doesn't tell you how much to trust it. Jev separates the decision from the uncertainty. For the same sentence, it returns: JSON { "choice": "negative", "confidence": 1.0, "score": 1.55, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } The winning sentiment option has a confidence value of 1.0. But the score — which is the result of the separate confidence question we asked Jev, rated on a scale from 0 to 2 — is 1.55, landing between "Somewhat confident" and "Very confident." The separate confidence score provides an additional signal about the classification and that information can be actionable in a way that a raw sentiment score isn't. For unambiguous text, Jev is equally unambiguous: JSON { "choice": "positive", "confidence": 1.0, "score": 2.0, ... } This structured uncertainty output is the key differentiator of Part 3. If you need to act differently on low-confidence classifications — routing them to human review, flagging them for follow-up, or simply surfacing the uncertainty in a UI — VADER's scores don't provide a model-derived estimate of uncertainty about the classification. Jev returns an explicit confidence assessment alongside the decision. Confidence information should not be confused with accuracy: a model can be highly confident and still be wrong. What We're Building A Neo4j Java UDF that calls the Jev API over HTTPS, passing the input text as the evaluation state and asking two typed questions: a sentiment classification (positive, negative, or neutral) and a confidence score. For non-blank input, the UDF returns a Map<String, Object> with six keys: choice, confidence, score, positive, negative and neutral. For blank input, it returns only choice and confidence as a guard response without calling the API. No Wasm runtime, no bundled lexicon, no native libraries. The plugin has no Wasm runtime, native library, or bundled sentiment lexicon. Its only runtime dependency beyond Neo4j itself is Jackson, which is relocated into a private package namespace to avoid classpath conflicts with Neo4j's own Jackson dependencies. Prerequisites Java 21. OpenJDK 21 or later. Confirm with java -version.Maven 3.9.6. Install manually rather than via Homebrew on Apple Silicon. Confirm with mvn -version.Neo4j Desktop. Download and install Neo4j for Desktop. We're using version 2026.07.0. Update the neo4j.version property in pom.xml to match your installation.A Jev API key. Sign up for access at TypeSafe AI. The key is passed to Neo4j via a JVM property in neo4j.conf — more on this in the deployment section. Getting the Code The full source is on GitHub. Clone the repo and navigate to the neo4j-jev-udf directory: Shell cd ~ git clone --filter=blob:none --sparse https://github.com/VeryFatBoy/neo4j.git cd neo4j git sparse-checkout set jev-udf mv jev-udf ../jev-udf cd ../jev-udf/neo4j-jev-udf The Maven Project The pom.xml has two things worth noting. First, org.neo4j:neo4j is declared as provided scope — Neo4j is already present in the database JVM at runtime. Second, we use maven-shade-plugin to relocate Jackson into a private package namespace. Neo4j ships its own version of Jackson internally, and without relocation, the two can conflict at runtime in ways that produce silent failures. XML <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>jev-neo4j-poc</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <properties> <maven.compiler.source>21</maven.compiler.source> <maven.compiler.target>21</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <neo4j.version>2026.07.0</neo4j.version> </properties> <dependencies> <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>${neo4j.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.17.2</version> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>21</source> <target>21</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.5.1</version> <executions> <execution> <phase>package</phase> <goals><goal>shade</goal></goals> <configuration> <artifactSet> <excludes> <exclude>org.neo4j:*</exclude> </excludes> </artifactSet> <relocations> <relocation> <pattern>com.fasterxml.jackson</pattern> <shadedPattern>com.example.jev.shaded.jackson</shadedPattern> </relocation> </relocations> <shadedArtifactAttached>false</shadedArtifactAttached> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> Update the neo4j.version property to match your own Neo4j Desktop installation. The Java UDF The UDF uses Java's built-in HttpClient — no third-party HTTP library needed. The Jev API key is read first from the environment variable TYPESAFE_API_KEY, then from a JVM system property of the same name as a fallback. The fallback matters because Neo4j Desktop runs as a separate process and doesn't inherit your shell's environment variables; we'll set the JVM property in neo4j.conf in the deployment section. Java package com.example; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.node.ArrayNode; import com.fasterxml.jackson.databind.node.ObjectNode; import org.neo4j.procedure.Description; import org.neo4j.procedure.Name; import org.neo4j.procedure.UserFunction; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.util.HashMap; import java.util.Map; public class JevUDF { private static final String API_URL = "https://api.typesafe.ai/v1/systemone"; private static final ObjectMapper mapper = new ObjectMapper(); private static final HttpClient client = HttpClient.newHttpClient(); private static String getApiKey() { String key = System.getenv("TYPESAFE_API_KEY"); if (key == null || key.isBlank()) { key = System.getProperty("TYPESAFE_API_KEY"); } if (key == null || key.isBlank()) { throw new RuntimeException("TYPESAFE_API_KEY not set as environment variable or JVM property"); } return key; } @UserFunction("com.example.jev.sentiment") @Description("Classifies text sentiment using Jev (TypeSafe AI). Returns choice, probabilities and confidence.") public Map<String, Object> sentiment(@Name("text") String text) throws Exception { if (text == null || text.isBlank()) { Map<String, Object> empty = new HashMap<>(); empty.put("choice", "neutral"); empty.put("confidence", 0.0); return empty; } String apiKey = getApiKey(); ObjectNode body = mapper.createObjectNode(); body.put("model", "jev-latest"); body.put("state", text); ObjectNode questions = mapper.createObjectNode(); ObjectNode sentiment = mapper.createObjectNode(); sentiment.put("type", "choice"); sentiment.put("instructions", "What is the overall sentiment of this text?"); ObjectNode criteria = mapper.createObjectNode(); criteria.put("positive", "Text expresses positive sentiment"); criteria.put("negative", "Text expresses negative sentiment"); criteria.put("neutral", "Text is neutral or factual"); sentiment.set("criteria", criteria); questions.set("sentiment", sentiment); ObjectNode confidence = mapper.createObjectNode(); confidence.put("type", "score"); confidence.put("instructions", "How confident are you in the sentiment classification?"); ArrayNode levels = mapper.createArrayNode(); levels.add("Not confident"); levels.add("Somewhat confident"); levels.add("Very confident"); confidence.set("criteria", levels); questions.set("confidence", confidence); body.set("questions", questions); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(API_URL)) .header("Content-Type", "application/json") .header("Authorization", "Bearer " + apiKey) .POST(HttpRequest.BodyPublishers.ofString(mapper.writeValueAsString(body))) .build(); HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() != 200) { throw new RuntimeException("Jev API error " + response.statusCode() + ": " + response.body()); } JsonNode result = mapper.readTree(response.body()); JsonNode answers = result.get("answers"); JsonNode sentimentAnswer = answers.get("sentiment"); JsonNode confidenceAnswer = answers.get("confidence"); Map<String, Object> output = new HashMap<>(); output.put("choice", sentimentAnswer.get("choice").asText()); output.put("confidence", sentimentAnswer.get("confidence").asDouble()); output.put("score", confidenceAnswer.get("score").asDouble()); output.put("positive", sentimentAnswer.get("probabilities").get("positive").asDouble()); output.put("negative", sentimentAnswer.get("probabilities").get("negative").asDouble()); output.put("neutral", sentimentAnswer.get("probabilities").get("neutral").asDouble()); return output; } } A few things worth noting in the Jev request body. The choice question uses a criteria object — a map of option names to descriptions. The score question uses a criteria array — an ordered list of level descriptions. These are different shapes, and mixing them up produces a 422 error from the API. Both questions are answered in a single API call; Jev evaluates them in parallel against the same state. Deploying to Neo4j Build the JAR: Shell mvn -q clean package Stop Neo4j in Desktop, then copy the JAR to your plugins folder: Shell cp target/jev-neo4j-poc-1.0-SNAPSHOT.jar \ ~/Library/Application\ Support/neo4j-desktop/Application/Data/dbmss/<your-dbms-id>/plugins/ On Linux, the plugins folder is typically under $NEO4J_HOME/plugins/. On Windows, the plugins folder is typically under %NEO4J_HOME%\plugins\. Edit neo4j.conf in the conf/ folder of the same dbms directory and add three lines: Plain Text dbms.security.procedures.unrestricted=com.example.jev.* dbms.security.procedures.allowlist=com.example.jev.* server.jvm.additional=-DTYPESAFE_API_KEY=<your-api-key> Two lines are shown in the security configuration. allowlist controls which extension functions are loaded and makes the function callable from Cypher. unrestricted gives matching extensions full database access, bypassing Neo4j's security restrictions and should be used only where necessary. For this example, both settings are configured for the com.example.jev.* namespace; use the narrowest configuration that your Neo4j version and deployment require. The server.jvm.additional line passes the API key to Neo4j's JVM as a system property. For this local Neo4j Desktop example, that's a practical way to make the key available without modifying the UDF code. The Neo4j Desktop process does not automatically receive environment variables exported in your terminal session, so the JVM property is the reliable alternative. Do not commit this value to source control. For production, use your deployment environment's secret-management mechanism and inject the value into the Neo4j process without storing the secret in a tracked configuration file. Restart Neo4j in Desktop, then select Query from the left-hand navigation pane, connect to your instance, and confirm the function is registered: Cypher SHOW FUNCTIONS YIELD name WHERE name STARTS WITH 'com.example' RETURN name; Expected: "com.example.jev.sentiment" Results Positive Sentence Cypher RETURN com.example.jev.sentiment('The movie was great!') AS result; Expected: JSON { "choice": "positive", "confidence": 1.0, "score": 2.0, "positive": 1.0, "negative": 0.0, "neutral": 0.0 } Negative Sentence Cypher RETURN com.example.jev.sentiment('The movie was absolutely terrible.') AS result; Expected: JSON { "choice": "negative", "confidence": 1.0, "score": 2.0, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } Mixed Sentiment Cypher RETURN com.example.jev.sentiment('The movie had some good moments but ultimately disappointed.') AS result; Expected: JSON { "choice": "negative", "confidence": 1.0, "score": 1.55, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } The winning sentiment option has a confidence value of 1.0. But the score drops to 1.55 out of 2 – somewhere between "Somewhat confident" and "Very confident." This is structured uncertainty output, and it's what distinguishes Jev from a rule-based scorer. There are two different quantities here. confidence belongs to the sentiment choice itself: it describes the probability concentration of the classification. score comes from a separate question that explicitly asks Jev how confident it is in that classification. They answer different questions and should not be interpreted as two versions of the same confidence value. Blank-Input Guard Cypher RETURN com.example.jev.sentiment('') AS result; Expected: JSON { "choice": "neutral", "confidence": 0.0 } The blank-input guard returns immediately without making an API call. Here neutral is a guard value rather than a Jev classification: no sentiment analysis is performed for blank or whitespace-only input. Scalability and Limitations Every call to com.example.jev.sentiment() makes an outbound HTTPS request to the Jev API. That shapes where this approach fits and where it doesn't. Latency. Jev responds in 70-500ms per call. For an occasional interactive request — a customer submitting a review, a user looking up a specific node — that latency may be acceptable. For a batch pipeline, if 1,000 calls are made sequentially at a 200ms average, that's over three minutes of API time; concurrent execution changes the wall-clock time but introduces its own rate-limit and resource considerations. Cost. At the pricing available when this article was written, input tokens cost approximately $0.042 per million and output tokens were free. The test requests in this article averaged approximately 381 input tokens per call, giving a per-call cost of around $0.000016 (381 ÷ 1,000,000 × $0.042). Pricing is subject to change; check TypeSafe AI's current pricing before estimating production cost. At this workload size, cost is unlikely to be the primary constraint; latency, throughput, network dependency, and data residency are more significant considerations. At that rate, 1,000 calls would cost roughly $0.016, while one million calls would cost roughly $16, assuming the same average input size. Network dependency. Parts 1 and 2 of this article series are fully local – no network, no external dependency, no data leaving the database. Part 3 introduces all three. The UDF itself runs inside Neo4j, but the inference does not: the text crosses the network boundary to TypeSafe AI, and only the structured result returns to Neo4j. If the Jev API is unreachable, the UDF throws. In a production deployment, that's an operational risk worth planning for: the HttpClient in this example has no explicit timeout configured; in production, set a connect timeout and a per-request timeout to prevent a slow or unresponsive API from blocking Neo4j worker threads indefinitely. A fallback return value and monitoring on the call path are also worth considering. Note also that the UDF makes a synchronous blocking HTTP call inside database query execution; concurrent Cypher invocations increase pressure on Neo4j worker threads, the API's rate limits, and the network. Data residency. Text passed to com.example.jev.sentiment() leaves Neo4j and is sent to TypeSafe AI's API. If your data residency requirements prohibit that, this approach isn't suitable. Review TypeSafe AI's data handling policies before deploying in a regulated environment. Caching as a mitigation. For workloads where the same text is scored repeatedly, results can be stored as node properties on first call and read directly on subsequent queries — skipping the API entirely. That's a natural pattern in Neo4j and largely eliminates the latency and cost concerns for stable datasets. The right use case. This approach is best suited to low-frequency, interactive workloads where explicit uncertainty information matters more than throughput and where data leaving the database is acceptable. For high-volume batch ingestion, Part 1 (the Java VADER UDF) remains a strong fit — local, fast, no network dependency. Part 3 trades local throughput and offline operation for a richer decision output and explicit uncertainty information. Comparing the Three Approaches Part 1: Java VADERPart 2: Wasm/VADERPart 3: JevOutputPolarity scores (float)Polarity scores (float)Typed decision + explicit uncertainty informationLocationFully localFully localNetwork (TypeSafe AI API)Latency< 1ms50-200ms (init)70-500msCostFreeFree~$0.000016/callOfflineYesYesNoExecution boundaryJava UDFWasm runtimeExternal APIUncertainty outputNoNoYesTypical fitHigh-volume batchSandboxed local inferenceInteractive, uncertainty-aware decisions Summary We added a third option to the series: calling TypeSafe AI's Jev from inside a Neo4j Java UDF, returning a typed sentiment decision with explicit uncertainty information directly from Cypher. The implementation has the smallest inference footprint of the three – no Wasm runtime, no bundled lexicon, no native libraries. The JAR contains only the UDF class, a relocated Jackson, and Java's built-in HttpClient. The main configuration consideration is passing the API key to Neo4j's JVM via server.jvm.additional in neo4j.conf, since the Neo4j Desktop process does not automatically receive environment variables exported in your terminal session. The key contribution of Part 3 isn't the mechanics — it's the explicit uncertainty output. For clear-cut sentences, Jev is unambiguous. For genuinely mixed sentiment, it returns both the decision and a structured confidence measure that Cypher can act on directly. That's something neither VADER nor the Wasm implementation provides. A deeper discussion of putting Jev inside Neo4j can be found in the free online book Lessons Learned Putting Jev Inside Neo4j. The full source code is available on GitHub.

By Akmal Chaudhri

CORE

Building High-Performance Time-Series Applications With Java and QuestDB

QuestDB is well-suited for applications necessitating high-performance ingestion and SQL access to time-oriented data. It is especially useful in financial markets, real-time analytics, observability, telemetry, and operational systems where data arrives continuously and must be queried with low latency. Unlike general-purpose databases, QuestDB is purpose-built with time as a core element of both storage and query processes. QuestDB appeals to enterprise developers by combining a time-series architecture with a familiar SQL interface. This technique reduces the learning curve for teams experienced in relational querying while providing a database optimized for append-heavy, chronological workloads. For systems needing recent-state queries, historical analysis, trend detection, and rapid ingestion, QuestDB delivers a practical solution. Why QuestDB Matters QuestDB excels when applications require both high ingestion rates and fast analytical queries on recent and historical data. This is essential for workloads with continuous data streams where rapid business response is critical, such as market prices, trading activity, telemetry, infrastructure metrics, clickstream data, operational events, and real-time business indicators. In these cases, the database must efficiently manage append-heavy data while supporting intuitive SQL queries across time ranges. As a result, QuestDB is well suited for financial platforms, observability systems, logistics, energy, IoT, and real-time analytics. For example, a trading system can query both the latest prices and historical intervals, an operations platform can analyze recent latency and throughput, and a logistics application can review vehicle or delivery telemetry over time. Its SQL-oriented model is especially valuable for enterprise teams, offering a familiar query language alongside an architecture designed for time-series workloads. Hands-On: Java With QuestDB We will build a simple Java application using QuestDB as the time-series database. For development, the easiest way to begin is with Docker: Shell docker run -d \ --name questdb-instance \ -p 9000:9000 \ questdb/questdb:10.0.1 QuestDB provides its Web Console and HTTP endpoint on port 9000. Once the container is running, you can open http://localhost:9000 in your browser to view the data. Configuring the Application Eclipse JNoSQL uses Jakarta APIs such as CDI and JSON-B, along with Eclipse MicroProfile Config. These APIs are available in popular Java runtimes including Helidon, Quarkus, Payara, and Open Liberty. Add the QuestDB driver dependency: XML <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-questdb</artifactId> <version>${jnosql.version}</version> </dependency> Next, configure the Time Series database and QuestDB connection in microprofile-config.properties: Properties files jnosql.timeseries.database=qdb jnosql.questdb.url=ws::addr=localhost:9000 The qdb value specifies the logical database for the JNoSQL Time Series manager. The QuestDB URL uses the connection format required by the driver. Modeling Sensor Data For this example, we will use a simple sensor reading model: Java @Entity public class SensorReading { @Id private Instant id; @Column private String sensor; @Column private double temperature; @Column private double humidity; // constructors, getters, and setters } The Instant field records the measurement time, while the other fields describe the sensor reading at that moment. You can also expose the entity through Jakarta Data: Java @Repository public interface SensorReadingRepository extends BasicRepository<SensorReading, Instant> { List<SensorReading> findBySensorOrderByIdDesc( String sensor, Limit limit); } Using TimeSeriesTemplate You can now insert a sequence of readings and query both the latest state and recent history: Java public class App { public static void main(String[] args) { var firstReading = new SensorReading( Instant.parse("2026-09-20T08:00:00Z"), "sensor-01", 21.4, 45.0 ); var secondReading = new SensorReading( Instant.parse("2026-09-20T09:00:00Z"), "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { TimeSeriesTemplate template = container.select(TimeSeriesTemplate.class).get(); template.insert(firstReading); template.insert(secondReading); template.insert(latestReading); var currentReading = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .limit(1) .singleResult(); System.out.println( "Current sensor reading: " + currentReading ); var history = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .skip(1) .limit(10) .result(); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } The first query retrieves the latest reading for sensor-01. The second retrieves a limited history, skipping the most recent observation. This approach better fits typical time-series use cases than retrieving records by identifier. Using Jakarta Data You can achieve the same functionality using a Jakarta Data repository: Java public class App2 { public static void main(String[] args) { var firstReading = new SensorReading( Instant.parse("2026-09-20T08:00:00Z"), "sensor-01", 21.4, 45.0 ); var secondReading = new SensorReading( Instant.parse("2026-09-20T09:00:00Z"), "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { SensorReadingRepository repository = container.select(SensorReadingRepository.class).get(); repository.save(firstReading); repository.save(secondReading); repository.save(latestReading); var currentReading = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.of(1) ) .stream() .findFirst(); System.out.println( "Current sensor reading: " + currentReading ); var history = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.range(2, 10) ); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } Notably, QuestDB’s time-series capabilities are available through the same Java programming model as other NoSQL time-series drivers. This lets the application focus on temporal queries such as latest state, ordering, and recent history, while the driver handles database-specific details. Why SQL Matters for QuestDB Adoption QuestDB stands out for its time-series specialization and SQL-oriented approach. When organizations adopt specialized databases, teams often struggle to learn new query languages and mental models. By providing a familiar SQL interface, QuestDB helps reduce these adoption barriers. Teams are already familiar with filtering, ordering, grouping, aggregation, and limiting result sets. Keeping these concepts allows programmers to focus on time-based challenges instead of learning a new query system. This is especially valuable when multiple roles need access to the same data. Financial systems illustrate this well. Market data, exchange rates, trades, and price movements are time-oriented, and many financial professionals already use SQL extensively. QuestDB lets teams leverage existing SQL expertise while gaining from a database optimized for high-frequency ingestion and time-based analysis. This advantage also applies to observability and operational analytics. Teams often need to query metrics for example request latency, throughput, error rates, or customer activity over specific intervals. Using familiar SQL constructs makes it easier to access application data for analysis. QuestDB does not function exactly like a traditional relational database. Its architecture and optimizations target time-series workloads. The result is an approachable query experience combined with a storage engine designed for specialized performance. For enterprises, this combination is powerful: specialized time-series performance without requiring teams to abandon the familiar query model. Conclusion QuestDB is a strong fit for applications that need fast ingestion, recent-state queries, historical analysis, and SQL over continuously changing data. With Eclipse JNoSQL 1.1.18, Java developers can use these capabilities through familiar Jakarta APIs, keeping the application model consistent while QuestDB handles the time-series specialization underneath.

By Otavio Santana

CORE

Building IoT Time-Series Applications With Java and Apache IoTDB

Apache IoTDB is well-suited for environments where time-series data from connected devices and industrial systems is generated continuously and requires efficient querying. Typical use cases include monitoring temperature, pressure, vibration, energy usage, machine status, and device telemetry. This approach also applies to manufacturing, smart infrastructure, fleet monitoring, utilities, and edge computing. IoTDB stands out for its focus on large-scale time-series workloads from devices and industrial systems. It is purpose-built for high-frequency data ingestion, historical analysis, and time-based queries. This makes it ideal for applications that require insight into both the current state and historical trends of devices or processes. Why Apache IoTDB Matters Apache IoTDB excels at managing large volumes of device-generated data over time. It addresses issues beyond storage, such as continuous data ingestion, efficient organization by device and timestamp, and fast queries for both recent and historical values. This makes IoTDB well-suited for systems with temporal, device-oriented data models. Common use cases include industrial IoT, smart factories, energy monitoring, connected vehicles, building automation, predictive maintenance, and edge computing. For example, manufacturers can track machinery data, utilities can analyze energy consumption across many meters, and fleet operators can monitor location and engine telemetry. In each scenario, IoTDB enables applications to determine current status, review device behavior over time, and detect measurements outside expected ranges. Hands-On: Java With Apache IoTDB We will build a simple Java application using Apache IoTDB. For simplicity, IoTDB will run locally in Docker. Shell docker run -d \ --name iotdb-instance \ -p 6667:6667 \ -e dn_rpc_address=0.0.0.0 \ apache/iotdb:2.0.11-standalone Next, create the database using IoTDB’s Table SQL dialect: Shell docker exec -it iotdb-instance \ /iotdb/sbin/start-cli.sh \ -h 127.0.0.1 \ -p 6667 \ -u root \ -pw root \ -sql_dialect table \ -e "CREATE DATABASE IF NOT EXISTS jnosql" For this local example, we use the default root/root credentials and disable client redirection. These settings are suitable for single-node development only. In production, use secure credentials and configure redirection based on your IoTDB cluster topology. Configuring the Application Next, configure the connection in microprofile-config.properties: Properties files jnosql.timeseries.database=jnosql jnosql.iotdb.host=localhost jnosql.iotdb.port=6667 jnosql.iotdb.username=root jnosql.iotdb.password=root jnosql.iotdb.enable.redirection=false Eclipse JNoSQL is built on Jakarta APIs such as CDI, JSON-B, and Eclipse MicroProfile Config. These APIs are supported by popular runtimes including Helidon, Quarkus, Open Liberty, and other Jakarta EE-compatible environments. Then add the Apache IoTDB driver: XML <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-iotdb</artifactId> <version>${jnosql.version}</version> </dependency> Modeling Sensor Data A sensor-oriented model aligns well with IoTDB: Java @Entity public class SensorReading { @Id private Instant id; @Column private String sensor; @Column private double temperature; @Column private double humidity; // constructors, getters, and setters } The Instant field records the timestamp of the reading. The other fields capture the measurements at that time. The entity can also be exposed through Jakarta Data: Java @Repository public interface SensorReadingRepository extends BasicRepository<SensorReading, Instant> { List<SensorReading> findBySensorOrderByIdDesc( String sensor, Limit limit); } Using TimeSeriesTemplate With the infrastructure in place, insert several readings and retrieve both the latest value and recent history: Java "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { TimeSeriesTemplate template = container.select(TimeSeriesTemplate.class).get(); template.insert(firstReading); template.insert(secondReading); template.insert(latestReading); var currentReading = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .limit(1) .singleResult(); System.out.println( "Current sensor reading: " + currentReading ); var history = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .skip(1) .limit(10) .result(); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } The first query answers a common IoT question: what is the latest reading from this sensor? The second retrieves its recent history, excluding the latest observation. Using Jakarta Data The same use case can be implemented using the repository: Java public class App2 { public static void main(String[] args) { var firstReading = new SensorReading( Instant.parse("2026-09-20T08:00:00Z"), "sensor-01", 21.4, 45.0 ); var secondReading = new SensorReading( Instant.parse("2026-09-20T09:00:00Z"), "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { SensorReadingRepository repository = container.select(SensorReadingRepository.class).get(); repository.save(firstReading); repository.save(secondReading); repository.save(latestReading); var currentReading = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.of(1) ) .stream() .findFirst(); System.out.println( "Current sensor reading: " + currentReading ); var history = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.range(2, 10) ); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } Both approaches model the domain in terms of latest state and historical observations. This is where a time-series database like Apache IoTDB is more effective than using a general-purpose database with timestamps as regular fields. Why Apache IoTDB Is More Than Sensor Storage While Apache IoTDB is often associated with IoT, its capabilities go well beyond basic sensor data storage. It excels when organizations must manage large volumes of device-centric data over time while retaining relationships among measurements, devices, and timestamps. As a result, IoTDB is valuable for industrial systems, utilities, transportation, smart infrastructure, manufacturing, and edge computing. For example, in predictive maintenance, a machine may continuously report vibration, temperature, pressure, energy consumption, and operating condition. The true value is not only found in the latest readings, but in studying how these measurements change together over time. Maintenance systems may compare recent data with historical trends, spot anomalies before failures, or correlate changes across multiple devices. This requires more than simple sensor data storage. This approach also benefits energy systems, where smart meters, solar panels, batteries, and grid equipment generate continuous measurement streams that require time-based analysis. In transportation, vehicles produce ongoing data such as speed, location, fuel consumption, battery status, and engine telemetry. Similarly, smart buildings rely on time-series data from HVAC systems, occupancy sensors, and energy meters. IoTDB is especially effective when system architecture is organized around devices and their measurements. It can serve as the historical data layer for operational systems, while other technologies manage transactional or business information. For example, a relational database might store details about a machine, its owner, contracts, or maintenance schedules, while IoTDB manages the extensive measurement data the machine generates. This separation lets enterprise applications treat telemetry as a primary workload, rather than forcing high-frequency device data into models meant for business entities and transactions. Conclusion Apache IoTDB is a strong fit for applications that need to ingest and analyze device-generated data over time, especially in IoT, industrial, energy, and edge scenarios. With Eclipse JNoSQL 1.1.18, Java developers can access these capabilities through familiar Jakarta APIs, keeping the application model consistent while still leveraging IoTDB’s time-series specialization.

By Otavio Santana

CORE

OpenSearch Heap Sizing: Swap, Page Cache, and the 50% Rule

OpenSearch is an open-source, distributed search and analytics suite derived as a fork of Elasticsearch and maintained under the Apache 2.0 license. When it comes to memory configuration, the guidance is often reduced to a few rules of thumb: swapoff -a, vm.swappiness=1, or bootstrap.memory_lock, and allocating 50% of available memory to the JVM heap while leaving the rest for Lucene and the filesystem page cache, OpenSearch off-heap caches, network buffers, and other system needs. These recommendations are repeated throughout documentation, blog posts, and operational guides, yet their origins and the mechanisms that justify these specific values are rarely examined. Undoubtedly, they provide a reasonable and safe starting point or a safe upper bound in most of the cases, but a safe default is not necessarily an optimal configuration. All of this raises even more questions. How do these defaults affect cluster performance? What is the optimal JVM heap ratio? Does memory given up by the JVM actually become filesystem page cache, and at what point does that trade-off stop paying off? How do read/write latency correlate with the heap ratio? These questions become particularly important in resource-constrained environments and in the cloud, where long-term contracts may make existing instances significantly cheaper, making horizontal or vertical scaling a difficult decision. In this article, we'll try to answer these questions through benchmarking. This is Act 1 of a two-act series. Act 1 focuses on identifying the cause of the latency problems we observed with the current defaults. Act 2 will explore what other heap-ratio values might look like for read/write loads. Along the way, I'll share the tools and commands used throughout the investigation, making this article a practical reference as well for you and for myself when I inevitably need to retrace the investigation months later. Knowledge Context The story also crosses several boundaries, such as the Kernel VM, the JVM, and Lucene. So, it’s important to outline the concepts mentioned in this part of the article beforehand, both for the context and, optionally, to enrich the AI context if you'd like to summarize everything. AreaWhere memory livesWhy it matters hereLinux page cacheFile-backed RAMLucene relies heavily on it for index data; under memory pressure, these pages can be reclaimed and read again later.Linux swapDisk-backed anonymous memoryAnonymous process memory can be swapped out under pressure. vm.swappiness influences this decision but does not prohibit it.Linux PSIKernel pressure signalShows time tasks spend stalled due to CPU, memory, or I/O pressure. We'll use I/O PSI while investigating latency.JVM heapAnonymous memoryControlled by Xms/Xmx; contains Java objects and several OpenSearch data structures.JVM native memoryAnonymous/file-backed memory outside XmxIncludes code cache, metaspace, stacks, direct buffers, and native allocations. Heap metrics do not account for all of it.OpenSearch cachesHeap/off-heap, depending on cacheTheir sizes may depend on heap size, which becomes important when we change jvm_heap_ratio.OpenSearch indexing bufferHeapIts size depends on heap and therefore becomes an important variable in Act 2.Lucene mmapFile-backed/page cacheLucene index files mapped into the process do not consume JVM heap; resident pages compete for physical RAM. Environment I used Aiven for OpenSearch on Azure, with cluster metrics exported to Thanos. The OpenSearch Benchmark metrics don't provide everything we need to answer our questions, particularly host-level metrics such as Linux PSI and swap activity. Exporting the cluster metrics to Thanos allows us to use PromQL queries later to retrieve the additional metrics needed for the investigation. The cluster consists of 3 nodes: CPUAMD EPYC 7763v (Milan)vCPU / RAM2 vCPU, 8 GiBDisk Size175 GiB per nodeAzure Regionazure-westeuropeAzure DiskPremiumV2_LRSAzure SKUStandard_D2as_v5 OpenSearch Version3.6.0JDKjava-21-openjdk-headlessGCG1GC Act 1. The Latency and an Extra GB The symptom: elevated query latency across the cluster, first reported by the customer after a kernel and Azure image upgrade. The load pattern on OpenSearch itself remained unchanged, as did the cluster configuration and settings. The monitoring panels give us the first clue. In the screenshots below, the green vertical line marks the moment of the upgrade. After that point, the page cache grows by roughly a gigabyte, while I/O PSI, previously close to zero, starts showing significant spikes. Nothing crashed, no alert fired, and from the JVM point of view everything looked normal. So where did that extra gigabyte of page cache come from? Nothing was actually freed. It moved. The interesting part isn't just that memory went to swap; it's which memory. When you have thousands of running clusters, there is always a small fraction of them operating close to the edge: relatively stable, yet sensitive enough that even a small change can noticeably affect performance. Like a star nearing the end of its lifetime, they may look stable right up until something disturbs the balance. The immediate cause of the page-cache change was identified fairly quickly: Azure applies tuning parameters that differ from the Linux kernel defaults, and those parameters were not applied by the older image. Once applied, the larger buffers and read_ahead increased the filesystem cache footprint, putting additional pressure on anonymous memory and eventually pushing some of it to swap. But rather than stopping there, let's use this incident as an opportunity to experiment with the heap ratio and make the behavior of OpenSearch instances explicit and less dependent on such environmental changes. Evidence It Is on Swap; None of It Locked First, find what is going on on a node itself: Shell PID=$(pgrep -f 'org.opensearch.bootstrap.OpenSearch') grep -E 'VmRSS|RssAnon|RssFile|RssShmem|VmSwap|VmLck' /proc/$PID/status Shell VmRSS: 4366408 kB # resident RssAnon: 3329896 kB # heap + anonymous native RssFile: 1036496 kB # resident mmap'd Lucene pages RssShmem: 16 kB VmSwap: 2679496 kB # on swap VmLck: 0 kB # bootstrap.memory_lock=fasle, none locked Shell grep -E 'MemFree|MemAvailable|Cached|SwapFree' /proc/meminfo Shell MemFree: 258216 kB MemAvailable: 3479264 kB Cached: 3403196 kB SwapCached: 854396 kB SwapFree: 4745444 kB The Swap Device Is dm-crypt Then confirm there is somewhere for it to go, and on what kind of device: Shell swapon --show Plain Text NAME TYPE SIZE USED PRIO /dev/dm-4 partition 8G 3.5G -1 The dm-* swap device is the interesting detail to catch and to keep in mind. This is an encrypted device, so once a page is requested it could drive more I/O -> more dm-crypt allocations -> more high-order pressure. A self-reinforcing loop and a good example of read amplification. Paging Is Live, Not Historical The next logical step is to check whether it is live paging or just a stale historical tail. The vmstat 1 5 the Linux Virtual Memory Statistics Tool should give us an exact answer for this, where non-zero swap blocks in and out (marked as si, so): Shell vmstat 1 5 Plain Text procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu------- r b swpd free buff cache si so bi bo in cs us sy id wa st gu 1 0 3620928 159716 5864 3704212 113 80 3361 577 3147 12 8 7 84 1 0 0 0 0 3625240 189036 5860 3705260 0 5228 24268 5228 4841 4072 14 12 69 4 0 0 0 0 3625240 166196 5860 3710696 0 0 32 0 3572 2556 21 5 74 0 0 0 0 0 3625240 162668 5860 3716200 8 0 164 0 3729 2619 20 7 73 0 0 0 12 0 3625312 147692 5860 3724288 0 140 9207 524 3679 4024 22 14 63 1 0 0 Non-zero si/so in 4 of 5 samples show the live paging process, swpd also climbing across five seconds, which is good proof. The Swapped Pages Are Anonymous, Not File-Backed Let's also check swap memory consumption for each of the process's mappings, to confirm that swap is heap-related: Shell PID=$(pgrep -f org.opensearch.bootstrap.OpenSearch) awk '/^[0-9a-f]/{h=$0} /^Swap:/{if($2>0)print $2" kB "h}' /proc/$PID/smaps | sort -rn | head -20 Plain Text 1160380 kB 708400000-7ffe00000 rw-p 00000000 00:00 0 62296 kB 7f30c0000000-7f30c3f4b000 rw-p 00000000 00:00 0 61452 kB 7f1fc8000000-7f1fcbc03000 rw-p 00000000 00:00 0 61448 kB 7f1fa8000000-7f1fabc02000 rw-p 00000000 00:00 0 61444 kB 7f2f38000000-7f2f3bc01000 rw-p 00000000 00:00 0 61444 kB 7f24b4000000-7f24b7c01000 rw-p 00000000 00:00 0 61444 kB 7f22ac000000-7f22afc01000 rw-p 00000000 00:00 0 61444 kB 7f2238000000-7f223bc01000 rw-p 00000000 00:00 0 61444 kB 7f216c000000-7f216fc01000 rw-p 00000000 00:00 0 61444 kB 7f2168000000-7f216bc01000 rw-p 00000000 00:00 0 61444 kB 7f2148000000-7f214bc01000 rw-p 00000000 00:00 0 61444 kB 7f1fec000000-7f1fefc01000 rw-p 00000000 00:00 0 61444 kB 7f1fe8000000-7f1febc01000 rw-p 00000000 00:00 0 61444 kB 7f1fcc000000-7f1fcfc01000 rw-p 00000000 00:00 0 61444 kB 7f1fac000000-7f1fafc01000 rw-p 00000000 00:00 0 60772 kB 7f1fc0000000-7f1fc3c14000 rw-p 00000000 00:00 0 59952 kB 7f311e000000-7f3123f41000 rw-p 00000000 00:00 0 59340 kB 7f30b4000000-7f30b7c9b000 rw-p 00000000 00:00 0 57180 kB 7f3110000000-7f3113e3e000 rw-p 00000000 00:00 0 43084 kB 7f3130800000-7f3133d70000 rwxp 00000000 00:00 0 The Largest Swapped Region Is But why is what we are seeing above a heap-related area? There are a few clues for that. The region size 0x708400000 - 0x7ffe00000 is exactly 4,154,458,112 bytes = 3,962 MiB as we use -Xms == -Xmx and the whole thing is committed at startup, and nothing else in a JVM process is a single contiguous ~4 GB anonymous rw-p mapping. Second, It's the lowest mapping in the address space, smaps_rollup [rollup] line starts at exactly 708400000: Shell cat /proc/$PID/smaps_rollup Shell 708400000-7ffd3bb56000 ---p 00000000 00:00 0 [rollup] Private_Dirty: 3213788 kB Swap: 2680532 kB SwapPss: 2679412 kB Locked: 0 kB The JIT Code Cache Is Swapped Too Decoding the top swapped regions: 1160380 kB at 708400000 – is the JVM heap, the compressed‑oops heap base and matches the [rollup] start from smaps_rollup.The dozens of 61444 kB regions – these areas are probably related to native/off‑heap: Netty, JNI, Lucene native, etc.43084 kB marked rwxp – the JIT code cache, also swapped out, a bad sign. Together, these regions account for almost exactly the ~2.5 GB of swapped memory we observed earlier: cold heap regions, native/off-heap allocations, code cache, and possibly thread stacks. Practically, this means two consequences: GC can amplify swap latency. 1 GB of the JVM heap was swapped out. G1 does not necessarily touch all of those pages during a mixed collection, but any GC phase that accesses a swapped page incurs a major fault and has to bring it back through the dm-crypt device. Hence, short GC work can produce significantly longer pauses.A swapped page can also contain executable code. The next call into a swapped-out compiled method can trigger a major fault before the code can run. The resulting latency may land on an otherwise random request and be difficult to attribute directly to GC, index I/O, or the query itself. Evidence of Sustained Anon Memory Churn workingset_refault_anon counts anonymous memory refault events after reclaim; it does not count unique pages. Together with pswpin and pswpout, it shows how much anonymous memory paging has accumulated since boot. Shell grep -E 'workingset_(refault|activate)_anon|pswpin|pswpout' /proc/vmstat Plain Text workingset_refault_anon 89061837 workingset_activate_anon 1856069 pswpin 86690374 pswpout 61618603 These counters are cumulative, so fetching them at 10-minute intervals clearly shows that this wasn't a one-time eviction. There was sustained process: nearly 89 million anonymous pages were repeatedly swapped out and faulted back in. vm.swappiness = 1 and GC Amplification In this story vm.swappiness=1 was set since the cluster's inception. It does what Linux defines it to do, but it doesn't provide the protection we wanted. I suspect this matters particularly in the most resource-constrained deployments. How do we know that? The entire result above is a counterexample. swappiness biases the kernel's choice between reclaiming file-backed and anonymous pages. It does not prevent anonymous pages from being swapped out. Even at 1, this can still happen under sustained memory pressure. On a resource-constrained node whose index is several times larger than its RAM, pressure on the filesystem cache is not an exceptional condition; this is the normal operating state. This has two important consequences: It is not self-healing. Nothing proactively pages anonymous memory back in on a schedule. A swapped-out page returns to RAM only when it is accessed again, and cold memory, as it's defined, may remain untouched for a long time. As a result, the cold JVM memory can remain in swap indefinitely.It may remain invisible until something touches it. At steady state, a cold tail of the heap can remain in swap without producing obvious symptoms. The problem becomes visible when those pages are touched again, causing major page faults and potentially amplifying GC and request latency. Key Takeaways So, the root cause of the latency problems is an oversized heap combined with page cache pressure (the cold heap tail has been swapped out). vm.swappiness=1 did not protect the heap. It biases what gets reclaimed; it does not prevent anonymous memory from being swapped out.vm.swappiness=1 should not be relied on with the other defaults in production. An oversized heap can lead to GC amplification that is difficult to detect.Most of the swapped-out memory wasn't heap at all, but malloc arenas and the JIT code cache, none of it inside Xmx, so heap metrics didn't show it.Swap on dm-crypt exacerbates the issue, resulting in longer GC pauses and random request latency spikes. dm-crypt may be unavoidable in production due to security requirements. The fix isn't another swappiness tweak. It's two things: stop committing heap you don't use, and make the heap you do commit non-evictable. Follow Up In Act 2, we'll answer the remaining questions raised at the beginning of this article and look more closely at the trade-off introduced by bootstrap.memory_lock. This setting makes the heap resident and swap-immune, but it also turns jvm_heap_ratio from a soft default into a permanent memory commitment. The question then becomes: what heap ratio best suits different read and write workloads? There is one more complication to mention in advance: in a resource-constrained environment, merge storms can distort benchmark results, making an otherwise good heap ratio appear poor. See the screenshot below.

By Maxim Muzafarov

Wasm Inside Neo4j: Building the Example That Didn't Exist

In a recent DZone article, Running Sentiment Analysis Inside Neo4j With a Java Plugin, we explored several approaches to running sentiment analysis inside the Neo4j database engine. One of those approaches — embedding a Wasm runtime inside a Java UDF — was described like this: Theoretically, we could embed a Wasm runtime such as wasmtime inside a Java UDF and execute the VADER Wasm module from within Neo4j, getting Wasm's sandbox guarantees inside Neo4j's plugin model. It's technically feasible but no published working example appears to exist and the complexity cost is high relative to the alternatives. An interesting idea to watch, but not practical today. This article builds that working example. We'll show how to embed a real VADER sentiment analyzer compiled to Wasm inside a Neo4j Java UDF, returning a full polarity score map callable directly from Cypher. We'll cover the tools and inspection techniques needed to understand what the Wasm compiler generates and why the Java calling convention looks the way it does. The full source code is available on GitHub. What We're Building We're embedding a wasmtime Wasm runtime inside a Neo4j Java UDF using wasmtime-java, a community JNI binding for the Wasmtime runtime. It's not an official Bytecode Alliance product, but it ships prebuilt native libraries for all major platforms and is sufficient for this proof-of-concept. A Rust function compiled to WebAssembly rides inside the plugin JAR alongside the Java code. When Cypher calls the UDF, Java initializes the Wasm runtime, loads the binary, and invokes the Rust function — all inside the Neo4j JVM process with no external API calls and no network round-trips. Note: This article was tested specifically against wasmtime-java 0.19.0. The API used here is version-specific; newer releases or alternative JVM Wasm runtimes may expose different interfaces and calling conventions. Prerequisites You'll need the following installed if you wish to follow along. We're using Apple Silicon (ARM64) as our development platform, so we'll note where the setup differs from other platforms. Java We're using OpenJDK 21 (tested with 21.0.12.1). Install it using your platform's package manager or download it directly from adoptium.net. On macOS via Homebrew: Shell brew install openjdk@21 On Ubuntu/Debian: Shell sudo apt install openjdk-21-jdk On Windows, download and run the installer from Adoptium. Confirm your Java version: Shell java -version You should see a Java 21 runtime. If you're on Apple Silicon, also confirm you're running a native ARM64 JVM with: Shell uname -m You should see arm64. Not running under ARM64 will likely break the wasmtime-java JNI library loading. Maven We're using Maven 3.9.6. On Apple Silicon, be cautious about installing Maven via Homebrew as, at the time of writing, the Homebrew Maven formula pulls in OpenJDK 26 as a dependency, which conflicts with a Java 21 installation. If your package manager installs an incompatible JDK alongside Maven, verify the runtime with mvn -version and configure JAVA_HOME as necessary. Installing Maven manually is the safest approach: Shell cd ~ curl -O https://archive.apache.org/dist/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.tar.gz tar xzf apache-maven-3.9.6-bin.tar.gz Then add Maven to your PATH and make it persist across terminal sessions: Shell echo 'export PATH="$HOME/apache-maven-3.9.6/bin:$PATH"' >> ~/.zshrc source ~/.zshrc On Linux, add the same line to ~/.bashrc instead.On Windows, download the zip from maven.apache.org and add the bin folder to your system PATH via System Properties. Confirm Maven is using Java 21: Shell mvn -version You should see Java version: 21 in the output. Rust We're using Rust 1.96.0. Install via rustup if not already present: Shell curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh On Windows, download and run rustup-init.exe from rustup.rs. To pin to the specific Rust version we tested with: Shell rustup toolchain install 1.96.0 rustup default 1.96.0 Then add the WASI target: Shell rustup target add wasm32-wasip1 This target works identically across macOS, Linux, and Windows. WABT The WebAssembly Binary Toolkit gives us wasm-objdump for inspecting Wasm binaries. We tested with version 1.0.41. On macOS: Shell brew install wabt On Ubuntu/Debian: Shell sudo apt install wabt On Windows, download the latest release from github.com/WebAssembly/wabt/releases. wit-bindgen This is the interface types generator for WebAssembly. There are two distinct version numbers to be aware of: the wit-bindgen-cli command-line tool and the wit-bindgen Rust crate used as a dependency inside the Wasm module. These can differ. We tested with CLI version 0.59.0 and Rust crate version 0.40.0 (specified in Cargo.toml). The generated binary identifies the crate version through the export name cabi_realloc_wit_bindgen_0_40_0. Install the pinned CLI version via Cargo on all platforms: Shell cargo install wit-bindgen-cli --version 0.59.0 Confirm it's installed: Shell wit-bindgen --version Neo4j Desktop We're using Neo4j Desktop with a local database instance. Download from Neo4j for Desktop. The pom.xml in this article is pinned to Neo4j 2026.07.0 — update the neo4j.version property to match your own Desktop installation. wasmtime-java Platform Support The wasmtime-java library ships prebuilt JNI native libraries for: macOS aarch64macOS x86_64Linux aarch64Linux x86_64Windows x86_64 No additional setup is needed, as Maven pulls the correct native library for your platform automatically. Version Summary For reference, here are all the component versions used in this article: ComponentVersionOpenJDK21.0.12.1Maven3.9.6Rust1.96.0WABT1.0.41wit-bindgen CLI0.59.0wit-bindgen crate0.40.0vader_sentiment crate0.1.1wasmtime-java0.19.0Neo4j2026.07.0 Getting the Code Clone the repository before following along. All source files are provided so you don't need to create them manually. Shell cd ~ git clone --filter=blob:none --sparse https://github.com/VeryFatBoy/neo4j.git cd neo4j git sparse-checkout set wasm-udf mv wasm-udf ../wasm-udf cd ../wasm-udf Project Structure Before creating any files, here's the final layout we're building toward. There are two separate projects: A Rust crate that compiles to Wasm.A Maven project that hosts the Neo4j UDF. First, the Rust crate: Plain Text sentimentable/ ├── Cargo.toml ├── src/ │ └── lib.rs └── wit/ └── sentimentable.wit Second, the Maven project: Plain Text neo4j-wasm-udf/ ├── pom.xml └── src/ └── main/ ├── java/ │ └── com/example/ │ ├── WasmUDF.java │ └── SentimentUDF.java └── resources/ ├── add.wat ├── add.wasm └── sentimentable.wasm The Wasm binaries in resources/ are bundled into the plugin JAR at build time. The Rust crate and Maven project are kept separate, and the Wasm binary is the handoff point between them. The project layout is also shown in Figure 1. Figure 1. Two-Project Layout How the Wasm Plumbing Works Before diving into the code, it's worth understanding the three layers that make this possible. Core Wasm and WASI WebAssembly defines a portable binary format and a stack-based execution model. On its own, it only understands numbers, such as integers and floats. When a Wasm module needs system capabilities, like memory allocation or I/O, it uses WASI (WebAssembly System Interface), a standardized set of system calls that a host runtime implements. Our Rust code targets wasm32-wasip1, which means it compiles to Wasm with WASI preview 1 system calls. The wasmtime runtime implements those calls on the host side. wasmtime-java This library wraps the wasmtime Wasm runtime in a JNI binding, making it callable from Java. It ships prebuilt native libraries for all major platforms, so adding it as a Maven dependency is all that's needed — no separate wasmtime installation required. The Java API lets us load a Wasm binary, set up a WASI context, and call exported functions directly. wit-bindgen and the String ABI Core WebAssembly functions operate on Wasm value types such as integers and floats. WIT (WebAssembly Interface Types) and the Component Model provide higher-level interface types such as strings, tuples, and records; wit-bindgen generates the lowering and lifting code needed to represent those types at the Wasm boundary. For strings, it uses a pointer-and-length convention: the caller allocates memory inside the Wasm module using a generated cabi_realloc function, writes the string bytes there and passes the memory address and byte length as two integers. The Rust code reads the string from that address. For return values, the lowering strategy depends on the type, which we'll see when we inspect the generated binary. With those three pieces in place, the calling chain looks like this: Plain Text Cypher query -> Neo4j routes to @UserFunction -> Java initializes wasmtime engine + WASI context -> Java allocates string in Wasm memory -> Java calls exported Wasm function -> Rust executes VADER scoring -> Java reads result from Wasm memory -> Java returns Map<String, Double> to Neo4j -> Neo4j returns result to Cypher Graphically, the calling chain is also shown in Figure 2. Figure 2. Calling Chain We built up to this through two simpler stepping-stone cases: Case 1: A trivial integer addition to prove the chain works.Case 2: A single compound score to introduce string passing and WASI. The full walkthrough of both, including the WasmUDF.java implementation is in a technical report on the GitHub repo. The Maven Project XML <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>neo4j-wasm-udf</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <properties> <maven.compiler.source>21</maven.compiler.source> <maven.compiler.target>21</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <neo4j.version>2026.07.0</neo4j.version> </properties> <dependencies> <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>${neo4j.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>io.github.kawamuray.wasmtime</groupId> <artifactId>wasmtime-java</artifactId> <version>0.19.0</version> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>21</source> <target>21</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.5.1</version> <executions> <execution> <phase>package</phase> <goals><goal>shade</goal></goals> <configuration> <artifactSet> <excludes> <exclude>org.neo4j:*</exclude> </excludes> </artifactSet> <shadedArtifactAttached>false</shadedArtifactAttached> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> Two things worth noting here: org.neo4j:neo4j is declared as provided scope — Neo4j is already present in the database JVM at runtime, so we exclude it from the bundled JAR.We use maven-shade-plugin rather than maven-jar-plugin to produce a fat JAR that bundles wasmtime-java and its native libraries alongside our code. Update the neo4j.version property to match your own Neo4j Desktop installation. Case 3: Full Polarity Map VADER produces four scores: compound, positive, negative, and neutral. In this case, we update the Rust function to return all four and the Java UDF to return them as a Map<String, Double> — matching the return shape of the Java VADER UDF from the previous article. The sentimentable.wit File We change the return type from a single f32 to a tuple of four f32 values: Plain Text package local:sentimentable; world sentimentable { export sentimentable: func(input: string) -> tuple<f32, f32, f32, f32>; } We use a tuple rather than a named record. Both would work, but a tuple is simpler on the Java side — we read four consecutive f32 values from memory at known offsets without needing to decode field names. The lib.rs File Rust wit_bindgen::generate!({ world: "sentimentable", }); struct Component; impl Guest for Component { fn sentimentable(input: String) -> (f32, f32, f32, f32) { lazy_static::lazy_static! { static ref ANALYZER: vader_sentiment::SentimentIntensityAnalyzer<'static> = vader_sentiment::SentimentIntensityAnalyzer::new(); } let scores = ANALYZER.polarity_scores(input.as_str()); ( *scores.get("compound").unwrap_or(&0.0) as f32, *scores.get("pos").unwrap_or(&0.0) as f32, *scores.get("neg").unwrap_or(&0.0) as f32, *scores.get("neu").unwrap_or(&0.0) as f32, ) } } export!(Component); Build: Shell cd ~/wasm-udf/sentimentable cargo build --target wasm32-wasip1 --release Inspecting the Binary Let's check the exports first: Shell wasm-objdump -x target/wasm32-wasip1/release/sentimentable.wasm | grep "^Export" -A 6 Four exports should be present: memory, sentimentable, cabi_realloc, and cabi_realloc_wit_bindgen_0_40_0. Now let's find the type signature of the sentimentable function. Find the sig index: Shell wasm-objdump -x target/wasm32-wasip1/release/sentimentable.wasm | grep "func\[9\]" | head -1 Then look it up: Shell wasm-objdump -x target/wasm32-wasip1/release/sentimentable.wasm | grep "type\[9\]" You should see: Plain Text - type[9] (i32, i32) -> i32 The signature is (i32, i32) -> i32 . This is the key difference between returning a single scalar and returning a tuple: wit-bindgen uses a direct f32 return for a single value, but switches to an indirect result pointer when returning a tuple. What's written at that pointer is four f32 values (16 bytes) at consecutive 4-byte offsets. The Java side reads all four. This illustrates an important distinction between the WIT interface definition and the generated core Wasm ABI. The WIT signature and the Wasm-level signature are different layers: wit-bindgen lowers WIT types to a core Wasm ABI, and the lowering strategy depends on the return type. A single scalar such as f32 is returned directly as a Wasm value. A tuple is returned indirectly through linear memory, with the caller receiving a pointer to where the values were written. The Java calling code must match the generated ABI rather than the WIT definition, which is why inspecting the binary with wasm-objdump before writing the Java wrapper is essential. Figure 3 shows the memory layout. Figure 3. Memory Layout. Figure 4 compares Cases 2 and 3. Figure 4. Case 2 vs. Case 3 ABI Comparison The SentimentUDF.java file Java package com.example; import io.github.kawamuray.wasmtime.Engine; import io.github.kawamuray.wasmtime.Func; import io.github.kawamuray.wasmtime.Linker; import io.github.kawamuray.wasmtime.Memory; import io.github.kawamuray.wasmtime.Module; import io.github.kawamuray.wasmtime.Store; import io.github.kawamuray.wasmtime.WasmFunctions; import io.github.kawamuray.wasmtime.WasmValType; import io.github.kawamuray.wasmtime.wasi.WasiCtx; import io.github.kawamuray.wasmtime.wasi.WasiCtxBuilder; import org.neo4j.procedure.Description; import org.neo4j.procedure.Name; import org.neo4j.procedure.UserFunction; import java.io.InputStream; import java.nio.ByteBuffer; import java.nio.ByteOrder; import java.nio.charset.StandardCharsets; import java.util.HashMap; import java.util.Map; public class SentimentUDF { @UserFunction("com.example.wasm.sentiment") @Description("Scores text using VADER sentiment analysis compiled to Wasm. Returns compound, positive, negative, neutral.") public Map<String, Double> sentiment(@Name("text") String text) throws Exception { if (text == null || text.isBlank()) { return Map.of("compound", 0.0, "positive", 0.0, "negative", 0.0, "neutral", 1.0); } byte[] wasmBytes; try (InputStream is = SentimentUDF.class.getResourceAsStream("/sentimentable.wasm")) { if (is == null) throw new RuntimeException("sentimentable.wasm not found in resources"); wasmBytes = is.readAllBytes(); } WasiCtx wasi = new WasiCtxBuilder().inheritStdout().inheritStderr().build(); try (Store<Void> store = Store.withoutData(wasi); Engine engine = store.engine(); Module module = Module.fromBinary(engine, wasmBytes); Linker linker = new Linker(engine)) { WasiCtx.addToLinker(linker); linker.module(store, "", module); Memory memory = linker.get(store, "", "memory").get().memory(); Func reallocFn = linker.get(store, "", "cabi_realloc").get().func(); WasmFunctions.Function4<Integer, Integer, Integer, Integer, Integer> realloc = WasmFunctions.func(store, reallocFn, WasmValType.I32, WasmValType.I32, WasmValType.I32, WasmValType.I32, WasmValType.I32); byte[] inputBytes = text.getBytes(StandardCharsets.UTF_8); int len = inputBytes.length; int strPtr = realloc.call(0, 0, 1, len); ByteBuffer buf = memory.buffer(store); buf.position(strPtr); buf.put(inputBytes); Func sentimentFn = linker.get(store, "", "sentimentable").get().func(); WasmFunctions.Function2<Integer, Integer, Integer> scoreFn = WasmFunctions.func(store, sentimentFn, WasmValType.I32, WasmValType.I32, WasmValType.I32); int resultPtr = scoreFn.call(strPtr, len); // read four f32 values at 4-byte offsets: compound, pos, neg, neu ByteBuffer resultBuf = memory.buffer(store); resultBuf.order(ByteOrder.LITTLE_ENDIAN); float compound = resultBuf.getFloat(resultPtr); float positive = resultBuf.getFloat(resultPtr + 4); float negative = resultBuf.getFloat(resultPtr + 8); float neutral = resultBuf.getFloat(resultPtr + 12); Map<String, Double> result = new HashMap<>(); result.put("compound", (double) compound); result.put("positive", (double) positive); result.put("negative", (double) negative); result.put("neutral", (double) neutral); return result; } } } The return type is (Map<String, Double> ), the null guard returning a neutral map and the four getFloat() reads at consecutive 4-byte offsets from the result pointer. Build and Deploy Copy the Wasm binary, build and deploy: Shell cp ~/wasm-udf/sentimentable/target/wasm32-wasip1/release/sentimentable.wasm \ ~/wasm-udf/neo4j-wasm-udf/src/main/resources/ cd ~/wasm-udf/neo4j-wasm-udf mvn -q clean package cp target/neo4j-wasm-udf-1.0-SNAPSHOT.jar \ ~/Library/Application\ Support/neo4j-desktop/Application/Data/dbmss/<your-dbms-id>/plugins/ Stop Neo4j, restart it, and run the verification queries. Positive sentence: Cypher RETURN com.example.wasm.sentiment('The movie was great') AS scores; Result: JSON { "compound": 0.624893307685852, "positive": 0.577464759349823, "negative": 0.0, "neutral": 0.4225352108478546 } Capitalization test: Cypher RETURN com.example.wasm.sentiment('The movie was GREAT!') AS scores; Result: JSON { "compound": 0.7290259003639221, "positive": 0.6307692527770996, "negative": 0.0, "neutral": 0.3692307770252228 } Empty string guard: Cypher RETURN com.example.wasm.sentiment('') AS scores; Result: JSON { "compound": 0.0, "positive": 0.0, "negative": 0.0, "neutral": 1.0 } All three cases behave correctly. Summary We set out to build the working example that our previous article said didn't exist. Here's what we showed. We embedded a real VADER sentiment analyzer compiled to Wasm inside a Neo4j Java UDF, returning a full polarity map — compound, positive, negative and neutral — matching the return shape of the Java VADER UDF from the previous article. The wit-bindgen tuple ABI writes four f32 values to consecutive memory addresses; the Java side reads them back with a LITTLE_ENDIAN ByteBuffer. All four scores are correct, capitalization sensitivity works, and the empty string guard returns a sensible neutral map. The result is a workable integration pattern rather than a universal replacement for a native Java implementation. With the per-call initialization used in this proof of concept, the approach is best suited to low-frequency workloads where Wasm isolation and portability justify the additional complexity. For high-throughput workloads, the natural next step is to benchmark and reuse the Wasmtime engine and compiled module while keeping execution state appropriately isolated between calls. In the next article, we'll look at running TypeSafe AI's Jev inside Neo4j for calibrated sentiment decisions. Stay tuned! The full source code is available on GitHub.

By Akmal Chaudhri

CORE

Part 3: End-to-End Tracing and Observability Across Goose, agentgateway, and Quarkus

Enterprise context — Acme FinServ. SOC 2 CC7 (system monitoring) requires that Acme can detect and investigate anomalous activity. When an agent-driven workflow touches customer data at 2 AM, "we have logs somewhere" is not an answer an auditor accepts. The distributed trace built in this part is the forensic evidence trail: a single trace ID that ties the Goose prompt to every agentgateway policy decision and every Quarkus tool call, so a post-incident review can reconstruct exactly which agent did what, in what order, and how long each governed hop took. The Core Problem In Part 1, we built a Quarkus MCP tool server. In Part 2, we secured it with agentgateway's JWT authentication, RBAC, and ExtMCP guardrails. The architecture works — but when something goes wrong in production, you're flying blind. Agentic workflows are fundamentally different from traditional request-response APIs. A single user prompt like "Debug customer CUST-4091" triggers a multi-round-trip loop: Goose calls tools/list to discover available toolsThe LLM selects getCustomerStatus and Goose sends tools/callThe LLM reads the response, sees primaryRegion: US-EAST-1, and chains a second tools/call to getZoneHealthLogsThe LLM correlates both results and generates a diagnostic summary Each of these hops crosses process boundaries: Goose → agentgateway → Quarkus. Without distributed tracing, you see four isolated HTTP requests in your access logs. You cannot tell they belong to the same agentic workflow. When step 3 takes 12 seconds instead of 200ms, you have no waterfall to pinpoint whether the latency came from agentgateway policy evaluation, Quarkus bean validation, or a slow downstream call. This creates black holes in telemetry dashboards — the exact gap that autonomous agents exploit to degrade silently. The Solution: W3C Trace Context Across All Three Layers The fix is standard distributed tracing, applied to the MCP transport layer: agentgateway exports spans for every proxied MCP request and propagates traceparent headers to the backend.Quarkus with quarkus-opentelemetry picks up the incoming traceparent, creates child spans for tool execution and bean validation, and exports them to the same Jaeger instance.Jaeger correlates both sides into a single trace waterfall — one view from agent prompt to tool result. Prerequisites Everything from Parts 1 and 2, plus: Podman – for running Jaeger (podman compose) Verify Podman is available: Shell podman --version Step 1: Launching the Observability Backend We use Jaeger v2 as both the OTLP collector and the trace UI. A single container accepts traces from agentgateway on port 4317 (OTLP gRPC) and from Quarkus on port 4318 (OTLP HTTP), and serves the query UI on port 16686. Shell cd part3-observability podman compose up -d This starts Jaeger v2 with OTLP collection enabled by default. Verify it's running: Shell curl -sf http://localhost:16686/ > /dev/null && echo "Jaeger UI is ready" Open http://localhost:16686 — you'll see an empty Jaeger UI. We'll populate it with MCP traces in the following steps. Production Alternative: Grafana Tempo For production deployments, replace Jaeger with Grafana Tempo backed by object storage (S3/GCS). The OTLP endpoint stays the same — only the compose.yml changes. Grafana provides richer dashboards, alerting, and long-term trace retention. Step 2: Enabling OpenTelemetry in Quarkus Add the quarkus-opentelemetry extension to Part 1's pom.xml: Properties files <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-opentelemetry</artifactId> </dependency> Configure the exporter in application.properties: Properties files # OpenTelemetry quarkus.otel.service.name=customer-tools quarkus.otel.exporter.otlp.traces.endpoint=http://localhost:4318 quarkus.otel.exporter.otlp.traces.protocol=http/protobuf quarkus.otel.traces.sampler=always_on quarkus.otel.traces.suppress-non-application-uris=false PropertyPurposeservice.nameIdentifies this service in Jaeger's service dropdowntraces.endpointOTLP HTTP receiver — Jaeger's port 4318 (base URL only; Quarkus appends /v1/traces)traces.protocolhttp/protobuf — Quarkus uses its Vert.x-based HTTP exportertraces.sampleralways_on — sample every span (reduce in production)suppress-non-application-urisfalse — include MCP endpoint spans (they'd be filtered otherwise) When no OTLP collector is running (Parts 1 and 2 without Jaeger), Quarkus logs a connection warning, but the MCP server works normally. When the collector IS running (Part 3), traces flow automatically. Zero code changes to the MCP tools. Rebuild Part 1: Shell cd part1-quarkus-mcp mvn package -DskipTests What Quarkus Auto-Instruments With quarkus-opentelemetry on the classpath and the SDK enabled, Quarkus automatically creates spans for: LayerSpan NameWhat It CapturesHTTP serverPOST /mcpInbound MCP request with method, status, latencyCDI beansCustomerServiceTools.getCustomerStatusTool execution time within the MCP handlerBean ValidationHibernateValidatorParameter validation before tool logic runsREST clientOutbound HTTP callsAny downstream API calls (future extensions) No @WithSpan annotations needed. The Quarkus OpenTelemetry extension instruments the reactive pipeline automatically. Step 3: Configuring W3C Trace Context in agentgateway agentgateway supports native OpenTelemetry trace export. Add a tracing block to the gateway configuration: Properties files config: adminAddr: localhost:15000 tracing: otlpEndpoint: http://localhost:4317 otlpProtocol: grpc randomSampling: 1.0 FieldPurposeotlpEndpointOTLP receiver — Jaeger's port 4317otlpProtocolgrpc for OTLP/gRPC (also supports http)randomSamplingSample 100% of traces (reduce to 0.01–0.1 in production) How Trace Propagation Works When agentgateway receives an MCP request: Creates a root span for the proxy operation (e.g., agentgateway.mcp.proxy)Injects a traceparent header into the forwarded request to Quarkus:traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01Quarkus reads the traceparent, creates a child span under the same trace ID, and records tool executionBoth spans export to Jaeger via OTLP, where they appear as a single correlated trace This is standard W3C Trace Context propagation — the same mechanism used across all OpenTelemetry-instrumented services. Configuration Files Part 3 provides two agentgateway configurations: ConfigUse Caseconfig-traced.yamlTracing only — proxy + OTLP export, no security layersconfig-traced-guardrails.yamlTracing + ExtMCP guardrails — observe the guardrail evaluation spans too Step 4: Running the Interactive Demo Start all services with the one-command script: Shell cd part3-observability ./start-all.sh The script starts Jaeger, Quarkus (with OTel enabled), and agentgateway (with trace export), then launches the demo SPA on :8890. Open the MCP Observability Console at http://localhost:8890/index.html and walk through the three demo steps: Initialize – Establishes an MCP session through agentgateway. The architecture diagram animates the trace propagation: root span creation in agentgateway, traceparent injection, child span in Quarkus, and OTLP export to Jaeger.List Tools – Discovers all 5 tools through the traced proxy. The trace waterfall panel shows the agentgateway proxy span and the Quarkus HTTP span side by side with timing.Multi-Tool Workflow – Simulates Goose's multi-turn reasoning: getCustomerStatus (finds region US-EAST-1) → getZoneHealthLogs (checks zone health) → getSLACompliance (correlates SLA metrics). Each step generates a full trace with waterfall visualization. The stat tiles track traces generated, spans collected, and Jaeger status. Click Open Jaeger to view the real trace waterfalls in the Jaeger UI at http://localhost:16686. Step 5: Generating Traces via CLI To generate additional traces manually, simulate a multi-turn agentic workflow: Shell # Step 1: Initialize MCP session export MCP_SESSION_ID=$(curl -s -D - http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"curl","version":"1.0"}}' \ | grep -i "mcp-session-id:" | sed 's/.*: //' | tr -d '\r') # Step 2: Discover tools curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}' \ | grep '^data: ' | sed 's/^data: //' | jq . # Step 3: Agent calls getCustomerStatus (first tool invocation) curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"getCustomerStatus","arguments":{"customerId":"CUST-4091"}}' \ | grep '^data: ' | sed 's/^data: //' | jq . # Step 4: Agent chains getZoneHealthLogs based on the region from step 3 curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"getZoneHealthLogs","arguments":{"zoneId":"US-EAST-1"}}' \ | grep '^data: ' | sed 's/^data: //' | jq . # Step 5: Agent fetches SLA compliance for correlation curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"getSLACompliance","arguments":{"serviceId":"api-gateway"}}' \ | grep '^data: ' | sed 's/^data: //' | jq . Each of these requests generates a trace that flows through agentgateway into Quarkus and lands in Jaeger. Step 6: Visualizing the Trace Waterfall in Jaeger Open http://localhost:16686 in your browser. Finding Traces In the Service dropdown, select customer-tools (Quarkus) or agentgatewayClick Find TracesClick on any trace to open the waterfall view Reading the Waterfall A typical tools/call trace shows the following span hierarchy: Shell agentgateway.mcp.proxy [12ms] └─ POST /mcp [8ms] ← Quarkus HTTP server └─ CustomerServiceTools.getCustomerStatus [2ms] ← CDI tool execution SpanServiceWhat It Tells Youagentgateway.mcp.proxyagentgatewayTotal proxy overhead including policy evaluationPOST /mcpcustomer-toolsQuarkus HTTP handling time for the MCP requestgetCustomerStatuscustomer-toolsPure tool execution time (business logic) What to Look For Proxy overhead: The gap between the agentgateway span and the Quarkus span shows network + policy evaluation time. If this grows, check guardrail server latency.Validation time: Bean Validation spans appear before tool execution. Regex-heavy patterns like ^CUST-[0-9]{4,8}$ are fast, but complex validators on large payloads can add latency.Multi-turn correlation: When Goose chains multiple tool calls (e.g., getCustomerStatus → getZoneHealthLogs), each appears as a separate trace. The mcp-session-id tag lets you filter all traces belonging to one agent session.Error traces: Failed validations (invalid customer ID format) or guardrail rejections (blocked poison payloads) produce error spans with exception details. Connecting Goose for Real Traces Launch Goose pointed at agentgateway and prompt a multi-tool workflow: Shell goose session "Debug customer CUST-4091 — check their account status, then pull health logs for their region and SLA compliance for api-gateway." This generates a burst of correlated traces in Jaeger showing Goose's multi-turn tool orchestration from the proxy layer down to individual tool execution spans. What We Achieved Starting from the secured architecture in Part 2, we added full observability without changing any MCP tool code: LayerWhat We AddedConfig ChangeQuarkusquarkus-opentelemetry dependencypom.xml + application.propertiesagentgatewaytracing block in config YAMLconfig-traced.yamlObservability backendJaeger all-in-one via Podman Composecompose.yml The entire stack runs locally with a single ./start-all.sh command and produces end-to-end trace waterfalls in Jaeger. Production Considerations ConcernLocal (this tutorial)ProductionTrace backendJaeger all-in-one (in-memory)Grafana Tempo + object storageSampling rate100% (default: 1.0)1-10% or adaptive samplingTrace retentionContainer lifetimeDays/weeks in durable storageAlertingManual Jaeger inspectionGrafana alerting on span latency/error rateMetricsTraces onlyAdd Prometheus + quarkus-micrometer for RED metrics Coming Up in Part 4 With tracing in place, you can now see every MCP tool call flowing through the system. In Part 4, we will move beyond single-agent tool calls to multi-agent orchestration — using the Agent-to-Agent (A2A) protocol to coordinate autonomous agents that can delegate work, enforce governance via AGENTS.md, and call back into our MCP tool services.

By Daniel Oh

CORE

Embabel vs LangGraph4j: Two Agentic Philosophies for Investment and Risk Analysis in BFSI

Quick Summary Both Embabel and LangGraph4j let a Java developer build multi-step AI agents without leaving the JVM.Embabel hands the framework a goal and a bag of typed actions, and lets a planner decide the order on its own.LangGraph4j asks the developer to draw the exact graph of nodes and edges by hand.We will understand both philosophies through a real Embabel agent, a small Kolkata street-crossing example, a comparison table, and finally a bigger question — is Java catching up with Python in enterprise AI work? Where the Story Starts If you are a Java developer today, you are watching two worlds collide. On one side are large language models, which grew up almost entirely in Python. On the other side is enterprise Java, which has spent twenty-five years learning to build systems that banks, insurance companies, and hospitals can actually trust. Two frameworks are now trying to bring these two worlds together on the JVM: Embabel and LangGraph4j. Both help you build an "agent" — a piece of software that uses an LLM to complete a task in several steps, rather than in one single prompt. But the way they think about "steps" is completely different. That difference is what this article is about. Meeting Embabel Through a Real Piece of Code The best way to understand Embabel is to look at actual code, not a slide. Here is a small agent that gives retirement planning advice, written the Embabel way. Java package com.example.demo; import com.embabel.agent.api.annotation.AchievesGoal; import com.embabel.agent.api.annotation.Action; import com.embabel.agent.api.annotation.Agent; import com.embabel.agent.api.common.OperationContext; import com.embabel.agent.domain.io.UserInput; import java.util.Arrays; import java.util.List; @Agent(name = "RetirementPlannerAgent", description = "This agent provides retirement planning advice.") public class RetirementPlannerAgent { record RetirementUserInput(int presentAge, int targetRetirementAge, double annualIncome) { } record RetirementPlanAdvice(String advice) { } record RetirementPlanAdvices(List<RetirementPlanAdvice> advices) { } enum RiskToleranceLevel { LOW, MEDIUM, HIGH } @Action(description = "Identify the present age, target retirement age, and annual income of the user from the user message.") public RetirementUserInput identifyRetirementUserInput(UserInput userInput, OperationContext context) { String content = userInput.getContent(); return context.ai().withDefaultLlm() .creating(RetirementUserInput.class) .fromPrompt(""" Identify the present age, target retirement age, and annual income of the user from the following message and return them as a JSON object. User message: %s """.formatted(content)); } @Action(description = "Identify the risk tolerance level of the user based on the present age, target retirement age, and annual income.") public RiskToleranceLevel identifyRiskToleranceLevel(RetirementUserInput retirementUserInput, OperationContext context) { return context.ai().withDefaultLlm() .creating(RiskToleranceLevel.class) .fromPrompt(""" Identify the risk tolerance level of the user based on the following information and return it as a JSON object. Permitted values for risk tolerance level are: %s Present age: %d Target retirement age: %d Maximum years to retirement: %d Annual income: %.2f """.formatted(Arrays.toString(RiskToleranceLevel.values()), retirementUserInput.presentAge(), retirementUserInput.targetRetirementAge(), (retirementUserInput.targetRetirementAge() - retirementUserInput.presentAge()), retirementUserInput.annualIncome())); } @Action(description = "Provide retirement plan advice based on the user's risk tolerance level.") @AchievesGoal(description = "Provide retirement plan advice based on the user's risk tolerance level.") public RetirementPlanAdvices provideRetirementPlanAdvice(RiskToleranceLevel riskToleranceLevel, OperationContext context) { String systemPrompt = """ You are a retirement planning advisor. Based on the user's risk tolerance level, provide a list of retirement plan advices. """; return context.ai().withDefaultLlm() .creating(RetirementPlanAdvices.class) .fromPrompt(""" %s User's risk tolerance level: %s """.formatted(systemPrompt, riskToleranceLevel.name())); } } Now look closely at what is missing from this code. There is no method called runAgent() that calls identifyRetirementUserInput(), then identifyRiskToleranceLevel(), then provideRetirementPlanAdvice(), in that order. Nowhere did the developer type out the sequence. Instead, each @Action simply states two things: What type it needs as input (its precondition).What type it produces as output (its effect). provideRetirementPlanAdvice needs a RiskToleranceLevel. identifyRiskToleranceLevel happens to produce a RiskToleranceLevel from a RetirementUserInput. And identifyRetirementUserInput produces that RetirementUserInput from the raw UserInput. Embabel's planner looks at all this at runtime and works out, on its own, that this is the only order in which the goal (@AchievesGoal) can be reached. This is Embabel's whole philosophy in one sentence: give the framework a goal and a set of typed building blocks, and let it plan. The Same Investment Advisory Workflow, Wired by Hand in LangGraph4j Now let us build the exact same three-step advisory flow — read the user's numbers, work out risk tolerance, give advice — the LangGraph4j way. Here, the developer draws the graph explicitly, and the shared state is a plain key-value map (an AgentState) rather than Soham's strongly typed records. Java package com.example.demo; import org.bsc.langgraph4j.CompiledGraph; import org.bsc.langgraph4j.StateGraph; import org.bsc.langgraph4j.action.NodeAction; import org.bsc.langgraph4j.state.AgentState; import java.util.List; import java.util.Map; import java.util.Optional; import static org.bsc.langgraph4j.StateGraph.END; import static org.bsc.langgraph4j.StateGraph.START; import static org.bsc.langgraph4j.action.AsyncNodeAction.node_async; /** * @author Soham Sengupta * @since 2026-09-13 * @description The retirement/investment advisory workflow from the * Embabel example above, this time wired explicitly as a LangGraph4j * graph. Every step, and the order between the steps, is declared * here by the developer - there is no planner discovering it. */ public class RetirementPlannerGraph { // Step 1: pull the present age, target retirement age, and income out of free text. static class IdentifyUserInputNode implements NodeAction<AgentState> { @Override public Map<String, Object> apply(AgentState state) throws Exception { String userMessage = state.<String>value("userMessage").orElseThrow(); // In a real system: call the LLM here (say, via langchain4j) and parse // presentAge / targetRetirementAge / annualIncome out of userMessage. return Map.of( "presentAge", 32, "targetRetirementAge", 60, "annualIncome", 1200000.0); } } // Step 2: classify how much investment risk this user can reasonably take. static class IdentifyRiskToleranceNode implements NodeAction<AgentState> { @Override public Map<String, Object> apply(AgentState state) throws Exception { int presentAge = state.<Integer>value("presentAge").orElseThrow(); int targetRetirementAge = state.<Integer>value("targetRetirementAge").orElseThrow(); // In a real system: call the LLM here with these values and ask it to // return LOW, MEDIUM, or HIGH as the risk tolerance level. String riskTolerance = (targetRetirementAge - presentAge) > 20 ? "HIGH" : "MEDIUM"; return Map.of("riskTolerance", riskTolerance); } } // Step 3: turn the risk tolerance into a concrete list of investment advice. static class ProvideAdviceNode implements NodeAction<AgentState> { @Override public Map<String, Object> apply(AgentState state) throws Exception { String riskTolerance = state.<String>value("riskTolerance").orElseThrow(); // In a real system: call the LLM here to draft actual advice - suitable // Indian investment instruments for this riskTolerance level, and so on. List<String> advice = List.of("Suggested investment mix for a " + riskTolerance + " risk profile."); return Map.of("advice", advice); } } public static void main(String[] args) throws Exception { StateGraph<AgentState> graph = new StateGraph<>(AgentState::new) .addNode("identifyUserInput", node_async(new IdentifyUserInputNode())) .addNode("identifyRiskTolerance", node_async(new IdentifyRiskToleranceNode())) .addNode("provideAdvice", node_async(new ProvideAdviceNode())) .addEdge(START, "identifyUserInput") .addEdge("identifyUserInput", "identifyRiskTolerance") .addEdge("identifyRiskTolerance", "provideAdvice") .addEdge("provideAdvice", END); CompiledGraph<AgentState> workflow = graph.compile(); Optional<AgentState> result = workflow.invoke( Map.of("userMessage", "I am 32, want to retire at 60, and earn 12 lakh a year.")); result.ifPresent(state -> System.out.println(state.data())); } } Two things stand out next to the Embabel version. First, there is a main method here that explicitly lists identifyUserInput -> identifyRiskTolerance -> provideAdvice as edges — in the Embabel version, that sequence was never written down anywhere; it was worked out by the planner. Second, the shared state (AgentState) is just a bag of string keys and values, read back out with state.value("presentAge"), instead of Soham's own strongly typed RetirementUserInput and RiskToleranceLevel. For this particular workflow, which happens to be a strict straight line with no branching, LangGraph4j's graph is refreshingly easy to read top to bottom. The real difference shows up once branching enters the picture, which is exactly where our next example — crossing a Kolkata road — comes in. A Bit of History: From Servlet to Spring, Now From Spring AI to Embabel To understand why Embabel is built this way, it helps to know who built it. Embabel comes from Rod Johnson — the same person who created the Spring Framework more than two decades ago. Back in the early 2000s, enterprise Java was drowning in heavy J2EE application servers and Enterprise Java Beans. Rod was solving real problems in the finance industry at the time, found the existing tools too heavy, and wrote a book and a framework that simplified things a great deal. That framework became Spring, and it changed how an entire generation of Java developers worked. In 2025, Rod did something similar again, this time for AI agents. When he introduced Embabel to the Java community, he framed it using a comparison that Java developers will find very familiar: Spring AI is to Embabel roughly what the plain old Servlet API once was to Spring MVC. Spring AI gives you the low-level plumbing — talking to a model, building a prompt, calling a tool. Embabel sits one level above that, giving you the actual application framework — goals, actions, planning, and a proper domain model — the same kind of jump in abstraction that Spring itself brought to raw Servlets and EJBs, twenty years back. Embabel (pronounced "Em-BAY-bel") is written mainly in Kotlin, but as you can see from the retirement planner code above, it feels completely natural to use from plain Java. It is also built to sit closely with Spring, which is exactly why an existing Spring shop can pick it up without much friction. GOAP: The Planning Engine Hiding Inside Embabel The planning idea inside Embabel is not new — it is borrowed from video games, and it is called GOAP, short for Goal-Oriented Action Planning. GOAP was built to make game characters (think of soldiers in an old shooter game) decide, on their own, a believable sequence of actions to reach a goal, instead of following a fixed script. GOAP needs three things: A state of the world as it stands right now.A set of actions, each with a precondition (what must be true to run it) and an effect (what becomes true after it runs).A goal, which is simply a desired state. A search algorithm (usually the well-known A* algorithm) then works out the cheapest chain of actions that gets you from where you are to where you want to be. Embabel uses exactly this idea, but instead of asking the LLM to "think step by step" about the plan (which is often unreliable) or asking the developer to hard-code the entire flow (which is rigid), it asks a deterministic planner to search over your own typed Java or Kotlin methods. The precondition of an action is simply the input type it needs. The effect is simply the output type it produces. Your domain classes — RetirementUserInput, RiskToleranceLevel, RetirementPlanAdvices — literally become the "world state" the planner reasons about. No LLM guesswork is involved in deciding the order; the LLM is only used inside each action, for the part it is actually good at — understanding and generating language. Understanding the GOAP + OOAD Loop, With a Kolkata Traffic Signal All this can feel a bit abstract, so let us make it concrete with something every Kolkata resident understands very well — crossing a busy road. Picture Soham standing at a signal with his four-year-old son, Kit, holding his hand. It is a typical Kolkata crossing — buses, yellow taxis, and autos, and the signal, while present, is not always fully obeyed. Sometimes there is a traffic constable standing in the middle of the road, waving vehicles through by hand, overriding the signal completely. Step one — model the world as an object (this is the OOAD part). In Object-Oriented Analysis and Design, we are trained to represent a real-world situation as a class with clearly named fields. Here, the "world" Soham is observing can be written as one simple record: record RoadSituation(boolean signalGreen, boolean trafficClear, boolean handHeld, boolean policeOnDuty) { } Step two — define the goal. The goal is not "the signal is green." The goal is "Soham and Kit have reached the other side, safely." Reaching a green signal is only useful if it actually leads there. Step three — define the actions, each with its precondition and effect (this is the GOAP part). Notice that none of these actions know about each other. Each one only knows what it needs and what it produces: holdChildHand – produces handHeld = true. This is usually the very first thing a responsible parent does, well before even looking at the signal.waitForSafeSignal – needs the current RoadSituation, and produces signalGreen = true once either the light turns green or a traffic constable is present and waving pedestrians across.confirmTrafficClear – because a green signal in Kolkata does not always mean an auto will not sneak through, this action looks both ways and produces trafficClear = true.crossTheRoad (the @AchievesGoal action) – only runs once handHeld, trafficClear, and (signalGreen or policeOnDuty) are all true. Step four — let the planner loop. This is the actual "GOAP + OOAD loop": the planner looks at the current RoadSituation object, picks whichever action's precondition is already satisfied and whose effect moves the world closer to the goal, executes it, updates the RoadSituation, and checks again if the goal is reached. It keeps looping — plan, act, update state, re-check — until crossTheRoad finally fires. The beautiful part is what happens on a day when the signal itself is not working — a fairly common event in Kolkata, especially during a power cut or during Puja season when the police fully take charge of a crossing. If tomorrow you add one more action, say waitForPoliceWave, which also produces a "safe to proceed" fact, the planner will simply discover this new path on its own the next time it runs. Nobody needs to redraw anything, because nobody drew anything explicit in the first place. The Same Scenario as Embabel Code Here is a simplified sketch of the above scenario, written in the same style as the retirement planner. Treat it as a teaching example rather than a compiled, production-ready class: Java /** * @author Soham Sengupta * @since 2026-09-13 * @description A small agent that plans how Soham and his four-year-old * son Kit can safely cross a busy Kolkata road. Written purely to show * how Embabel's GOAP-style planner reasons over typed domain objects * (OOAD) to reach a goal, without the developer wiring the order by hand. */ @Agent(name = "StreetCrossingAgent", description = "Plans a safe road crossing for a parent and a young child.") public class StreetCrossingAgent { record RoadSituation(boolean signalGreen, boolean trafficClear, boolean handHeld, boolean policeOnDuty) { } record CrossingPlan(String narrative) { } @Action(description = "Hold Kit's hand before anything else - the first rule of the road.") public RoadSituation holdChildHand(UserInput userInput) { return new RoadSituation(false, false, true, false); } @Action(description = "Wait till the signal turns green, or till a traffic constable waves pedestrians across.") public RoadSituation waitForSafeSignal(RoadSituation situation, OperationContext context) { boolean safeToMove = situation.signalGreen() || situation.policeOnDuty(); return new RoadSituation(safeToMove, situation.trafficClear(), situation.handHeld(), situation.policeOnDuty()); } @Action(description = "Look right, then left, then right again - the signal alone is not a guarantee in Kolkata traffic.") public RoadSituation confirmTrafficClear(RoadSituation situation) { return new RoadSituation(situation.signalGreen(), true, situation.handHeld(), situation.policeOnDuty()); } @Action(description = "Cross only when hand is held, the way is clear, and the signal or constable allows it.") @AchievesGoal(description = "Soham and Kit have reached the other side of the road safely.") public CrossingPlan crossTheRoad(RoadSituation situation) { return new CrossingPlan( "Soham held Kit's hand tight, waited for the green man, checked both sides once more, then crossed together."); } } Now compare this with how the same scenario would look in LangGraph4j's philosophy — as an explicit graph you draw yourself, branches and all: Java package com.example.demo; import org.bsc.langgraph4j.CompiledGraph; import org.bsc.langgraph4j.StateGraph; import org.bsc.langgraph4j.action.AsyncEdgeAction; import org.bsc.langgraph4j.state.AgentState; import java.util.Map; import java.util.Optional; import static org.bsc.langgraph4j.StateGraph.END; import static org.bsc.langgraph4j.StateGraph.START; import static org.bsc.langgraph4j.action.AsyncNodeAction.node_async; public class StreetCrossingGraph { // Stand-ins for a real signal sensor and a quick look both ways. private static boolean checkSignal() { return true; } private static boolean lookBothWays() { return true; } public static void main(String[] args) throws Exception { StateGraph<AgentState> graph = new StateGraph<>(AgentState::new) .addNode("holdHand", node_async(state -> Map.of("handHeld", true))) .addNode("waitForSignal", node_async(state -> { boolean policeOnDuty = state.<Boolean>value("policeOnDuty").orElse(false); return Map.of("signalGreen", checkSignal() || policeOnDuty); })) .addNode("checkTraffic", node_async(state -> Map.of("trafficClear", lookBothWays()))) .addNode("cross", node_async(state -> Map.of("crossed", true))) .addEdge(START, "holdHand") .addEdge("holdHand", "waitForSignal") .addConditionalEdges("waitForSignal", AsyncEdgeAction.edge_async(state -> state.<Boolean>value("signalGreen").orElse(false) ? "proceed" : "retry"), Map.of("proceed", "checkTraffic", "retry", "waitForSignal")) .addConditionalEdges("checkTraffic", AsyncEdgeAction.edge_async(state -> state.<Boolean>value("trafficClear").orElse(false) ? "proceed" : "retry"), Map.of("proceed", "cross", "retry", "checkTraffic")) .addEdge("cross", END); CompiledGraph<AgentState> workflow = graph.compile(); Optional<AgentState> result = workflow.invoke(Map.of("policeOnDuty", false)); result.ifPresent(state -> System.out.println(state.data())); } } Notice the two addConditionalEdges calls — this is how LangGraph4j handles a branch: an edge action returns a label ("proceed" or "retry"), and a small map resolves that label to the actual next node. This is exactly the graph the developer must draw by hand, action by action and branch by branch, for a scenario that Embabel's planner worked out on its own. As a flowchart, that graph looks like this: Both pieces of code reach the same goal. But in Embabel, nobody drew this flowchart — the planner found it. In LangGraph4j, this flowchart is the code. If a new real-world case turns up tomorrow, Embabel's planner can absorb it automatically as long as the new action's types fit; LangGraph4j needs a human to open the graph and add a new node or edge. Comparing the Two Philosophies Aspect Embabel LangGraph4j Core idea Give a goal and typed actions; a GOAP/A* planner works out the order Developer explicitly wires nodes and edges into a graph Mental model "What do I want, and what building blocks do I have?" "What are my steps, and how do they branch?" Control flow Discovered at runtime by the planner Declared upfront by the developer Role of the LLM Used only inside actions, never for deciding sequence Can be used inside nodes; sequence is still fixed by the graph Adapting to a new case Often automatic, if a new action's types fit the gap Needs a human to add a new node or edge Tracing "why this order" Needs the planner's own logging/tooling to see the chosen path Very direct — the graph is already the flowchart Roots Kotlin-first, Java-friendly, close to Spring A faithful Java port of Python's LangGraph, works with Langchain4j and Spring AI Maturity (as of late 2026) Young, pre-1.0, moving fast Older and more widely adopted, with a large existing community When to Reach for Which Reach for Embabel when: Your goal is clear, but the exact path to it can honestly vary depending on the situation.You want a deterministic, non-LLM planner deciding the order, not the LLM guessing it.You are already deep in the Spring ecosystem and like strongly typed domain models.New cases keep appearing over time, and you would rather add one new action than redraw a graph. Reach for LangGraph4j when: You already know the exact stages of your workflow — say, a well-understood pipeline of retrieve, rerank, generate, and validate.You want the flow to be visible as an actual graph, easy to explain to a non-technical stakeholder.Your team is porting an existing Python LangGraph pipeline and wants the Java version to mirror it closely.You value a larger, more mature community with more examples to learn from, at least for now. Neither approach is "better" in an absolute sense. Embabel bets on planning; LangGraph4j bets on explicitness. Pick the one that matches how well you actually know your workflow in advance. To Conclude: Java Still Has a Say in Enterprise AI Python remains, without question, the home of AI research — the notebooks, the training loops, the enormous ecosystem of machine learning libraries were built there first, and will likely stay there. Nobody sensible is arguing Java should train the next large language model. But training a model is only one part of the story. The other part — the much bigger part, in terms of sheer lines of code running in the real world — is taking an already-trained model and safely wiring it into systems that already exist: a bank's core banking platform, an insurance company's policy engine, a hospital's records system. The overwhelming majority of that existing code, in most large enterprises, is written in Java and Spring, not Python. That is Java's home ground, built up over more than two decades. This is exactly the ground both Embabel and LangGraph4j are fighting on. Java also tends to run this kind of orchestration work faster than Python at execution time, which matters once you are calling these agents thousands of times a day inside a live enterprise system. And when your applications are already written in Java, keeping the AI layer in Java too — rather than routing every call out to a separate Python service — often turns out to be the simpler, safer choice. So, the real contest in enterprise AI is perhaps not "who trains the smarter model" — Python wins that one comfortably. It is "who can be trusted to make that model's decisions reliably inside a bank's core system, an insurance engine, or a hospital record system." That is precisely the kind of trust Java has spent two decades earning. With Rod Johnson effectively writing a sequel to his own Spring story, and with LangGraph4j bringing a proven Python pattern faithfully onto the JVM, Java is not sitting out this wave of AI. It has simply chosen to fight the battle it already knows how to win. This piece focused on Embabel's goal-and-planner philosophy. Embabel also has other ideas worth a separate deep-dive later — like its approach to agentic search and enterprise memory. A hands-on, step-by-step guide to setting up Embabel from scratch will follow as a companion piece. Here's the link to the source code: https://github.com/trainerpb/embabel-hello-world/tree/feature/revision.

By Soham Sengupta

Java

DZone's Featured Java Resources

Monthly Top Java Experts

The Latest Java Topics