Data

Apache IoTDB is well-suited for environments where time-series data from connected devices and industrial systems is generated continuously and requires efficient querying. Typical use cases include monitoring temperature, pressure, vibration, energy usage, machine status, and device telemetry. This approach also applies to manufacturing, smart infrastructure, fleet monitoring, utilities, and edge computing. IoTDB stands out for its focus on large-scale time-series workloads from devices and industrial systems. It is purpose-built for high-frequency data ingestion, historical analysis, and time-based queries. This makes it ideal for applications that require insight into both the current state and historical trends of devices or processes. Why Apache IoTDB Matters Apache IoTDB excels at managing large volumes of device-generated data over time. It addresses issues beyond storage, such as continuous data ingestion, efficient organization by device and timestamp, and fast queries for both recent and historical values. This makes IoTDB well-suited for systems with temporal, device-oriented data models. Common use cases include industrial IoT, smart factories, energy monitoring, connected vehicles, building automation, predictive maintenance, and edge computing. For example, manufacturers can track machinery data, utilities can analyze energy consumption across many meters, and fleet operators can monitor location and engine telemetry. In each scenario, IoTDB enables applications to determine current status, review device behavior over time, and detect measurements outside expected ranges. Hands-On: Java With Apache IoTDB We will build a simple Java application using Apache IoTDB. For simplicity, IoTDB will run locally in Docker. Shell docker run -d \ --name iotdb-instance \ -p 6667:6667 \ -e dn_rpc_address=0.0.0.0 \ apache/iotdb:2.0.11-standalone Next, create the database using IoTDB’s Table SQL dialect: Shell docker exec -it iotdb-instance \ /iotdb/sbin/start-cli.sh \ -h 127.0.0.1 \ -p 6667 \ -u root \ -pw root \ -sql_dialect table \ -e "CREATE DATABASE IF NOT EXISTS jnosql" For this local example, we use the default root/root credentials and disable client redirection. These settings are suitable for single-node development only. In production, use secure credentials and configure redirection based on your IoTDB cluster topology. Configuring the Application Next, configure the connection in microprofile-config.properties: Properties files jnosql.timeseries.database=jnosql jnosql.iotdb.host=localhost jnosql.iotdb.port=6667 jnosql.iotdb.username=root jnosql.iotdb.password=root jnosql.iotdb.enable.redirection=false Eclipse JNoSQL is built on Jakarta APIs such as CDI, JSON-B, and Eclipse MicroProfile Config. These APIs are supported by popular runtimes including Helidon, Quarkus, Open Liberty, and other Jakarta EE-compatible environments. Then add the Apache IoTDB driver: XML <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-iotdb</artifactId> <version>${jnosql.version}</version> </dependency> Modeling Sensor Data A sensor-oriented model aligns well with IoTDB: Java @Entity public class SensorReading { @Id private Instant id; @Column private String sensor; @Column private double temperature; @Column private double humidity; // constructors, getters, and setters } The Instant field records the timestamp of the reading. The other fields capture the measurements at that time. The entity can also be exposed through Jakarta Data: Java @Repository public interface SensorReadingRepository extends BasicRepository<SensorReading, Instant> { List<SensorReading> findBySensorOrderByIdDesc( String sensor, Limit limit); } Using TimeSeriesTemplate With the infrastructure in place, insert several readings and retrieve both the latest value and recent history: Java "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { TimeSeriesTemplate template = container.select(TimeSeriesTemplate.class).get(); template.insert(firstReading); template.insert(secondReading); template.insert(latestReading); var currentReading = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .limit(1) .singleResult(); System.out.println( "Current sensor reading: " + currentReading ); var history = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .skip(1) .limit(10) .result(); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } The first query answers a common IoT question: what is the latest reading from this sensor? The second retrieves its recent history, excluding the latest observation. Using Jakarta Data The same use case can be implemented using the repository: Java public class App2 { public static void main(String[] args) { var firstReading = new SensorReading( Instant.parse("2026-09-20T08:00:00Z"), "sensor-01", 21.4, 45.0 ); var secondReading = new SensorReading( Instant.parse("2026-09-20T09:00:00Z"), "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { SensorReadingRepository repository = container.select(SensorReadingRepository.class).get(); repository.save(firstReading); repository.save(secondReading); repository.save(latestReading); var currentReading = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.of(1) ) .stream() .findFirst(); System.out.println( "Current sensor reading: " + currentReading ); var history = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.range(2, 10) ); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } Both approaches model the domain in terms of latest state and historical observations. This is where a time-series database like Apache IoTDB is more effective than using a general-purpose database with timestamps as regular fields. Why Apache IoTDB Is More Than Sensor Storage While Apache IoTDB is often associated with IoT, its capabilities go well beyond basic sensor data storage. It excels when organizations must manage large volumes of device-centric data over time while retaining relationships among measurements, devices, and timestamps. As a result, IoTDB is valuable for industrial systems, utilities, transportation, smart infrastructure, manufacturing, and edge computing. For example, in predictive maintenance, a machine may continuously report vibration, temperature, pressure, energy consumption, and operating condition. The true value is not only found in the latest readings, but in studying how these measurements change together over time. Maintenance systems may compare recent data with historical trends, spot anomalies before failures, or correlate changes across multiple devices. This requires more than simple sensor data storage. This approach also benefits energy systems, where smart meters, solar panels, batteries, and grid equipment generate continuous measurement streams that require time-based analysis. In transportation, vehicles produce ongoing data such as speed, location, fuel consumption, battery status, and engine telemetry. Similarly, smart buildings rely on time-series data from HVAC systems, occupancy sensors, and energy meters. IoTDB is especially effective when system architecture is organized around devices and their measurements. It can serve as the historical data layer for operational systems, while other technologies manage transactional or business information. For example, a relational database might store details about a machine, its owner, contracts, or maintenance schedules, while IoTDB manages the extensive measurement data the machine generates. This separation lets enterprise applications treat telemetry as a primary workload, rather than forcing high-frequency device data into models meant for business entities and transactions. Conclusion Apache IoTDB is a strong fit for applications that need to ingest and analyze device-generated data over time, especially in IoT, industrial, energy, and edge scenarios. With Eclipse JNoSQL 1.1.18, Java developers can access these capabilities through familiar Jakarta APIs, keeping the application model consistent while still leveraging IoTDB’s time-series specialization.

By Otavio Santana CORE

OpenSearch is an open-source, distributed search and analytics suite derived as a fork of Elasticsearch and maintained under the Apache 2.0 license. When it comes to memory configuration, the guidance is often reduced to a few rules of thumb: swapoff -a, vm.swappiness=1, or bootstrap.memory_lock, and allocating 50% of available memory to the JVM heap while leaving the rest for Lucene and the filesystem page cache, OpenSearch off-heap caches, network buffers, and other system needs. These recommendations are repeated throughout documentation, blog posts, and operational guides, yet their origins and the mechanisms that justify these specific values are rarely examined. Undoubtedly, they provide a reasonable and safe starting point or a safe upper bound in most of the cases, but a safe default is not necessarily an optimal configuration. All of this raises even more questions. How do these defaults affect cluster performance? What is the optimal JVM heap ratio? Does memory given up by the JVM actually become filesystem page cache, and at what point does that trade-off stop paying off? How do read/write latency correlate with the heap ratio? These questions become particularly important in resource-constrained environments and in the cloud, where long-term contracts may make existing instances significantly cheaper, making horizontal or vertical scaling a difficult decision. In this article, we'll try to answer these questions through benchmarking. This is Act 1 of a two-act series. Act 1 focuses on identifying the cause of the latency problems we observed with the current defaults. Act 2 will explore what other heap-ratio values might look like for read/write loads. Along the way, I'll share the tools and commands used throughout the investigation, making this article a practical reference as well for you and for myself when I inevitably need to retrace the investigation months later. Knowledge Context The story also crosses several boundaries, such as the Kernel VM, the JVM, and Lucene. So, it’s important to outline the concepts mentioned in this part of the article beforehand, both for the context and, optionally, to enrich the AI context if you'd like to summarize everything. AreaWhere memory livesWhy it matters hereLinux page cacheFile-backed RAMLucene relies heavily on it for index data; under memory pressure, these pages can be reclaimed and read again later.Linux swapDisk-backed anonymous memoryAnonymous process memory can be swapped out under pressure. vm.swappiness influences this decision but does not prohibit it.Linux PSIKernel pressure signalShows time tasks spend stalled due to CPU, memory, or I/O pressure. We'll use I/O PSI while investigating latency.JVM heapAnonymous memoryControlled by Xms/Xmx; contains Java objects and several OpenSearch data structures.JVM native memoryAnonymous/file-backed memory outside XmxIncludes code cache, metaspace, stacks, direct buffers, and native allocations. Heap metrics do not account for all of it.OpenSearch cachesHeap/off-heap, depending on cacheTheir sizes may depend on heap size, which becomes important when we change jvm_heap_ratio.OpenSearch indexing bufferHeapIts size depends on heap and therefore becomes an important variable in Act 2.Lucene mmapFile-backed/page cacheLucene index files mapped into the process do not consume JVM heap; resident pages compete for physical RAM. Environment I used Aiven for OpenSearch on Azure, with cluster metrics exported to Thanos. The OpenSearch Benchmark metrics don't provide everything we need to answer our questions, particularly host-level metrics such as Linux PSI and swap activity. Exporting the cluster metrics to Thanos allows us to use PromQL queries later to retrieve the additional metrics needed for the investigation. The cluster consists of 3 nodes: CPUAMD EPYC 7763v (Milan)vCPU / RAM2 vCPU, 8 GiBDisk Size175 GiB per nodeAzure Regionazure-westeuropeAzure DiskPremiumV2_LRSAzure SKUStandard_D2as_v5 OpenSearch Version3.6.0JDKjava-21-openjdk-headlessGCG1GC Act 1. The Latency and an Extra GB The symptom: elevated query latency across the cluster, first reported by the customer after a kernel and Azure image upgrade. The load pattern on OpenSearch itself remained unchanged, as did the cluster configuration and settings. The monitoring panels give us the first clue. In the screenshots below, the green vertical line marks the moment of the upgrade. After that point, the page cache grows by roughly a gigabyte, while I/O PSI, previously close to zero, starts showing significant spikes. Nothing crashed, no alert fired, and from the JVM point of view everything looked normal. So where did that extra gigabyte of page cache come from? Nothing was actually freed. It moved. The interesting part isn't just that memory went to swap; it's which memory. When you have thousands of running clusters, there is always a small fraction of them operating close to the edge: relatively stable, yet sensitive enough that even a small change can noticeably affect performance. Like a star nearing the end of its lifetime, they may look stable right up until something disturbs the balance. The immediate cause of the page-cache change was identified fairly quickly: Azure applies tuning parameters that differ from the Linux kernel defaults, and those parameters were not applied by the older image. Once applied, the larger buffers and read_ahead increased the filesystem cache footprint, putting additional pressure on anonymous memory and eventually pushing some of it to swap. But rather than stopping there, let's use this incident as an opportunity to experiment with the heap ratio and make the behavior of OpenSearch instances explicit and less dependent on such environmental changes. Evidence It Is on Swap; None of It Locked First, find what is going on on a node itself: Shell PID=$(pgrep -f 'org.opensearch.bootstrap.OpenSearch') grep -E 'VmRSS|RssAnon|RssFile|RssShmem|VmSwap|VmLck' /proc/$PID/status Shell VmRSS: 4366408 kB # resident RssAnon: 3329896 kB # heap + anonymous native RssFile: 1036496 kB # resident mmap'd Lucene pages RssShmem: 16 kB VmSwap: 2679496 kB # on swap VmLck: 0 kB # bootstrap.memory_lock=fasle, none locked Shell grep -E 'MemFree|MemAvailable|Cached|SwapFree' /proc/meminfo Shell MemFree: 258216 kB MemAvailable: 3479264 kB Cached: 3403196 kB SwapCached: 854396 kB SwapFree: 4745444 kB The Swap Device Is dm-crypt Then confirm there is somewhere for it to go, and on what kind of device: Shell swapon --show Plain Text NAME TYPE SIZE USED PRIO /dev/dm-4 partition 8G 3.5G -1 The dm-* swap device is the interesting detail to catch and to keep in mind. This is an encrypted device, so once a page is requested it could drive more I/O -> more dm-crypt allocations -> more high-order pressure. A self-reinforcing loop and a good example of read amplification. Paging Is Live, Not Historical The next logical step is to check whether it is live paging or just a stale historical tail. The vmstat 1 5 the Linux Virtual Memory Statistics Tool should give us an exact answer for this, where non-zero swap blocks in and out (marked as si, so): Shell vmstat 1 5 Plain Text procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu------- r b swpd free buff cache si so bi bo in cs us sy id wa st gu 1 0 3620928 159716 5864 3704212 113 80 3361 577 3147 12 8 7 84 1 0 0 0 0 3625240 189036 5860 3705260 0 5228 24268 5228 4841 4072 14 12 69 4 0 0 0 0 3625240 166196 5860 3710696 0 0 32 0 3572 2556 21 5 74 0 0 0 0 0 3625240 162668 5860 3716200 8 0 164 0 3729 2619 20 7 73 0 0 0 12 0 3625312 147692 5860 3724288 0 140 9207 524 3679 4024 22 14 63 1 0 0 Non-zero si/so in 4 of 5 samples show the live paging process, swpd also climbing across five seconds, which is good proof. The Swapped Pages Are Anonymous, Not File-Backed Let's also check swap memory consumption for each of the process's mappings, to confirm that swap is heap-related: Shell PID=$(pgrep -f org.opensearch.bootstrap.OpenSearch) awk '/^[0-9a-f]/{h=$0} /^Swap:/{if($2>0)print $2" kB "h}' /proc/$PID/smaps | sort -rn | head -20 Plain Text 1160380 kB 708400000-7ffe00000 rw-p 00000000 00:00 0 62296 kB 7f30c0000000-7f30c3f4b000 rw-p 00000000 00:00 0 61452 kB 7f1fc8000000-7f1fcbc03000 rw-p 00000000 00:00 0 61448 kB 7f1fa8000000-7f1fabc02000 rw-p 00000000 00:00 0 61444 kB 7f2f38000000-7f2f3bc01000 rw-p 00000000 00:00 0 61444 kB 7f24b4000000-7f24b7c01000 rw-p 00000000 00:00 0 61444 kB 7f22ac000000-7f22afc01000 rw-p 00000000 00:00 0 61444 kB 7f2238000000-7f223bc01000 rw-p 00000000 00:00 0 61444 kB 7f216c000000-7f216fc01000 rw-p 00000000 00:00 0 61444 kB 7f2168000000-7f216bc01000 rw-p 00000000 00:00 0 61444 kB 7f2148000000-7f214bc01000 rw-p 00000000 00:00 0 61444 kB 7f1fec000000-7f1fefc01000 rw-p 00000000 00:00 0 61444 kB 7f1fe8000000-7f1febc01000 rw-p 00000000 00:00 0 61444 kB 7f1fcc000000-7f1fcfc01000 rw-p 00000000 00:00 0 61444 kB 7f1fac000000-7f1fafc01000 rw-p 00000000 00:00 0 60772 kB 7f1fc0000000-7f1fc3c14000 rw-p 00000000 00:00 0 59952 kB 7f311e000000-7f3123f41000 rw-p 00000000 00:00 0 59340 kB 7f30b4000000-7f30b7c9b000 rw-p 00000000 00:00 0 57180 kB 7f3110000000-7f3113e3e000 rw-p 00000000 00:00 0 43084 kB 7f3130800000-7f3133d70000 rwxp 00000000 00:00 0 The Largest Swapped Region Is But why is what we are seeing above a heap-related area? There are a few clues for that. The region size 0x708400000 - 0x7ffe00000 is exactly 4,154,458,112 bytes = 3,962 MiB as we use -Xms == -Xmx and the whole thing is committed at startup, and nothing else in a JVM process is a single contiguous ~4 GB anonymous rw-p mapping. Second, It's the lowest mapping in the address space, smaps_rollup [rollup] line starts at exactly 708400000: Shell cat /proc/$PID/smaps_rollup Shell 708400000-7ffd3bb56000 ---p 00000000 00:00 0 [rollup] Private_Dirty: 3213788 kB Swap: 2680532 kB SwapPss: 2679412 kB Locked: 0 kB The JIT Code Cache Is Swapped Too Decoding the top swapped regions: 1160380 kB at 708400000 – is the JVM heap, the compressed‑oops heap base and matches the [rollup] start from smaps_rollup.The dozens of 61444 kB regions – these areas are probably related to native/off‑heap: Netty, JNI, Lucene native, etc.43084 kB marked rwxp – the JIT code cache, also swapped out, a bad sign. Together, these regions account for almost exactly the ~2.5 GB of swapped memory we observed earlier: cold heap regions, native/off-heap allocations, code cache, and possibly thread stacks. Practically, this means two consequences: GC can amplify swap latency. 1 GB of the JVM heap was swapped out. G1 does not necessarily touch all of those pages during a mixed collection, but any GC phase that accesses a swapped page incurs a major fault and has to bring it back through the dm-crypt device. Hence, short GC work can produce significantly longer pauses.A swapped page can also contain executable code. The next call into a swapped-out compiled method can trigger a major fault before the code can run. The resulting latency may land on an otherwise random request and be difficult to attribute directly to GC, index I/O, or the query itself. Evidence of Sustained Anon Memory Churn workingset_refault_anon counts anonymous memory refault events after reclaim; it does not count unique pages. Together with pswpin and pswpout, it shows how much anonymous memory paging has accumulated since boot. Shell grep -E 'workingset_(refault|activate)_anon|pswpin|pswpout' /proc/vmstat Plain Text workingset_refault_anon 89061837 workingset_activate_anon 1856069 pswpin 86690374 pswpout 61618603 These counters are cumulative, so fetching them at 10-minute intervals clearly shows that this wasn't a one-time eviction. There was sustained process: nearly 89 million anonymous pages were repeatedly swapped out and faulted back in. vm.swappiness = 1 and GC Amplification In this story vm.swappiness=1 was set since the cluster's inception. It does what Linux defines it to do, but it doesn't provide the protection we wanted. I suspect this matters particularly in the most resource-constrained deployments. How do we know that? The entire result above is a counterexample. swappiness biases the kernel's choice between reclaiming file-backed and anonymous pages. It does not prevent anonymous pages from being swapped out. Even at 1, this can still happen under sustained memory pressure. On a resource-constrained node whose index is several times larger than its RAM, pressure on the filesystem cache is not an exceptional condition; this is the normal operating state. This has two important consequences: It is not self-healing. Nothing proactively pages anonymous memory back in on a schedule. A swapped-out page returns to RAM only when it is accessed again, and cold memory, as it's defined, may remain untouched for a long time. As a result, the cold JVM memory can remain in swap indefinitely.It may remain invisible until something touches it. At steady state, a cold tail of the heap can remain in swap without producing obvious symptoms. The problem becomes visible when those pages are touched again, causing major page faults and potentially amplifying GC and request latency. Key Takeaways So, the root cause of the latency problems is an oversized heap combined with page cache pressure (the cold heap tail has been swapped out). vm.swappiness=1 did not protect the heap. It biases what gets reclaimed; it does not prevent anonymous memory from being swapped out.vm.swappiness=1 should not be relied on with the other defaults in production. An oversized heap can lead to GC amplification that is difficult to detect.Most of the swapped-out memory wasn't heap at all, but malloc arenas and the JIT code cache, none of it inside Xmx, so heap metrics didn't show it.Swap on dm-crypt exacerbates the issue, resulting in longer GC pauses and random request latency spikes. dm-crypt may be unavoidable in production due to security requirements. The fix isn't another swappiness tweak. It's two things: stop committing heap you don't use, and make the heap you do commit non-evictable. Follow Up In Act 2, we'll answer the remaining questions raised at the beginning of this article and look more closely at the trade-off introduced by bootstrap.memory_lock. This setting makes the heap resident and swap-immune, but it also turns jvm_heap_ratio from a soft default into a permanent memory commitment. The question then becomes: what heap ratio best suits different read and write workloads? There is one more complication to mention in advance: in a resource-constrained environment, merge storms can distort benchmark results, making an otherwise good heap ratio appear poor. See the screenshot below.

Platform migrations are not only technology problems. They're visibility problems as well. Organizations often know what platforms they operate and who uses them. What they typically do not know is what runs on those platforms. Which workflows are business critical?Which workflows can be migrated with minimal effort?Which workflows require redesign?Which capabilities have no equivalent in a target platform? Without answers to these questions, migration planning becomes guesswork. During a large enterprise analytics modernization initiative, I helped evaluate an established platform environment that had evolved across multiple teams and environments over time. The challenge was not merely collecting infrastructure information. It was understanding workflow behavior at scale. What followed was the development of a workflow introspection platform that analyzed workflow definitions, categorized capabilities, and evaluated migration readiness. The most interesting outcome wasn't the migration itself. It was the framework that made migration planning more structured, reviewable, and evidence-based. The Visibility Problem in Analytics Platforms Most analytics platforms expose administrative metadata. This metadata can typically answer questions such as: Which workflows exist?Who owns them?When were they last modified? What it cannot usually answer is: What does this workflow do?Which capabilities does it use?How difficult would it be to migrate?What dependencies exist across workflows? This information typically resides within workflow definitions themselves. The management layer knows the workflow exists. The workflow definition explains how it operates. The challenge is connecting those two layers. To answer meaningful migration questions, we needed to move beyond administrative metadata and inspect workflow logic directly, subject to authorization, access controls, confidentiality objections, and security requirements. Building a Workflow Discovery Pipeline The first challenge involved collecting workflow artifacts at scale. Authorized workflow definitions were gathered and standardized into a common format that could be analyzed consistently across environments. Collection and storage should minimize or exclude embedded credentials, personal information, confidential business logic, and other sensitive content not required for the assessment. As workflows were collected, differences in metadata structures, naming conventions, and ownership information became apparent. To address this, the discovery process incorporated validation, normalization, and quality checks designed to improve consistency while preserving source provenance and documenting unresolved data quality issues. Once the collection process stabilized, workflow artifacts could be prepared for deeper analysis. Turning Workflow Files into Structured Data The next challenge was understanding what those workflows contained. Most workflow definitions were stored as structured configuration files. A simplified structure resembled: XML <Workflow> <Nodes> <Node Tool="Input" /> <Node Tool="Join" /> <Node Tool="Output" /> </Nodes> </Workflow> While simple on the surface, real-world workflows contained significantly more complexity. Workflows included: Data preparation logicTransformationsAnalytical operationsBusiness rulesAutomation componentsPlatform-specific functionality The parser extracted only the metadata needed for the migration assessment and transformed it into a structured profile that could be analyzed systematically. Production implementations should apply data minimization, retention limits, access controls, and audit logging to these profiles. For each workflow, we generated: Java public class WorkflowProfile { private String workflowId; private int toolCount; private List<String> tools; private List<String> dependencies; private ComplexityScore complexity; } This transformed workflow definitions into structured datasets that can be queried. Instead of analyzing workflow files individually, we could evaluate usage patterns across the environment. The Tool Parity Matrix Extracting workflow information was useful. It still didn't answer the most important question: Can this workflow be migrated? To solve this problem, I built what became the core component of the migration readiness engine: the tool parity matrix. The concept was simple. Each source platform capability was mapped to its nearest apparent equivalent in a target platform for the defined use case. Functional mapping was a starting point, not a substitute for validating behavior, performance, security, privacy, licensing, resilience, support, and regulatory requirements. Each mapping received two attributes. Parity Level Java enum ParityLevel { FULL, PARTIAL, NONE } Migration Effort Java enum MigrationEffort { LOW, MEDIUM, HIGH } The goal was not perfect accuracy. The goal was consistent decision support, with sourcing assumptions documented, calibrated, and subject to expert review. Calculating Migration Readiness Once tool mappings existed, preliminary migration scoring became possible. The resulting score was a prioritization aid rather than a determination that a workflow could be migrated safely or successfully. Workflows composed primarily of supported capabilities received higher readiness scores. Workflows containing specialized functionality received lower readiness scores and required additional evaluation. Conceptually: Python def migration_readiness(workflow): total_score = 0 for tool in workflow.tools: total_score += parity_score(tool) return total_score / len(workflow.tools) Similarly, relative effort could be estimated using weighted migration effort values. Translating those values into time or cost requires calibrated weights, historical data, documented assumptions, and validation for the relevant environment. This enabled workflows to be grouped into preliminary migration waves for engineering and stakeholder review. For example: Readiness / effort profileOutcomeHigh readiness / low effortMigration wave 1Medium readiness / medium effortMigration wave 2Lower readiness / higher effortAdditional review required What previously required largely subjective evaluation could now be informed by consistent, documented criteria, while retaining expert review for exceptions and consequential decisions. Discovering Hidden Platform Dependencies One unexpected benefit of the readiness engine was dependency discovery. As workflows were analyzed collectively, patterns began to emerge. Certain capabilities appeared repeatedly. Some workflow categories were straightforward to migrate. Others consistently required additional review because they relied on specialized functionality. Most importantly, platform-wide analysis revealed concentrations of functionality that might otherwise remain invisible until migration execution began. By surfacing these patterns early, architectural decisions could be made proactively rather than reactively, which can improve planning accuracy and help identify migration risks before execution. Building the Consolidation Layer Large platform environments rarely operate with perfectly consistent metadata. Different environments often use: Different naming conventionsDifferent ownership modelsDifferent deployment practicesDifferent classification standards To address this, a consolidation pipeline normalized workflow metadata into a unified model. The process included: Discovery → Normalization → Deduplication → Classification → Migration scoring Once normalized, workflows could be analyzed collectively rather than as isolated artifacts. This supported environment-wide reporting and migration planning, subject to the permitted use of the underlying data. Connecting Technical Data to Organizational Data One lesson from large migrations is that technical readiness alone is insufficient. Organizations also need to understand workload ownership. The readiness engine therefore connected workflow metadata to authorized organizational ownership information. Access to ownership data should be limited to legitimate planning purposes and handled in accordance with applicable privacy, employment, and records-management requirements. This allowed migration plans to answer both technical and operational questions. Instead of saying "These workflows require additional review," we could identify which groups owned those workflows and engage stakeholders earlier in the planning process. Technical analysis became actionable. Lessons Learned 1. Migration Planning Is a Data Problem Most migration programs begin with meetings. They should begin with visibility. Without understanding platform usage, migration planning becomes speculation. 2. Metadata Is Not Enough Administrative information provides useful context. It rarely provides sufficient context. Meaningful migration planning requires understanding workflow behavior. 3. Scoring Enables Scale Humans can review a limited number of workflows. Large environments require consistent evaluation frameworks. Well-designed scoring systems can support prioritization and repeatability, but they require representative inputs, documented limitations, monitoring, and expert review. Final Thoughts Many organizations approach platform migrations as technology replacement exercises. They are discovery exercises. Before deciding where workloads should move, you first need to understand what those workloads do. The migration readiness engine addressed part of that problem by combining workflow discovery, workflow analysis, dependency identification, and migration scoring into a repeatable planning framework. The result was not simply a migration plan. It was a structured approach for understanding complex analytics ecosystems and supporting migration decisions with better evidence, documented assumptions, and appropriate review. Disclaimer: This article represents my personal views and is not written on behalf of, endorsed by, or intended to represent the views of my employer. The architecture, code, scoring methods, and examples are simplified and illustrative and should be validated for the relevant technical, security, privacy, legal, and regulatory environment. This article is provided for general informational purposes and does not constitute legal, technical, or other professional advice.

Hey Mates! “How does a map work in Go?” is one of my favorite interview questions. It sounds simple, but it opens up a conversation about hashing, collisions, memory layout, and why two implementations with the same average O(1) lookup complexity can behave quite differently. With Go 1.24, that conversation got more interesting: the map implementation switched to Swiss Tables. Let’s walk through how the old implementation worked, what changed, and how the new design finds your keys. TL;DR Go 1.24, released in February 2025, completely replaced the internal implementation of the built-in `map` with a design based on Google's Swiss Tables. The syntax and the behavior guaranteed by the Go specification did not change, so existing programs required no migration. In microbenchmarks, some map operations became up to 60% faster. Across the Go team's full-application benchmark suite, geometric mean CPU time improved by about 1.5%. Datadog reported roughly 70% less memory for one exceptionally large map—an impressive case study, not a universal promise. Why Replace map at All? map is one of the most frequently used data structures in Go. It appears in caches, configuration, indexes, and every `map[string]any` we would rather not discuss. The old implementation served Go well for more than a decade. Hash-table research did not stop, however. At CppCon 2017, Google engineers Sam Benzaquen, Alkis Evlogimenos, Matt Kulukundis, and Roman Perepelitsa presented a new cache-friendly design. It became known as Swiss Tables, after the Google Zürich office where the team worked. In 2018, Google released an implementation in the C++ Abseil library. The design then spread across ecosystems: C++: absl::flat_hash_map in Abseil;Rust: the standard HashMap is built on hashbrown, a Swiss Tables implementation, since Rust 1.36;Go: first through third-party packages such as dolthub/swiss and cockroachdb/swiss, then in the built-in map starting with Go 1.24. The route into Go was collaborative. Community members built early prototypes. Peter Mattis of CockroachDB combined those ideas with solutions for Go-specific requirements in cockroachdb/swiss. The Go 1.24 runtime implementation is heavily based on that work. Before we get to the “Swiss” part, let us quickly review hash tables. If collisions and load factors are already familiar, skip to section 3. Hash Tables From First Principles Imagine a theater coat check where coats are retrieved by surname. You say “Smith,” and the attendant applies a simple rule to decide which section to search first, section 17, perhaps. They do not scan the entire room; they go directly to one small area and inspect a few tags. The key is the surname.The value is the coat.The hashfunction turns a key into the starting section. The same key always produces the same result within a particular map.A slot stores one key/value pair. Lookup is O(1) on average because the hash takes us to a small part of the table instead of forcing us to scan every entry. This is an average-case property, not an unconditional guarantee. The unavoidable problem is a collision: there are finitely many locations, so different keys eventually choose the same starting point. Two classic strategies handle this: Chaining. Section 17 holds a list of key/value pairs. Lookup walks that small list. Hans Peter Luhn of IBM described this approach in 1953.Open addressing. Slot 17 is occupied, so try another slot, then another, until a suitable one is found. The order of locations is called the probe sequence. Open addressing was used in 1954 and formally published in 1957. Both ideas are about 70 years old. The interesting part is how modern implementations make them friendly to modern CPUs. The Old Implementation: Go 1.23 and Earlier The old Go map was a hybrid. It used fixed-size buckets, while excess collisions were handled with chains of overflow buckets. In spirit, it was closer to chaining. A map had an hmap header pointing to an array of buckets. Each `bmap` bucket contained exactly 8 slots: Plain Text flowchart LR subgraph HMAP["hmap header"] direction TB C["count: number of entries"] B["B: log2 bucket count"] P["buckets: array pointer"] end P --> ARR["bucket array: 2^B buckets"] ARR --> BKT subgraph BKT["bmap bucket: 8 slots"] direction TB TH["tophash: 8 filter bytes"] K["8 keys"] V["8 values"] OV["overflow pointer"] end OV --> OB1["overflow bucket"] OB1 --> OB2["another overflow bucket"] Three details matter: tophash: Eight bytes at the start of the bucket, one per slot. Each byte contains the top bits of that key's hash. Comparing a byte is cheaper than comparing a full string key, so it acts as a fast filter.Keys and values are stored separately: Eight keys followed by eight values. This reduces alignment padding.Overflow buckets: When the primary bucket cannot hold another entry, the runtime allocates another bucket and links it into a chain. To find grape, the runtime roughly did this: Calculate `hash("grape")`.Use the low `B` bits to select a bucket.Check the bucket's eight `tophash` bytes one at a time.When a byte matches, compare the full key. If the key matches, return the value.If the bucket has no match, follow its pointer to the overflow bucket and repeat.If the chain ends, the key is absent. The old map grew when average occupancy exceeded 6.5 entries per 8-slot bucket, a load factor of 81.25%, or when too many overflow buckets accumulated. The number of primary buckets doubled. Crucially, growth was incremental. Go did not move the entire old array at once. Each write evacuated a little more data. One unlucky insertion therefore did not have to copy a gigabyte-sized map in a single pause. Pointer chasing. Every overflow hop reads another part of the heap and increases the risk of a CPU-cache miss. A long overflow chain can therefore be expensive.Serial metadata checks. The runtime inspected `tophash` byte by byte and slot by slot. Eight slots could mean eight loop iterations and several branches.Memory overhead. Overflow buckets and their pointers cost memory. Raising the load factor much above 81% made overflow chains more common and lookup slower. The garbage collector could also have more pointers to scan. The goal was clear: fewer pointers, better locality, and less serial work. Swiss Tables provide exactly that. Meet Swiss Tables A Swiss Table is open addressing adapted to modern CPUs. Three decisions drive the design: Split the hash into an address and a short fingerprint.Pack the metadata for a group of slots into one machine word.Compare the fingerprints of 8 slots in parallel, then compare full keys only for the candidates. A 64-bit hash is divided into two unequal pieces: Plain Text 64-bit key hash ┌─────────────────────────────────────────────┬─────────────┐ │ h1: upper 57 bits │ h2: 7 bits │ │ chooses where probing begins │ fingerprint │ └─────────────────────────────────────────────┴─────────────┘ h1 selects the initial group.h2 is a 7-bit fingerprint used as a cheap filter before a full-key comparison. In the coat-check analogy, h1 chooses a section, and h2 is a short mark on each tag that quickly rules out unrelated coats. Slots are arranged in groups of 8. Each group has a 64-bit control word, one byte per slot: Plain Text control word: 64 bits = 8 bytes ┌────┬────┬────┬────┬────┬────┬────┬────┐ │ ∅ │ 15 │ 27 │ 5C │ ∅ │ 3A │ 71 │ † │ └────┴────┴────┴────┴────┴────┴────┴────┘ │ │ │ │ └─ occupied, h2 = 0x15 └─ deleted tombstone └─ empty Each byte describes its slot: Plain Text | Byte value | Meaning | | `0b1000_0000` (`0x80`) | **empty** slot | | `0b1111_1110` (`0xFE`) | **deleted** slot, or tombstone | | `0b0xxx_xxxx` | occupied; the low seven bits contain h2 | Occupied slots always have a zero high bit, while empty and deleted slots have a one. That encoding enables efficient parallel tests. Suppose we are looking for h2 = 0x27. Conceptually, the operation looks like this: Plain Text probe word: 27 │ 27 │ 27 │ 27 │ 27 │ 27 │ 27 │ 27 == │ == │ == │ == │ == │ == │ == │ == control word: ∅ │ 15 │ 27 │ 5C │ ∅ │ 3A │ 71 │ † result: 0 │ 0 │ 1 │ 0 │ 0 │ 0 │ 0 │ 0 ↑ candidate slot 2 On amd64, Go recognizes this operation as an intrinsic and uses SIMD instructions. Other architectures have a portable SWAR implementation — SIMD Within A Register — that processes all eight bytes using arithmetic on one 64-bit word. The following simplified code is optional. The important result is a bit mask of slots whose h2 may match: Go func matchH2(ctrl uint64, h2 uint8) uint64 { broadcast := uint64(h2) * 0x0101010101010101 x := ctrl ^ broadcast return (x - 0x0101010101010101) &^ x & 0x8080808080808080 } There are two reasons a candidate is not yet a result: h2 has only seven bits, so an occupied slot has a 1/128 chance of sharing the same fingerprint by accident;The portable bit trick can produce a rare extra candidate because of subtraction borrow. Neither affects correctness. Go always performs a full-key comparison before returning a value. If the initial group has no matching key, open addressing continues at group granularity. Go uses a quadratic probe sequence. Three rules are enough to understand lookup: Matching h2 bytes produce candidate slots whose full keys are checked;If the group contains an empty slot, stop — the key is absent;If the group has no match and no empty slot, visit the next group in the probe sequence. One probe handles the metadata for eight slots, so densely populated tables can still be searched efficiently. Because group metadata is cheaper to inspect, Swiss Tables can remain more densely populated. Go allows an average load up to 7/8 = 87.5%, compared with 81.25% in the old implementation. More useful entries in the same backing storage usually means less memory per key. Go's Extension: A Directory of Tables Go could not simply port Abseil's implementation. Two language and runtime requirements needed additional design work. A conventional Swiss Table grows all at once: allocate a larger array and move every entry. For a gigabyte-sized map, the insertion that triggers growth would suffer a noticeable pause. Go is widely used for latency-sensitive services, and its old maps already bounded growth work per insertion. A large Go map is a directory of independent Swiss Tables, not one unbounded table. Each table covers part of the hash space and has a maximum capacity of 1024 slots, or 128 groups: Plain Text flowchart TD M["map[K]V"] --> DIR["directory: array of table pointers"] DIR --> T0["table 0: up to 1024 slots"] DIR --> T1["table 1: up to 1024 slots"] DIR --> TN["table N"] T1 --> G0["group 0: control word + 8 slots"] T1 --> G1["group 1: control word + 8 slots"] T1 --> GK["up to 128 groups"] A variable number of upper hash bits selects the table. This technique is a form of extendible hashing. Growth is local. When a table below the limit fills, only that table grows. Once a table reaches the limit, it splits into two. The amount of growth work caused by one insertion is therefore bounded by one table of at most 1024 slots: roughly at most 896 live entries at the 7/8 load threshold (not the entire map). Small maps get a special fast path. A map that starts small and never exceeds 8 entries lives directly in a single group with no table directory. A map that has already grown, or was created with a large `hint`, does not shrink back into this representation after deletions. Unlike many hash tables, Go explicitly permits modifying a map during iteration: an entry deleted before it is reached must not be produced;an entry updated before it is reached must produce its latest value;a newly inserted entry may or may not be produced. Growth reshuffles storage, so an iterator cannot simply walk the current array. Go's iterator retains the old table to determine traversal order. Before returning an entry, it consults the current table to confirm that the key still exists and to obtain the latest value. According to the Go team, iteration is the most complex part of the implementation. Deletion and Tombstones With open addressing, deleting an entry cannot always turn its slot into an ordinary empty slot. Lookup stops at an empty slot. Creating one in the middle of another key's probe sequence could therefore make a live key unreachable. A tombstone, encoded as deleted (`0xFE`), means: “an entry used to be here; continue probing.” Insertions may reuse tombstones. Go avoids a tombstone when the group already contains another empty slot. Any lookup reaching that group would stop there anyway, so turning the deleted slot into empty cannot break a probe sequence. Accumulated tombstones disappear during a later grow or split. Live entries are copied into new groups and deleted slots are not. Go 1.24 does not implement a separate same-size grow for this purpose. And Performance Numbers According to the Go team and early production reports: Microbenchmarks: some map operations are up to 60% faster than in Go 1.23. Results vary widely, and a few edge cases regress.Full-application benchmarks: the Go team's suite showed about 1.5% geometric-mean CPU-time improvement for the whole application. A specific program may behave differently.Memory: Datadog measured roughly 70% less map memory for one very large map with about 3.5 million entries in a high-traffic environment. This favorable case benefited from denser storage, no overflow buckets, and growth that did not retain two enormous bucket arrays at once. Savings were much smaller in another environment. Large maps often benefit more because overflow chains and cache misses hurt the old design more strongly. The exact result depends on key and value sizes and on the mix of reads, writes, deletes, hits, and misses. Measure your own workload. Takeaways The Go Swiss Tables story is a good example of changing a foundational component carefully: Start with a proven design already used by Abseil and Rust's `hashbrown`.Adapt it to Go's invariants: bounded growth latency and modification during iteration.Preserve the public API and specified behavior. Hash tables are seven decades old, yet a better match between data layout and modern hardware can still remove CPU time and memory across a large ecosystem. “Solved” problems often have room for another good engineering pass. References [Faster Go maps with Swiss Tables] — the primary Go team article[Go 1.24 Release Notes] — runtime changes and `GOEXPERIMENT=noswissmap`[Go 1.24 `internal/runtime/maps` source] — original source[Go 1.26 runtime source] — the release in which the old map implementation was removed[Abseil Swiss Tables Design Notes] — original sourceMatt Kulukundis at CppCon 2017Datadog's workload-specific memory case study[`hashbrown`] — the Swiss Tables implementation behind Rust's standard `HashMap`

Batch processing remains vital because many business operations aren't suited to interactive requests. Tasks such as recalculating prices, reconciling transactions, migrating records, generating reports, processing invoices, reclassifying customers, or applying rules across millions of records may require considerable time. Handling these as standard requests leads to fragile systems, increased user wait times, frequent timeouts, challenging retries, and possible data inconsistencies. A batch model handles large workloads predictably, incrementally, and with control over progress and recovery. Rather than processing a massive operation as a single loop, batch processing uses jobs, steps, chunks, checkpoints, filtering, and restartability. This approach separates long-running data tasks from the user experience while delivering a structured execution model. In this article, we will focus on Jakarta Batch and examine its sustained relevance for modern enterprise applications. Why Batch Processing Still Matters in Enterprise Systems Modern applications offer various methods for background processing, such as message queues, event-driven architectures, schedulers, reactive pipelines, and distributed stream-processing platforms. While each addresses specific needs, batch processing is most effective when operations have a defined start and end, involve a known or discoverable dataset, and require controlled execution, progress tracking, restartability, or periodic processing. Batch processing remains essential in enterprise systems. Workloads such as financial reconciliation, billing, payroll, reporting, data migration, regulatory processing, catalog updates, and large-scale reclassification are still prevalent. In these scenarios, the priority is to process large volumes of work safely and predictably, rather than responding to individual events quickly. Batch provides a model specifically designed for these requirements. How Jakarta Batch Works Jakarta Batch organizes background processing into jobs and steps. A job defines the overall batch operation, while each step represents a specific stage. In chunk-oriented processing, a step follows a simple pipeline: read, process, write, and repeat until it processes all input. The Jakarta Batch runtime manages this lifecycle so application code can focus on reading, transforming, and persisting data. A job is the top-level unit of execution and represents a complete business operation, such as importing records, recalculating customer classifications, processing invoices, or reconciling transactions. Jobs can accept parameters at startup, allowing the same batch definition to run with different inputs or business rules. A job consists of one or more steps, each representing a distinct phase of the workload. Simple jobs may have a single step, while complex processes can use multiple steps in sequence, such as importing data, validating it, and generating a final report. Within a chunk-oriented step, the ItemReader supplies data to the runtime one item at a time, from sources such as a database or file. The reader only retrieves the next item and does not need to know how it will be processed or persisted.The ItemProcessor receives each item and applies business rules, such as validation, transformation, classification, enrichment, or filtering. It may return a modified item or null if the item should be excluded from writing.The ItemWriter receives processed items and persists or exports them. Unlike the reader and processor, which handle items individually, the writer typically receives a group of items from the current chunk. This enables more efficient database or bulk operations. Jakarta Batch adds features around this pipeline to support enterprise workloads. The runtime manages chunk boundaries, transactions, checkpoints, execution status, failures, and restart behavior. Chunk size determines how much work is grouped before a write and checkpoint, making it a key parameter for balancing throughput, memory usage, database cost, and recovery. The core model is straightforward: Job → Step → Read → Process → Write → Repeat Jakarta Batch keeps the business pipeline simple while the runtime manages the execution mechanics needed for reliable, long-running data processing. The Sample: Customer Segmentation with Jakarta Batch This example demonstrates the Jakarta Batch model using an e-commerce customer segmentation scenario. Customers are assigned to tiers such as Bronze, Silver, Gold, and Platinum based on configurable spending thresholds. When thresholds change, the application reevaluates the customer base and updates only customers whose classification has changed. The full application includes MongoDB integration, a Jakarta Faces UI, a preview workflow, validation, and supporting services. The complete source code is available at https://github.com/soujava/mongodb-jakarta-batch. This section focuses on the classes directly involved in Jakarta Batch execution. Starting the Batch Job The application initiates the batch process through CustomerSegmentationService. Unlike the reader, processor, and writer, this class is not a batch artifact. Instead, it is an application service that retrieves Jakarta Batch’s JobOperator from BatchRuntime to start and monitor job executions. Java @ApplicationScoped public class CustomerSegmentationService { public static final String JOB_NAME = "customer-segmentation"; private volatile CustomerSegmentationPolicy currentPolicy; // initialization and status methods omitted public long start(CustomerSegmentationPolicy policy) { if (isRunning()) { throw new IllegalStateException( "A customer segmentation batch is already running"); } Properties parameters = new Properties(); parameters.setProperty( CustomerSegmentationPolicy.JOB_PARAMETER, policy.toJson()); long executionId = BatchRuntime.getJobOperator() .start(JOB_NAME, parameters); currentPolicy = policy; return executionId; } public boolean isRunning() { // implementation omitted } } The key API here is JobOperator, which Jakarta Batch provides as the interface for starting, stopping, restarting, and inspecting jobs. In this example, the segmentation policy is serialized into the job parameters to ensure each execution gets the correct business rules. Reading the Input The first batch artifact, CustomerItemReader, extends Jakarta Batch’s AbstractItemReader to implement a chunk-oriented reader. Java @Named("customerItemReader") @Dependent public class CustomerItemReader extends AbstractItemReader { @Inject private CustomerRepository customerRepository; private List<Customer> customers = List.of(); private int nextIndex; @Override public void open(Serializable checkpoint) { try (Stream<Customer> customerStream = customerRepository.findAll()) { customers = customerStream .sorted(Comparator.comparing(Customer::getId)) .toList(); } nextIndex = checkpoint instanceof Integer index ? index : 0; } @Override public Customer readItem() { if (nextIndex >= customers.size()) { return null; } return customers.get(nextIndex++); } @Override public Serializable checkpointInfo() { return nextIndex; } } These methods are part of the Jakarta Batch reader lifecycle defined by AbstractItemReader. open() prepares the reader and accepts a previous checkpoint if available. readItem() provides the next item to the runtime; returning null indicates there is no more input. checkpointInfo() reports the reader’s current position for checkpointing. For simplicity, this sample loads customers into memory. For larger workloads, the implementation might use pagination or a MongoDB cursor without changing the Jakarta Batch model. Processing Each Customer The next artifact implements Jakarta Batch’s ItemProcessor interface. Java @Named("customerTierProcessor") @Dependent public class CustomerTierProcessor implements ItemProcessor { @Inject @BatchProperty( name = CustomerSegmentationPolicy.JOB_PARAMETER) private String thresholdsJson; private CustomerSegmentationPolicy policy; @PostConstruct void initialize() { policy = CustomerSegmentationPolicy.fromJson( thresholdsJson); } @Override public Customer processItem(Object item) { if (!(item instanceof Customer customer)) { throw new IllegalArgumentException( "Expected a Customer item"); } CustomerTier calculatedTier = policy.tierFor(customer.getTotalSpent()); if (calculatedTier == customer.getTier()) { return null; } return Customer.builder() .id(customer.getId()) .name(customer.getName()) .totalSpent(customer.getTotalSpent()) .tier(calculatedTier) .build(); } } Here the Jakarta Batch contract is explicit: ItemProcessor defines processItem(). The runtime calls that method for every item produced by the reader. The processor applies the segmentation rule and either returns the transformed customer or null. Returning null has a specific meaning in Jakarta Batch: the item is filtered and does not continue to the writer. The @BatchProperty is also part of the Batch integration. It receives the thresholds property defined for this job execution, allowing the processor to reconstruct the CustomerSegmentationPolicy before processing begins. Writing the Results The final artifact extends AbstractItemWriter, Jakarta Batch’s base implementation for writing a chunk. Java @Named("customerItemWriter") @Dependent public class CustomerItemWriter extends AbstractItemWriter { @Inject private CustomerRepository customerRepository; @Override public void writeItems(List<Object> items) { List<Customer> customers = items.stream() .map(this::toCustomer) .toList(); customerRepository.saveAll(customers); } private Customer toCustomer(Object item) { if (item instanceof Customer customer) { return customer; } throw new IllegalArgumentException( "Expected a Customer item"); } } writeItems() is defined by the Jakarta Batch writer contract inherited from AbstractItemWriter. Unlike the processor, which receives one item at a time, the writer receives a collection of processed items. In this case, the collection contains only customers whose classification changed, as the processor has already filtered the others. At this point, the Java components of the pipeline are as follows: Plain Text CustomerItemReader extends AbstractItemReader ↓ CustomerTierProcessor implements ItemProcessor ↓ CustomerItemWriter extends AbstractItemWriter These types are what connect the application code to the Jakarta Batch runtime. Connecting the Artifacts With JSL The Java classes define the behavior, but Jakarta Batch requires explicit mapping of the reader, processor, and writer to each job. This orchestration is described in JSL: XML <?xml version="1.0" encoding="UTF-8"?> <job id="customer-segmentation" xmlns="https://jakarta.ee/xml/ns/jakartaee" version="2.0"> <step id="recalculate-customer-tiers"> <chunk item-count="20"> <reader ref="customerItemReader"/> <processor ref="customerTierProcessor"> <properties> <property name="thresholds" value="#{jobParameters['thresholds']}"/> </properties> </processor> <writer ref="customerItemWriter"/> </chunk> </step> </job> The ref values correspond directly to the names declared with @Named in the Java classes: Java @Named("customerItemReader") @Named("customerTierProcessor") @Named("customerItemWriter") The XML therefore tells the Jakarta Batch runtime: for this step, use this reader, then this processor, and finally this writer. It also maps the thresholds job parameter into the processor property. The item-count="20" sets the chunk size for this sample. Jakarta Batch coordinates reading and processing, periodically invoking the writer according to the chunk lifecycle and establishing transaction and checkpoint boundaries. The value 20 is for demonstration; real applications should tune chunk size based on processing cost, database behavior, transaction size, throughput, and recovery requirements. This structure is recommended for the article: present the class declaration first, then describe the lifecycle methods inherited from or required by Jakarta Batch. This approach helps the sample teach the API rather than simply presenting isolated methods. Conclusion Jakarta Batch is valuable because it transforms large-scale data processing into a structured execution model, eliminating the need for custom loops and ad hoc background logic. By separating reading, processing, and writing, and introducing runtime concepts such as jobs, steps, checkpoints, restartability, and chunk-oriented execution, it provides enterprise applications with a predictable approach to handling workloads involving thousands or millions of records. This allows implementations to focus on business logic, while the Batch runtime manages repetitive execution concerns, making the model easier to understand, optimize, and scale as workloads increase.

By Otavio Santana CORE

I went looking for a flag that could cap the amount spent on a session on DeepAgents. The kind of hard spending cap that stops an agent loop before it burns through a budget. What exists instead is a warning, and the gap between "warning" and "stopping" turned out to be the more interesting story. What Already Exists: A Number With No Teeth deepagents-code has a CostTrackingMiddleware that owns a thread's cumulative spend. This is a real, checkpointed dollar figure priced from actual per-request token usage, not an estimate. A separate feature built on top of that number, merged earlier, adds a one-time warning once the total crosses a configured threshold (default $50): Python # app.py, roughly what ships today threshold = self._session_cost_warning_threshold_usd if ( not self._session_cost_warning_shown and 0 < threshold < self._session_cost_usd ): self._session_cost_warning_shown = True self.notify( f"Estimated session cost is {format_cost(self._session_cost_usd)}, " f"above the configured {format_cost(threshold)} threshold. Consider " "/offload to reduce context usage or /clear to start fresh.", title="Session cost warning", severity="warning", timeout=12, markup=False, ) Read that once more: It's a self.notify(...) call. A toast! The agent's own loop has no idea this happened. Nothing in the code above touches the graph, the model call, or the next tool invocation. If you're watching the terminal, you see the warning and can intervene by hand (/offload, /clear, or just killing the process). If you're not watching and it's a headless CI run — for example, an unattended overnight session or a tool-call loop that's quietly retrying against a flaky API — this number keeps climbing, and nothing stops it. That's the gap: a cost number that can only ever inform a human, never the loop that's actually spending the money. Why the Fix Isn't "Just Check the Number Somewhere" The obvious instinct is to add a if cost > limit: stop check. The real work is in where that check has to live and what "stop" has to mean to a LangGraph agent loop. Where: The check needs to run before the next model call, not after. Checking after a call has already happened is too late to prevent its cost. LangGraph gives middleware a before_model hook for exactly this. LangChain's own ModelCallLimitMiddleware (a call-count limiter, not a cost one) already establishes the pattern: check a condition in before_model, and if it's tripped, return an update that redirects the graph to end instead of letting the model call happen. What "stop" means: A plain return None from before_model just lets the loop continue. To actually halt, the hook needs @hook_config(can_jump_to=["end"]) and has to return {"jump_to": "end", ...} which is a real graph-control signal, not a value the caller has to notice and act on: Python # cost_tracking.py — the actual hook, as committed @hook_config(can_jump_to=["end"]) def before_model( self, state: CostState, runtime: Runtime[ContextT], ) -> dict[str, Any] | None: """Halt the run before the next model call if the hard cost cap is met. Checked against the checkpointed cumulative total from the *previous* step -- the same figure the TUI's soft warning reads via `_set_session_cost` -- so this fires at the same point in the loop a user would already have seen the warning toast, just before the next request that would push spend further over the configured cap. """ if self._nested or self._hard_limit_usd is None: return None total_usd = state.get("_session_cost_usd") if ( isinstance(total_usd, bool) or not isinstance(total_usd, int | float) or not math.isfinite(total_usd) ): return None if total_usd < self._hard_limit_usd: return None if self._exit_behavior == "error": raise CostLimitExceededError(total_usd, self._hard_limit_usd) limit_message = _build_cost_limit_message(total_usd, self._hard_limit_usd) return {"jump_to": "end", "messages": [AIMessage(content=limit_message)]} Two details worth calling out, because they're the kind of thing that looks like overcaution until you hit the failure it's guarding against: isinstance(total_usd, bool) before the numeric check. In Python, bool is a subclass of int, so isinstance(True, int | float) is True and True < 5.0 evaluates fine (True == 1). Without the explicit bool guard, a stray True sitting in a state field meant for a float would silently be treated as $1.00 and could either falsely trip the halt or falsely pass it, depending on the cap. Cheap to guard against, expensive to debug if you don't. Checked only on the non-nested instance. CostTrackingMiddleware also runs on subagents, where its _session_cost_usd channel tracks that subagent's own local spend before it's transferred back to the parent's running total. Checking the hard cap there would be checking the wrong number - a subagent's small local total against a cap meant to bound the whole session's spend. self._nested gates this out entirely. The Part That Isn't the Algorithm: Getting the Number Across a Process Boundary Here's what made this bigger than a one-file change: dcode doesn't run the agent loop in the same process as the CLI. It starts a LangGraph server in a subprocess and talks to it over langgraph-sdk. Every configuration value the agent needs, like model name, sandbox type, recursion limit, and now this cap, has to survive that boundary, which in this codebase means round-tripping through environment variables on a ServerConfig dataclass: Python # _server_config.py max_cost_usd: float | None = None """Explicit hard cap, in USD, on the main thread's cumulative estimated cost.""" def to_env(self) -> dict[str, str | None]: return { ... "MAX_COST_USD": ( str(self.max_cost_usd) if self.max_cost_usd is not None else None ), ... } @classmethod def from_env(cls) -> ServerConfig: return cls( ... max_cost_usd=_read_env_float("MAX_COST_USD", default=None), ... ) _read_env_float didn't exist before this - every other numeric config value in this file is an int (recursion_limit, turn counts), so there was no float-reading helper to reuse. One new function, matching the existing _read_env_int's shape exactly, and the round-trip works the same way every other config value already does. The full path, in order: a --max-cost CLI flag → a resolver that checks the flag, then a config.toml entry, then "disabled" → create_cli_agent(max_cost_usd=...) → ServerConfig → serialized to an environment variable → the subprocess reads it back → create_cli_agent again, this time inside the subprocess → CostTrackingMiddleware(hard_limit_usd=...). Six hops for one float, and every one of them was necessary - skip the ServerConfig round-trip and the flag works in a unit test but silently does nothing the moment you actually run dcode, because the subprocess that runs the real agent loop never sees it. Verifying It Live Unit tests covering the before_model logic in isolation are necessary but not sufficient here. They'd pass even if one of those six hops silently dropped the value, because a unit test calls the middleware directly and never exercises the subprocess boundary at all. The only way to know the flag actually works is to run the real CLI against a real model: Shell $ dcode -n "Read sample.txt, then read it again, then read it a third time. \ Do this as three separate tool calls, one per turn." \ --model anthropic:claude-haiku-4-5 \ --max-cost 0.0001 \ --max-turns 6 I'll read sample.txt, then read it again on the next turn, then a third time on the turn after that. First read: Calling tool: read_file Session halted: estimated cost $0.02 has reached the configured limit of <$0.01. Raise the limit (e.g. `--max-cost`) or start a new session to continue. Task completed Usage Stats Provider Model Reqs InputTok OutputTok Cost anthropic claude-haiku-4-5 1 13.6K 147 $0.02 The model was told to make three tool calls, one per turn. It made exactly one. The middleware checkpointed that turn's real cost (0.02), the next beforemodel check found the cumulative total over the(deliberately absurd) 0.0001 cap, and the graph jumped to end with the injected message instead of continuing to spend on turns two and three. Total cost of proving this worked: two cents. Why This Is Worth a Hard Stop and Not Just a Bigger Warning You could imagine closing this gap by making the warning louder: repeat it every turn instead of once, or block user input until it's acknowledged. That doesn't fix the actual failure mode, which is specifically the unattended case. A louder toast is still a toast. Nothing short of a return value the graph itself has to obey closes that gap, which is why the fix has to live in before_model, not in the terminal UI layer where the existing warning already sits. It's also worth being honest about what this doesn't fix: the check runs before a model call, using the cost checkpointed from the previous one. A single turn can still overshoot the cap if the cap is 5.00 and the agent is at 4.99. The next call still happens in full and might land at 6.00 before the halt fires on the turn after. That's not a bug so much as an inherent property of checking after the fact rather than metering mid-request, and it's the same tradeoff that ModelCallLimitMiddleware makes for call counts. A cap is a backstop against runaway, unattended spend. It's not a precise billing guarantee down to the last cent. Takeaways The general lesson here isn't really about cost. It's that a warning and a limit are two different features wearing the same clothing, and it's easy to ship the first while believing you've shipped the second. The warning reads the same number, uses the same word ("threshold"), and looks like it's doing the same job right up until someone isn't in the room to read it. The tell is always the same: does the check return a value the system has to act on, or does it just call something with "notify" in the name? Second, a fix that only works in-process is only half a fix once your architecture has a subprocess boundary in it. The six-hop threading here wasn't extra caution, but it was the actual scope of the problem, and skipping any one hop would have shipped a flag that silently does nothing. Third, if you can run the real thing end-to-end for two cents, there's no good reason to trust a mock's word for whether a fix actually works.

By Ninaad Rao CORE

Production bugs frequently arrive with too little evidence. A screenshot captures the final visual state, a crash report identifies a failing stack, and a support ticket describes what appeared to happen. None reliably explains the sequence that produced the failure. Modern applications are asynchronous systems driven by navigation, network responses, feature flags, background work, local persistence, and changing UI state. A useful production bug report therefore needs more than the final frame. It needs a bounded, privacy-safe execution history that reconstructs the path leading to failure. The foundation of a replayable bug report is a semantic event stream. Continuous video recording is expensive, difficult to search, and likely to capture information unrelated to diagnosis. Structured events are smaller and describe meaningful transitions directly. Navigation changes, button actions, state mutations, network outcomes, feature flag evaluations, lifecycle transitions, and persistence failures can use a common event model. Java recordEvent( "ui.action", "checkout.submit", Map.of("screen", "Checkout", "cartState", "ready") ); The event records behavior rather than pixels. Stable identifiers such as checkout.submit are preferable to coordinates or view hierarchy paths because layouts change between releases. Each event should also contain a session identifier, application version, timestamp, and monotonic sequence number. Those fields transform disconnected observations into an ordered execution history. Continuous recording immediately creates a resource constraint. Retaining every event for an entire session eventually consumes excessive memory or storage. A bounded ring buffer solves this by retaining only the most recent diagnostic window. New events replace the oldest once the configured limit is reached. Java private final Deque<ReplayEvent> events = new ArrayDeque<>(); private static final int LIMIT = 500; public synchronized void record(ReplayEvent event) { if (events.size() == LIMIT) events.removeFirst(); events.addLast(event); } Production implementations can enforce both event count and byte size limits because payload sizes vary. High-frequency signals also require control. Scroll offsets, animation callbacks, connectivity heartbeats, and repeated state notifications can overwhelm useful evidence. Sampling or coalescing these signals keeps the buffer focused on transitions that materially affect application behavior. Ordering deserves separate treatment because wall-clock timestamps alone are unreliable. Device clocks can change, concurrent callbacks can receive identical timestamps, and asynchronous tasks can finish in an order different from their creation order. Assigning an atomic sequence number when each event enters the recorder establishes deterministic local ordering. Wall-clock time remains valuable for correlation with server logs, while the sequence number determines authoritative ordering inside the client session. Network operations are especially valuable during reconstruction, but complete request and response bodies usually are not. Recording the HTTP method, normalized route, status code, duration, retry count, failure category, and correlation identifier provides useful evidence without copying sensitive payloads. Java recordNetwork( "POST", "/checkout", statusCode, elapsedMillis, correlationId ); The correlation identifier connects client replay with backend observability. When trace context propagates through an API gateway and downstream services, the same failed interaction can be followed beyond the device. OpenTelemetry context propagation supports this model through trace and span context carried across execution boundaries. A replay system becomes substantially more useful when a client event can lead directly to the corresponding backend trace instead of creating an isolated client-side observability system. Events alone may still be insufficient when identical actions behave differently under different runtime conditions. Selective state snapshots fill that gap. A snapshot should not serialize the complete application object graph. It should capture small diagnostic facts that influence behavior, such as authentication status, active feature flags, connectivity mode, pending operation counts, cache generation, lifecycle state, and other safe application state. Java recordSnapshot(Map.of( "screen", "Checkout", "network", "cellular", "paymentState", "submitting", "feature.checkoutV2", true )); Snapshots can be generated at meaningful boundaries such as screen entry, transaction start, backgrounding, synchronization completion, or error detection. During reconstruction, they explain conditions surrounding an event without attempting to reproduce every byte of runtime memory. This keeps diagnostic bundles small while preserving state likely to affect execution. Privacy must be enforced during capture rather than treated as an upload-time cleanup operation. Session replay platforms commonly provide client-side masking because sensitive values should not leave the application in their original form. The same principle applies to a custom recorder. Event attributes should follow explicit allowlists, while passwords, authorization headers, tokens, payment information, message contents, email addresses, and unrestricted text fields should be excluded by default. Java private String sanitize(String key, String value) { if (SENSITIVE_KEYS.contains(key)) return "[REDACTED]"; return value; } Key filtering alone is insufficient because sensitive values can appear under unexpected field names. Stronger implementations can combine schema allowlists, endpoint-specific policies, value-pattern detection, maximum lengths, and explicit data classifications. Redaction should occur before information enters the ring buffer. Once sensitive data has reached memory, disk, crash attachments, or telemetry queues, later sanitization becomes considerably harder to guarantee. The recorder must also survive the failure being diagnosed. An entirely in-memory history disappears during process termination, watchdog kills, native crashes, or operating system eviction. Periodic checkpointing to a small protected file allows the next launch to recover the tail of the previous session. Writes should remain asynchronous and bounded so diagnostic instrumentation does not introduce latency or instability into normal execution. Crash time behavior should remain minimal. Attempting complex serialization after a fatal condition can itself fail. A safer design periodically persists compact checkpoints during healthy execution, and treats crash handling as a final marker whenever possible. On the next launch, the previous checkpoint, crash metadata, application version, device characteristics, and correlation identifiers can be assembled into a diagnostic bundle. Upload requires failure handling as well. A diagnostic system cannot assume connectivity exists immediately after a crash or restart. Bundles can enter a small persistent queue and upload when network conditions permit. Successful delivery removes the local copy, while repeated failures follow bounded retry and retention policies. This prevents diagnostic infrastructure from becoming another source of uncontrolled storage consumption or retry storms. Replay does not need to mean pixel-perfect visual reproduction. For engineering diagnosis, a deterministic event timeline is often more useful. Plain Text 10:42:11.018 screen.enter Checkout 10:42:14.201 ui.action checkout.submit 10:42:14.233 network.start POST /checkout 10:42:15.107 network.finish status=200 10:42:15.116 persistence.error order_write_failed 10:42:15.120 ui.state payment=failed A screenshot from this incident shows only a failed checkout screen. The timeline reveals something fundamentally different, as the server operation succeeded, but local persistence failed afterward. That distinction changes the investigation immediately. Instead of examining API availability or retry behavior, diagnosis can move directly toward local storage, transaction handling, or post-response state transitions. Structured replay can extend beyond manual inspection. Development builds can consume sanitized event sequences to configure test doubles, restore relevant feature flags, reproduce network outcomes, and drive selected state transitions. Full determinism is not always possible because operating system scheduling, race conditions, third-party services, and timing-sensitive behavior introduce variability. Even without executable replay, a causal timeline drastically reduces the search space by preserving conditions that conventional logs frequently lose. The recorder should remain diagnostic infrastructure rather than becoming another analytics pipeline. Capturing every possible signal increases CPU usage, storage requirements, privacy exposure, and noise. A small event vocabulary is usually more effective, just like navigation transitions, meaningful interactions, network starts and outcomes, state changes, persistence operations, lifecycle events, and failures. Instrumentation quality matters more than event volume. A practical rollout can begin around workflows responsible for the most expensive production failures rather than instrumenting every screen. Checkout, authentication, synchronization, uploads, and other difficult flows can receive semantic events and snapshots first. When an incident demonstrates that an important transition is missing, the event model can evolve deliberately. This keeps the recorder maintainable and prevents instrumentation from becoming an uncontrolled collection of arbitrary log statements. Screenshots remain useful evidence, but they should represent one frame inside a richer diagnostic record rather than the entire debugging strategy. A bounded semantic event buffer, deterministic ordering, selective state snapshots, trace correlation, capture time redaction, crash-safe persistence, and resilient upload can transform vague production reports into reconstructable execution histories. The result is more than improved logging. It is a practical application flight recorder that preserves the context normally lost at the exact moment a difficult production failure occurs.

By Uthej Mopathi CORE

I went looking for a clean way to show what prompt caching actually saves an agent, and the first thing I found was a fact that's easy to miss if you only read the "up to 90% savings" headline. Caching a fresh conversation's first turn costs more than not caching it. There's no cache to read from yet, so you pay the input price on the content plus a 25% premium to write to the cache, and get nothing back. The savings arrive starting on turn two, once there's something to read. Where This Lives in deepagents Deep Agents ships AnthropicPromptCachingMiddleware from langchain-anthropic in its default middleware stack. It doesn't decide what to cache by guessing. It tags exactly two things on every model call: Python # langchain_anthropic/middleware/prompt_caching.py (trimmed) def _apply_caching(self, request: ModelRequest) -> ModelRequest: overrides: dict[str, Any] = {} cache_control = self._cache_control # {"type": "ephemeral", "ttl": self.ttl} overrides["model_settings"] = {**request.model_settings, "cache_control": cache_control} system_message = _tag_system_message(request.system_message, cache_control) if system_message is not request.system_message: overrides["system_message"] = system_message tools = _tag_tools(request.tools, cache_control) if tools is not request.tools: overrides["tools"] = tools return request.override(**overrides) The system prompt gets one breakpoint on its last content block, and the tool list gets one breakpoint on its last tool. Tool definitions are sent as one contiguous block; a single trailing breakpoint caches the entire tool set. On top of that, the top-level cache_control on model_settings gets translated by Anthropic's own auto-caching behavior into a breakpoint at the end of whatever's cacheable in the request. This is what lets the cached prefix grow to cover prior conversation turns as an agent session gets longer, not just the fixed system+tools prefix. That's the whole mechanism. No decision logic, no cost estimation, no adaptive behavior. It tags the stable parts and lets the API's own prefix-match caching do the rest. The Actual Cost Shape, Measured Live Here's the part that isn't obvious from "caching saves money": it saves money on a schedule, not uniformly. Prompt caching is a prefix match. Turn 2 can only read from cache what turn 1 wrote, turn 3 reads what's accumulated through turn 2, and so on. That gives every agent conversation the same two-phase cost curve: write-only first turn, then reads that get proportionally cheaper as the stable prefix grows relative to what's new each turn. To measure this for real rather than project it, I ran the same 8-turn engineering conversation (a realistic back-and-forth about refactoring a blocking-I/O call inside an async FastAPI handler) through claude-opus-5 twice - once with no caching, once with top-level auto-caching (cache_control={"type": "ephemeral"} on every request, the exact mechanism AnthropicPromptCachingMiddleware uses via model_settings["cache_control"]). The system prompt was a real ~1,345-token engineering-assistant prompt, counted with count_tokens before running anything billed: Python """python run_experiment.py (needs ANTHROPIC_API_KEY)""" import anthropic MODEL = "claude-opus-5" INPUT_PRICE = 5.00 / 1_000_000 OUTPUT_PRICE = 25.00 / 1_000_000 CACHE_WRITE_MULT = 1.25 # 5-minute TTL CACHE_READ_MULT = 0.10 SYSTEM_PROMPT = open("system_prompt.txt").read() # a real ~1,345-token prompt TURNS = [...] # 8 real follow-up questions in one coherent conversation client = anthropic.Anthropic() def run(*, use_caching: bool) -> list[dict]: messages: list[dict] = [] turn_records = [] for user_text in TURNS: messages.append({"role": "user", "content": user_text}) kwargs = { "model": MODEL, "max_tokens": 400, "system": SYSTEM_PROMPT, "messages": messages, } if use_caching: kwargs["cache_control"] = {"type": "ephemeral"} resp = client.messages.create(**kwargs) usage = resp.usage messages.append({ "role": "assistant", "content": "".join(b.text for b in resp.content if b.type == "text"), }) turn_records.append({ "input_tokens": usage.input_tokens, "output_tokens": usage.output_tokens, "cache_creation_input_tokens": getattr(usage, "cache_creation_input_tokens", 0) or 0, "cache_read_input_tokens": getattr(usage, "cache_read_input_tokens", 0) or 0, }) return turn_records def cost_for_turn(r: dict) -> float: return ( r["input_tokens"] * INPUT_PRICE + r["cache_creation_input_tokens"] * INPUT_PRICE * CACHE_WRITE_MULT + r["cache_read_input_tokens"] * INPUT_PRICE * CACHE_READ_MULT + r["output_tokens"] * OUTPUT_PRICE ) The real output, 16 live API calls, $0.24 total: TurnNo-cache $Cached $Savingscache_read/creation (cached run)10.0170$0.0187$-10.2%read=0 write=138920.0173$0.0111$35.7%read=1389 write=6430.0175$0.0110$37.1%read=1453 write=4040.0177$0.0110$37.7%read=1493 write=4050.0179$0.0110$38.4%read=1533 write=3760.0182$0.0112$38.5%read=1570 write=6170.0184$0.0111$39.6%read=1631 write=4680.0185$0.0110$40.5%read=1677 write=27Total0.1423$0.0961$32.5% Turn 1 really is negative: 10.2% more expensive than no caching at all, exactly as the documented economics predict. A 5-minute-TTL cache write needs at least two requests to break even (1.25x write + 0.1x read ≈ 1.35x, against 2x for two uncached requests). After that, savings climb every single turn, because the thing growing is the cached portion of the prompt (the cache_read_input_tokens column climbing from 1,389 to 1,677 across the run), while the thing staying flat is the new portion (the write column). By turn 8, caching is saving 40.5% on that turn alone. This is also why a one-shot script or a short-lived Lambda almost never benefits from caching. If the conversation ends after one or two turns, you're stuck in the loss zone the math shows on turn 1. Caching pays for itself on sustained, multi-turn agent sessions, which is exactly the shape of a deepagents CLI session or a long-running subagent loop, not a single classification call. How Much the Stable Prefix Matters The 32.5% overall figure here is real, but it's specific to this run's cacheable prefix being a fairly modest ~1,345-token system prompt. That number moves with the size of the stable part of the prompt relative to what's genuinely new each turn. Swapping SYSTEM_PROMPT in the script above for a larger, more tool-heavy prefix (closer to what a deepagents agent with 15+ MCP tool schemas actually carries) and rerunning would show the savings percentage climb further. The shape of the curve doesn't change, only how much of the bill it's discounting. If you want to know your own number, this script is a five-minute run against your own agent's actual system prompt. Two Levers, and They Stack Caching cuts the price per token of everything that repeats turn over turn. It says nothing about how many tokens you're sending in the first place. That's a second, independent lever: tool-selection filtering, which cuts the token count itself by not sending schemas irrelevant to the current turn. Measured earlier on a 61-tool registry, a lexical-overlap scorer at top_k=10 gets 67% recall at an 84% reduction in tool-schema payload per turn. These aren't competing techniques — they compose. Caching makes the tokens you do send cheaper turn over turn; tool selection reduces how many tokens are in the first place, in the one part of the prompt (the tool list) caching's own breakpoint sits right in front of. A 6,000-token stable prefix that's 40% tool schemas doesn't stay 6,000 tokens once you're only sending the top-10 relevant tools instead of all 61: it shrinks, and then gets cached at the smaller size. Neither technique substitutes for the other: one is about the price of a token, the other is about whether you send it at all. Takeaways The counterintuitive part is the one worth remembering: caching a conversation that only runs one or two turns can cost more than not caching it, because the write premium has nothing to amortize against. It only pays off on sessions that actually run long enough to read back what turn one wrote. Second, if you're trying to actually cut what an agent costs, price-per-token and token-count are separate dials: caching turns one, tool selection turns the other, and a real cost reduction effort turns both.

By Ninaad Rao CORE

Every enterprise conversation about artificial intelligence eventually arrives at the same uncomfortable question: what happens to our data once it leaves our perimeter? Whether you are wiring a large language model into a claims processing pipeline, standing up a retrieval-augmented generation (RAG) system for internal knowledge search, or letting an agentic workflow take autonomous actions against production systems, the answer determines whether your AI initiative becomes a competitive advantage or a compliance incident waiting to happen. This article lays out a layered, defense-in-depth approach to securing enterprise data across the AI lifecycle — from data classification and access control, through transit and storage, vendor contracts, prompt hygiene, architecture patterns, and compliance mapping. It is written for architects, technical leads, and engineering managers who are past the "should we use AI" conversation and are now living in the "how do we do this safely at scale" reality. The guidance here is deliberately vendor-agnostic and framework-agnostic. The specific tools you choose — which cloud, which model provider, which vector database — will vary. The principles will not. 1. Why AI Changes the Data Security Calculus Traditional application security assumes a relatively closed loop: your code, your database, your network boundary. Data moves through defined pathways, and you can reason about every hop. AI systems break several of these assumptions at once. The boundary is porous by design. A large language model call is, functionally, an API request to a third party — even when that third party is a trusted enterprise vendor. Every prompt is an egress point. Every completion is an ingress point. Unlike a traditional API integration where the schema is fixed and the payload is structured, prompts are free text, which makes it far easier for sensitive data to slip in unnoticed. The system can be instructed by its input. In a conventional application, data and instructions are cleanly separated — SQL injection exists precisely because that separation sometimes breaks down, and we have spent two decades building defenses against it. In an AI system, the model's instructions and the data it processes often occupy the same channel: natural language. This is the root cause of prompt injection, and it means the data itself can become an attack vector, not just a target. The system can act, not just answer. Agentic AI — systems that call tools, write files, send emails, or modify records — collapses the distinction between "the AI leaked data" and "the AI did something harmful with data." A single compromised or manipulated agent can chain read access into write access, and write access into external communication, all within one interaction. The data footprint compounds. Vector embeddings, cached completions, fine-tuning datasets, evaluation logs, and conversation histories all represent new copies of your sensitive data, sitting in new places, governed by new retention rules that your existing DLP and archiving policies were never built to see. None of this means AI is unsafe to deploy in the enterprise. It means the security model has to be designed deliberately, layer by layer, rather than inherited by default from your existing application security posture. 2. Data Governance and Classification Everything downstream depends on getting this layer right first. If you don't know what data you have and how sensitive it is, no amount of encryption or access control will save you from sending the wrong thing to the wrong place. Classify Before You Integrate Before any AI system touches production data, classify it into tiers. A simple, workable scheme: Public – marketing content, published documentation, anything already externally visible.Internal – operational data with no direct regulatory exposure, but not meant for public release.Confidential – customer PII, employee data, financial figures, strategic plans.Restricted – regulated data categories: PHI under HIPAA, PCI cardholder data, biometric data, data covered by state insurance data security laws, or anything under a specific contractual non-disclosure obligation. Each tier should carry an explicit, written policy on whether and how it may be used with AI systems — including which AI systems (internal, VPC-isolated, or public API) and under what redaction or tokenization requirements. Data Minimization Is a Design Constraint, Not an Afterthought The single most effective control available to you is simply not sending data you don't need to send. This sounds obvious and is routinely ignored under deadline pressure. Practical patterns: Field-level scoping: If a prompt needs a claim status and adjuster name, don't serialize the entire claim record into context. Query for and pass only those fields.Row-level scoping in RAG: Retrieval pipelines should filter at the query layer (based on the requesting user's entitlements) before documents ever reach the context window, not rely on the model to "know" what it shouldn't discuss.Aggregate over raw where possible: If the use case is trend analysis, send aggregated statistics rather than the underlying raw records. Understand Retention and Training-Use Terms This is the question every enterprise security review should ask first, and the one most often skipped: does the AI provider retain my inputs and outputs, and are they used to train or improve models? Enterprise API tiers from major providers typically differ meaningfully from consumer-facing chat products on this point — enterprise agreements commonly include zero data retention (ZDR) options and explicit commitments that customer data is not used for model training. Consumer tiers, free tiers, and browser extensions are a different story and should be treated as such in your acceptable use policy. Don't assume; read the actual data processing terms for the specific tier and product you're using, and get it in writing. Data Lineage for AI-Touched Data Once data has passed through an AI system, it has effectively been transformed and potentially recombined. Maintain lineage records: which source systems fed which prompts, which model version processed them, and where the outputs were stored or acted upon. This becomes essential later for both incident response and regulatory audit. 3. Access Control for AI Systems Treat AI Service Accounts Like Any Other Privileged Identity An AI agent or pipeline that calls internal APIs is a service account. It should be provisioned, reviewed, and revoked exactly like any other service account — with the added scrutiny that its "instructions" can be influenced by untrusted input in ways a traditional service account's code path cannot. Least privilege, scoped by task: A customer-support chatbot that looks up order status needs read access to an orders API — not write access, not access to the full customer database, not admin scopes "just in case."Short-lived credentials: Prefer short-lived, automatically rotated tokens (OAuth client-credentials flows, workload identity federation) over long-lived static API keys.Per-tenant isolation: In multi-tenant SaaS or multi-client environments, ensure the AI system cannot cross tenant boundaries even if a prompt attempts to coax it into doing so — enforce this at the data access layer, not the prompt layer. Enforce Human Entitlements Downstream of the Model, Not Just Upstream A common and dangerous mistake: building a RAG or agentic system where the AI service account has broad access "for flexibility," and relying on the system prompt to tell the model which documents the current user is allowed to see. Prompts are not an access control mechanism. If the underlying retrieval or tool-calling layer can technically reach a document or record, a sufficiently motivated (or simply unlucky) input can potentially surface it. The correct pattern is to filter at the data layer using the actual requesting user's entitlements — row-level security in the database, document ACLs in the retrieval index, and scoped API tokens minted per-request based on the authenticated user, not the service account. Role-Based Access for AI Outputs Re-Entering the System When an agentic workflow's output writes back into production — updating a record, sending a notification, filing a claim note — that write should pass through the same RBAC and validation layer a human-initiated write would. Do not grant an AI agent a privileged bypass "because it's automated." Automation is exactly when you want the guardrails to be strongest, because there is no human in the loop to notice something is wrong before it happens. 4. Secrets and Credential Management This deserves its own section because AI systems introduce new and easy-to-miss places for secrets to leak. Never hardcode credentials in prompts, system prompts, or agent configuration files. It is tempting to embed an API key directly in a tool definition during a proof of concept. That habit does not survive contact with production.Never let secrets end up in agent memory or long-term conversation history. If your architecture includes persistent memory for an agent, explicitly exclude credential material, and audit what actually gets written to that memory store.Use a dedicated secrets manager — Azure Key Vault, AWS Secrets Manager, HashiCorp Vault, or your platform equivalent — and have the AI orchestration layer fetch credentials at call time rather than holding them statically.Rotate aggressively for anything touched by an AI pipeline. Given that prompts and tool definitions are more likely to be copy-pasted into documentation, shared in Slack for debugging, or logged verbosely during development, treat any credential that has been anywhere near an AI pipeline as higher-risk and rotate on a shorter cycle.Watch your logs. Verbose request/response logging — common during AI development for debugging prompts — is a frequent, unglamorous source of credential leakage. Redact before logging, not after. 5. Data in Transit and at Rest The fundamentals here are not AI-specific, but they are easy to underinvest in because AI integrations often move fast and get treated as "just another API call." TLS everywhere, including between internal orchestration services and the AI provider, and between internal services and any vector database or cache.Encrypt at rest, including: The primary data stores feeding your RAG pipeline.Vector embeddings themselves. Embeddings are not inherently anonymous — depending on the embedding model and dimensionality, source text can sometimes be partially reconstructed from vectors, so treat an embedding store with the same sensitivity as the source documents.Prompt and completion logs.Any cached responses (semantic caching layers are increasingly common for cost control and latency, and they represent another copy of potentially sensitive data at rest).Encrypt backups of all of the above, and include them explicitly in your data retention and destruction policies — a backup snapshot of a vector database is a backup of your confidential documents. 6. Vendor and Contractual Controls Technical controls only get you so far if the underlying contract with your AI provider doesn't back them up. Zero Data Retention Agreements Where available, negotiate zero data retention (ZDR) terms — an explicit commitment that request payloads are not retained beyond the time needed to serve the response, and are not logged, cached, or used for any secondary purpose. This is increasingly available as a contractual option from major enterprise AI providers and should be a standard line item in procurement for any AI vendor touching confidential or restricted data. Data Processing Agreements A proper Data Processing Agreement (DPA) should cover: Purpose limitation (data used only to provide the contracted service).Sub-processor disclosure (who else touches your data downstream of the primary vendor).Data residency commitments (does data ever leave a specific geographic or regulatory jurisdiction).Breach notification timelines.Audit rights. Enterprise Tier Versus Consumer/Shared Infrastructure Confirm explicitly whether you are on infrastructure that is logically or physically isolated from other customers, versus a shared multi-tenant consumer product. Ask directly: is my data ever used to train models that serve other customers? Is there any possibility of cross-tenant data mixing in caching or logging layers? Get the answer in the contract, not just in a sales conversation. Vendor Security Posture Review Standard vendor risk management practice applies, but with AI-specific questions added to the questionnaire: What is the model provider's own subprocessor chain?How is prompt injection or jailbreak resistance tested and monitored on their side?What certifications do they hold (SOC 2 Type II, ISO 27001, ISO 42001 for AI management systems specifically)?What is their incident response commitment and SLA for a security event affecting your data? 7. Prompt and Output Hygiene Treat Untrusted Content as Untrusted, Even Inside a Prompt Prompt injection is the AI-era equivalent of injection attacks in traditional application security, and it deserves the same rigor. Any content that originates outside your organization's direct control — an email, an uploaded document, a web page fetched by a tool, a third-party API response — should be treated as untrusted input, not as trusted instructions, even when it is concatenated into the same prompt as your system instructions. Practical mitigations: Clear structural separation between system instructions, trusted context, and untrusted content, using explicit delimiters and, where the platform supports it, distinct message roles.Instruction-following boundaries: Explicitly instruct the model that content within untrusted blocks should be treated as data to analyze, not as commands to follow — and validate this behavior in testing with adversarial inputs, not just happy-path examples.Least-privilege tool access during untrusted content processing: If an agent is currently processing an untrusted document, don't give it simultaneous access to high-privilege tools (sending email, executing code, modifying records) without a human confirmation step in between. Output Validation Before Action Any AI output that will be displayed to a user, stored in a system of record, or used to trigger a downstream action should pass through validation: Schema validation for structured outputs (if you asked for JSON, validate it actually conforms before using it).PII/sensitive-data scanning on outputs, not just inputs — a model can sometimes surface data it was never explicitly asked to reveal, particularly in RAG systems with imperfect retrieval filtering.Action confirmation gates for anything irreversible or high-impact — sending external communications, financial transactions, deleting records — even in a fully agentic workflow. A "dry run" or human-approval step for a defined set of high-risk action types is a small latency cost for a large risk reduction. PII Redaction Pipelines For any workload where the AI system doesn't strictly need to see PII to do its job, run a redaction or tokenization pass before the data reaches the prompt, and a re-hydration pass on the output if needed. This is particularly relevant when using third-party or shared-infrastructure LLM endpoints for tasks like summarization or classification, where the specific identity behind the data is often irrelevant to the task itself.

By Balaji Venkatasubramaniyar CORE

Retries are one of the simplest ways to make a distributed system appear more reliable. A transient connection failure, overloaded replica, or short-lived network interruption can disappear after another attempt, which is why retry support exists in major RPC frameworks and cloud SDKs. The danger begins when every layer makes the same decision independently. A mobile client retries an API gateway, the gateway retries a service, that service retries another service, and the final dependency retries a database call. The original request has not become more important, but the system has multiplied the work required to fail. AWS describes a five-deep service stack in which three attempts at each layer can drive 243 calls against the database when the deepest dependency is failing. Google’s SRE guidance similarly warns that retries can amplify overload and contribute to cascading failure. When Reliability Logic Becomes Additional Load The common retry policy focuses on a single caller. A request fails, exponential backoff delays the next attempt, and jitter prevents large client populations from retrying at exactly the same instant. Those mechanisms remain important. AWS recommends backoff and jitter because immediate, synchronized retries can worsen overload, while gRPC exposes retry limits, exponential backoff, retry throttling, and server pushback for the same class of problem. The missing property is coordination. Consider three logical layers, each configured for three total attempts. If the lowest dependency rejects every request, a single logical operation can create up to 27 downstream attempts. Adding more independently retrying layers increases that multiplier exponentially. Backoff changes when those attempts arrive; it does not change the fact that separate components are authorizing additional work from the same original operation. A typical Spring service can accidentally create this behavior with perfectly reasonable local configuration: Java @Retry(name = "paymentService", fallbackMethod = "paymentFailed") public PaymentResult charge(PaymentRequest request) { return paymentClient.charge(request); } private PaymentResult paymentFailed(PaymentRequest request, Exception ex) { throw new PaymentUnavailableException(ex); } Nothing in this method indicates whether the incoming request has already consumed retries elsewhere. A gateway may already have retried the service, and paymentClient may apply another retry policy. Local resilience therefore becomes global amplification. A Retry Budget Changes the Decision A retry budget treats retries as limited capacity rather than an unconditional reaction to failure. Google documents two complementary controls in its overload handling: a per-request cap of three attempts and a per-client budget that permits retries only while retries remain below 10% of request traffic. In the example described by Google, the per-client budget reduces retry-driven traffic growth from almost three times the original request rate to roughly 1.1 times under the modeled overload condition. Finagle applies the same general idea through a shared RetryBudget, explicitly describing the budget as protection against the amplifying effect of many clients retrying. For a service chain, the useful abstraction is a request-scoped budget propagated with the operation. An internal header such as X-Retry-Budget can represent remaining retry permits. The header is an application convention rather than a standard HTTP field, its purpose is to ensure that downstream components consume from the same finite allowance. The retry decision can then become explicit: Java boolean canRetry(int remaining, HttpStatusCode status) { return remaining > 0 && (status.value() == 429 || status.is5xxServerError()); } int nextBudget(int remaining) { return Math.max(0, remaining - 1); } A caller starts a logical operation with a small budget, such as two retry permits. Every additional attempt decrements the value before forwarding the request. A downstream service receiving zero can still return a meaningful failure, but it cannot create more retry traffic for that logical operation. This model should not make every 5xx automatically retryable. Retry classification still matters. Validation failures, deterministic application errors, and non-idempotent operations can be unsafe or pointless to repeat. AWS recommends idempotent API contracts when operations may be retried and describes caller-provided request identifiers as a way to recognize duplicate intent. Propagating One Budget Across Service Boundaries Budget propagation belongs close to outbound transport logic so business methods do not manually manipulate retry metadata. A Spring interceptor can read the current budget and attach the decremented value to the next attempt: Java int remaining = retryContext.remaining(); if (remaining <= 0) { throw new RetryBudgetExhaustedException(); } request.getHeaders().set( "X-Retry-Budget", Integer.toString(remaining - 1) ); return execution.execute(request, body); The receiving service extracts the header once and places the value in the request context. Internal HTTP clients and RPC adapters then share that context. This is conceptually similar to distributed context propagation used by tracing systems. OpenTelemetry propagators inject and extract cross-cutting context through carriers such as HTTP headers, although retry-budget metadata can remain a dedicated internal header rather than telemetry baggage. A budget also needs to cooperate with deadlines. A remaining retry permit is useless when the logical request has only a few milliseconds left. Retry authorization should therefore require both budget and time: Java boolean retryAllowed(RetryContext context) { return context.remaining() > 0 && context.deadline().isAfter(Instant.now().plusMillis(100)) && context.lastFailure().isTransient(); } Server feedback should override generic retry enthusiasm. HTTP defines Retry-After so a service can indicate when a follow-up request should occur, including with 503 Service Unavailable, 429 Too Many Requests can also carry Retry-After. A budget answers whether another attempt is permitted, while server feedback helps decide when that attempt is appropriate. Measuring Whether the Budget Is Working Retry budgets are control mechanisms, so observability must expose both logical requests and physical attempts. Finagle distinguishes logical success from individual attempts and publishes metrics for retry budget availability, exhaustion, and request retry limits. Without that separation, retries can hide dependency instability because a successful second attempt makes the logical request appear healthy while infrastructure performs additional work. Useful telemetry should record the initial request count, retry attempt count, budget exhaustion count, retry success rate, response classification, remaining budget, and end-to-end latency. The critical ratio is retry amplification, which is total physical attempts divided by logical requests. A healthy value depends on workload characteristics, but a sharp increase during an incident indicates that resilience logic is becoming a load. Tracing adds the missing causal view. Each attempt can remain a child span of the same logical operation, with attributes such as retry.attempt, retry.remaining, and retry.reason. The resulting trace shows whether an operation failed because a dependency was unavailable, because the deadline expired, or because the shared budget prevented another attempt. That distinction is operationally important as budget exhaustion is often evidence that the system deliberately stopped adding pressure rather than evidence that the retry mechanism malfunctioned. Retry metrics also need to be interpreted alongside service saturation and rejection rates. A rising retry-success rate may initially indicate useful recovery from transient faults, but rising attempt volume combined with increasing backend saturation indicates a different condition. At that point, preserving capacity can be more valuable than pursuing another successful attempt. Google’s overload guidance explicitly recommends allowing failures to propagate when widespread backend overload makes additional retries unlikely to help. Conclusion Retries remain essential for transient failures, but retries without coordination can turn a partial outage into a traffic multiplier. Backoff, jitter, idempotency, deadlines, and server pushback address important parts of the problem that a retry budget adds the missing global constraint by limiting how much extra work one logical operation may create. Propagating that budget across service boundaries converts retry behavior from isolated local policy into distributed load control. The strongest resilience policy is therefore not “retry until success,” but “retry only while the failure is transient, the operation is safe, time remains, and the system can afford another attempt.”

By Uthej Mopathi CORE