Big Data

Rebuilding an Elasticsearch index from PostgreSQL sounds simple: Create a new index.Copy the rows that belong in search.Replay anything that changed during the copy.Verify the result.Move the search alias. The difficult part is deciding where the bulk copy ends and the replay begins. A transaction can update a row before the rebuild starts and commit after the copy has taken its database snapshot. With the wrong watermark, that change may appear in neither the copy nor the replay. The rebuild finishes successfully, but the new index is wrong. I built a controlled recovery drill to measure this problem. The test used synthetic data rather than information from a client or production system. The setup included: 500,000 PostgreSQL rows449,917 rows initially matching the search predicate2,000 updates, 250 deletes, and 120 inserts during each rebuildEight concurrent writer connectionsA two-node Elasticsearch 8.19.3 cluster I ran the timestamp-based strategy five times. It produced the same 366 measurable inconsistencies in every run. The corrected transaction-aware strategy was tested in 30 runs across several configurations. All 30 passed the implemented pre-cutover checks. The most surprising result was not the total number of errors. It was that a field-level comparison of 5,000 documents reported no stale content in any of the five defective runs. First, Confirm That a Rebuild Is Necessary Before rebuilding from PostgreSQL, confirm that the existing index cannot be recovered. Resolve the application alias, identify the physical index behind it, and inspect the affected shards. Elasticsearch’s cluster allocation explain API can show whether an unassigned shard is caused by disk pressure, a missing node, an allocation rule, or the absence of a valid shard copy. If a usable snapshot exists, restoring it into a separate versioned index may also be faster than rebuilding the full projection. The rest of this article assumes that allocation repair and snapshot restoration are not available. Unsafe Watermark 1: updated_at The first rebuild strategy used a timestamp as the replay boundary: SQL -- Record the replay boundary. SELECT clock_timestamp(); -- Copy the current search projection. SELECT id, title, body, status, version, updated_at FROM searchable_content WHERE status = 'PUBLISHED' ORDER BY id; -- Replay rows changed after the boundary. SELECT id, title, body, status, version, updated_at FROM searchable_content WHERE status = 'PUBLISHED' AND updated_at > :rebuild_started_at ORDER BY id; This approach assumes that updated_at represents commit order. In the drill, the value was assigned when the update statement ran, not when the transaction committed. Consider this sequence: Plain Text Session A: UPDATE row 1 updated_at = 17:23:46.252262 transaction remains open Session B: reads the rebuild watermark watermark = 17:23:46.465777 Session A: COMMIT The transaction committed after the watermark, but the row carried a timestamp from before the watermark. The bulk scan did not include the update because its PostgreSQL snapshot was created while the transaction was still open. The replay did not include it because the row failed this condition: SQL updated_at > :rebuild_started_at The change fell between the two phases. This issue is not limited to clock_timestamp(). A timestamp assigned by an application, database statement, or trigger may still be created before the surrounding transaction commits. If application servers assign the value, differences between their clocks introduce another source of error. Timestamp replay also has two simpler problems. First, a deleted row no longer exists in the source table. A query against the current rows cannot tell Elasticsearch that the old document must be removed. Second, a row may remain in PostgreSQL but stop matching the search predicate. In this experiment, a row could change from PUBLISHED to DRAFT. The replay query selected only published rows, so it did not remove the old Elasticsearch document. At the end of each timestamp-based run, the defects were: Defect classCountMissing from Elasticsearch3Deleted from PostgreSQL but still indexed228Outside the search predicate but still indexed131Present in both systems but stale4Total366 PostgreSQL contained 449,863 rows matching the final predicate, while Elasticsearch contained 450,219 documents. The net count difference was 356, which understated the real problem. Three missing documents offset part of the 359 unexpected documents, and the four stale documents did not change the count at all. Unsafe Watermark 2: MAX(seq) A transactional outbox solves part of the problem. A trigger can write an outbox event in the same transaction as every insert, update, or delete. Deletes now leave tombstones, so the replay process knows which Elasticsearch documents to remove. A first outbox design might use a BIGSERIAL sequence. Before starting the copy, the rebuild records the largest visible sequence value: SQL SELECT MAX(seq) FROM reindex_outbox; After the copy, it replays everything above that value: SQL SELECT seq, content_id, op FROM reindex_outbox WHERE seq > :starting_seq ORDER BY seq; This avoids clock problems, but it still does not provide a commit boundary. Consider two transactions: Plain Text T1: UPDATE row 1 outbox seq 51 allocated transaction remains open T2: UPDATE row 2 outbox seq 52 allocated COMMIT Rebuild process: SELECT MAX(seq) returns 52 T1: COMMIT When the rebuild reads MAX(seq), T1 is still invisible. T2 has committed, so the largest visible sequence is 52. T1 later commits with sequence 51. A replay using seq > 52 will never return the T1 event. PostgreSQL sequences generate distinct values, but they do not represent commit order. The PostgreSQL sequence documentation also explains that a value returned by nextval() is not reclaimed if its transaction later aborts. Sequence allocation and transaction visibility are separate mechanisms. Replacing the timestamp with a counter did not fix the underlying problem because both values were assigned before transaction visibility was settled. Figure 1. Timestamp and sequence-maximum watermarks can both miss a transaction that starts before the watermark and commits afterward. A Transaction-Aware Replay Window The corrected implementation records the PostgreSQL transaction ID in every outbox event: SQL CREATE TABLE reindex_outbox ( seq bigserial PRIMARY KEY, content_id bigint NOT NULL, op text NOT NULL CHECK (op IN ('upsert', 'delete')), xact_id bigint NOT NULL DEFAULT pg_current_xact_id()::text::bigint, captured_at timestamptz NOT NULL DEFAULT clock_timestamp() ); The outbox event is inserted in the same transaction as the source change. Before beginning the bulk scan, the rebuild records the current PostgreSQL snapshot’s xmin: SQL SELECT pg_snapshot_xmin( pg_current_snapshot() )::text::bigint; PostgreSQL defines snapshot xmin as the lowest transaction ID that was still active when the snapshot was taken. This gives the rebuild a conservative lower bound for transactions that may not be represented in the bulk-copy snapshot. The replay population begins with: SQL SELECT DISTINCT content_id FROM reindex_outbox WHERE xact_id >= :snapshot_xmin; This may include changes that the bulk scan already copied. That is acceptable because the Elasticsearch operations use stable document IDs and are designed to be idempotent. The snapshot tells the rebuild where replay should begin. It does not tell the rebuild when replay is complete. A Final Boundary Is Still Needed An outbox consumer may catch up while writes continue. It may even report zero lag for a moment. That does not close the recovery window because another transaction can commit immediately afterward. The drill therefore uses a final quiescent boundary: Capture the transaction-aware replay lower bound.Bulk-copy the source projection while writes continue.Drain visible outbox events.Stop accepting new source mutations.Allow transactions already in progress to finish.Drain the remaining events.Verify the new index.Move the alias and resume writes. The synthetic writer had a finite plan and had already completed by the time the final barrier was applied. The experiment therefore measured finalization after the writer became idle. It did not measure the delay that production requests would experience while blocked by an active write barrier. A system that cannot pause writes needs a durable stream handoff. PostgreSQL logical decoding, for example, can combine an exported snapshot with a persistent replication position. Do Not Use One Sequence Cursor for the Drain My first outbox drain repeated the sequence problem in another place. It used: SQL WHERE seq > :last_applied_seq Suppose sequence 52 becomes visible while the transaction holding sequence 51 remains open. If the consumer advances its cursor to 52, sequence 51 is skipped when it later commits. The corrected implementation records applied events individually: SQL SELECT o.seq, o.content_id FROM reindex_outbox o LEFT JOIN reindex_progress p ON p.rebuild_id = :rebuild_id AND p.seq = o.seq WHERE o.xact_id >= :snapshot_xmin AND p.seq IS NULL ORDER BY o.seq; This is an applied-event ledger rather than one high-water mark. A late commit cannot appear below a cursor that has already moved past it. The replay also reads the current PostgreSQL state instead of trusting an old document payload stored in the outbox: If the row currently matches the search predicate, it is upserted.If the row was deleted, the Elasticsearch document is deleted.If the row still exists but no longer matches the predicate, the document is also deleted. Several events for the same source ID therefore converge on its current state. An outbox event is marked as applied only after its Elasticsearch operation succeeds. Unresolved operations remain unapplied and cause the cutover to fail. The Elasticsearch Bulk API also requires item-level checking. A request can return HTTP 200 while individual operations fail and the response contains errors: true. In a separate overload test, 525 rejected document operations appeared inside 22 HTTP-200 responses. Checking only the HTTP status would have reported those requests as successful. Why a 5,000-Document Sample Reported a Clean Result I tested three lightweight reconciliation methods against the timestamp-based rebuild: Counts divided into six ID rangesChecks against the 250 IDs selected for deletionA field-level comparison over 5,000 randomly selected source records The first two checks found problems immediately. All six count ranges differed, and 228 of the 250 deleted IDs were still present in Elasticsearch. The sampled content comparison found none of the four stale documents in any of the five runs. The full-corpus comparison found all four. For a population of N comparable documents containing s stale records, and a sample of n documents selected without replacement, the probability of finding at least one stale record is: Plain Text P(detect at least one) = 1 - C(N - s, n) / C(N, n) For this experiment: Plain Text N = 449,860 documents present in both systems s = 4 stale documents n = 5,000 sampled documents The probability of finding at least one stale document was approximately 4.37%. To reach a 95% detection probability, the sample would need 237,134 documents, or 52.7% of the comparable corpus. Reaching 99% would require 307,601 documents, or 68.4%. Figure 2. Detection probability as the sample size increases when four of 449,860 comparable documents are stale. This does not mean sampling is useless. A 5,000-document sample can quickly reveal a widespread transformation error. Sampling can also be useful as a regular monitoring check between rebuilds. It is weak as a cutover gate for rare defects. Transaction-boundary races may affect only a few records inside an otherwise reasonable-looking index. That is exactly the situation in which a modest random sample has little statistical power. What the Full-Corpus Check Compared The full comparison checked: The complete expected set of document IDsA digest over id, title, body, status, and version Each field was length-prefixed before hashing so that field boundaries could not be confused. PostgreSQL calculated the source-side digest. On the Elasticsearch side, the verifier read the actual fields from _source and calculated the digest again. The drill uses MD5 as a compact equality checksum, not as a security mechanism. The digest excludes updated_at, whose representation was not canonicalized in the same way on both sides. The result was: Reconciliation resultTimestamp rebuildTransaction-aware rebuildMissing from Elasticsearch30Deleted source rows still indexed2280Predicate-excluded rows still indexed1310Stale in 5,000-document sample00Stale in full-corpus comparison40Total measured inconsistencies3660 The digest check verifies the document fields included in this projection. It does not verify mappings, analyzers, synonyms, autocomplete, facets, relevance, or authorization rules. Those require functional search tests before the alias moves. For a much larger index, the comparison should stream sorted ID-and-digest pairs or compare deterministic partitions instead of loading everything into memory. A Practical Pre-Cutover Gate A rebuild should not move the alias until all of these conditions are true: The expected and actual document IDs match.The full-corpus content digests match.No committed outbox event in the recovery window remains unapplied.No Elasticsearch operation remains unresolved.The target index has the required active shards.Representative search canaries pass. The canaries should test the behavior users depend on, not only whether one document exists. Depending on the application, they may include: A known search queryAn autocomplete requestA filter or facetSpecial-character or multilingual contentA tenant or authorization boundary Elasticsearch supports removing the old alias target and adding the new target in one alias update operation. The response should still be checked, and the previous index should be retained until the rollback window has passed. It is also important to measure through verification. In this experiment, final drain, shard preparation, and full-corpus verification added roughly 6.5 to 13.8 seconds after the main rebuild work. A benchmark that stops at the last bulk request does not measure when the index is actually ready. Limitations This was a controlled experiment using synthetic data. The source contained one table, one search predicate, and no joins or enrichment. The main benchmark used one primary shard. Both Elasticsearch nodes ran inside one Podman virtual machine on an Apple M4 Max laptop. This allowed a replica to be allocated, but it did not represent separate machines, storage devices, network paths, or failure domains. The workload plan was seeded, so the operations and target IDs repeated across runs. Thread scheduling, timing, and throughput did not. The writer completed before the final barrier was activated, so the experiment measured quiescent finalization rather than the delay experienced by blocked production writes. The content digest covered five fields and excluded updated_at. It verified document content, not mappings or search behavior. Finally, snapshot xmin is a conservative recovery boundary, not a replacement for a mature change data capture system. Long-running transactions can widen the replay range. Outbox events must be retained for the full rebuild window, and every relevant source mutation path must participate in the same transactional capture process. Conclusion The timestamp watermark failed because the row timestamp was assigned before its transaction committed. MAX(seq) failed for the same reason: sequence allocation did not represent commit order. The reconciliation test exposed a separate weakness. Checking 5,000 documents sounded substantial, but it provided only a 4.37% chance of finding the four stale records created by this workload. A reliable rebuild needs two guarantees: A replay window based on transaction visibility rather than statement timingA complete pre-cutover comparison capable of finding rare inconsistencies Without both, the rebuild can complete, the document count can look reasonable, and the new index can still disagree with PostgreSQL. Companion Materials The synthetic workload, implementation, figures, and per-run results are available in the dzone-recovery-drill-v1.1 companion repository.

Embedded and IoT devices operate in environments where every kilobyte of memory and every millisecond of response time matters. Yet most of them still rely on C or C++, languages powerful enough for low-level control but notorious for memory-safety flaws that can crash systems or expose them to security vulnerabilities. The issue is so widespread that around 70% of Microsoft’s and Google’s reported vulnerabilities stem from memory-safety issues. For connected devices running critical operations, such instability isn’t an option. However, a structured C++ to Rust migration addresses the root cause by replacing unsafe memory handling with compile-time guarantees and predictable real-time performance. Embedded environments leave little room for trial and error; therefore, experience and a clear roadmap make a big difference. For this purpose, companies often work with teams skilled in Rust development services and familiar with embedded software development. With the right expertise, you can introduce new Rust components gradually and safely without interruptions to the existing IoT ecosystem. Now, let’s discover how to migrate from C/C++ to Rust, review the migration challenges and best practices to mitigate them, and analyze a success story of practical Rust migration for embedded systems. Why Consider Migrating From C/C++ to Rust? Modern embedded systems, especially IoT-based medical devices, put heavy pressure on safety, energy use, and long-term maintenance. Rust handles these demands better than traditional C or C++ thanks to its built-in memory safety, reliable concurrency model, and predictable real-time behavior. When organizations consider a C++ to Rust migration, these advantages quickly become deciding factors for modernizing connected devices. They also translate into more stable products and lower life cycle costs: Reduced security risks: Rust’s memory safety drastically lowers vulnerabilities, meaning fewer patches and less downtime for IoT systems.Lower long-term maintenance costs: Rust reduces bugs that emerge later in production, leading to fewer expensive fixes after deployment.Improved reliability: With Rust, IoT devices crash less often, which means more uptime and fewer disruptions for end-users.Better performance: Rust’s efficiency can lead to lower energy consumption, extending the life of battery-powered IoT devices and reducing energy costs.Faster time-to-market for updates: Rust’s safe concurrency allows quicker development of new features without compromising stability, which means you can roll out improvements to your IoT solutions faster.Future-proof technology: Rust’s growing popularity means a sustainable ecosystem, which ensures that your IoT product stays modern and well-supported for years to come.Increased development velocity: Rust reduces debugging time and makes the code easier to reason about, speeding up onboarding and increasing overall developer productivity. Three Common Myths About Rust Migration Many organizations hesitate to begin migrating embedded systems from С or C++ to Rust because they expect major risks or disruptions. In reality, most of these concerns stem from common misconceptions rather than technical limitations in embedded software development. Understanding what Rust migration actually involves helps teams make confident, well-informed decisions. Let’s review the three most common misconceptions. Myth 1: “We’ll Have to Rewrite Everything” You don’t need a full rewrite to begin a C++ to Rust migration. Many teams start small, moving one or two isolated modules that bring the most maintenance pain. Using the Foreign Function Interface (FFI), Rust can work alongside C or C++ within the same firmware. This feature makes it possible to modernize gradually while the rest of the system stays live. Myth 2: “Switching to Rust Will Interrupt Delivery” Migration can happen without stopping production. With a structured approach, Rust modules are added one by one while existing CI/CD pipelines continue to run. Each new piece is tested, deployed, and benchmarked before the next one is introduced. The development cycle continues as usual; only the language inside changes. Myth 3: “We’ll Have to Replace the Entire Engineering Team” Most C or C++ developers adapt to Rust faster than expected. The languages share similar low-level logic, which makes the learning curve manageable with proper mentoring and review. Once trained, the same team can handle both Rust and C++, maintaining delivery speed while building long-term expertise. Three Ways of C/C++ to Rust Migration There’s no one-size-fits-all approach to C++ to Rust migration. The right strategy depends on how your system works today, how critical uptime is, and what level of modernization you’re aiming for. When adopting Rust for IoT and embedded systems, companies typically choose one of three paths: gradual integration, full rewrite, or hybrid migration. 1. Gradual Integration This path is ideal for safety-critical or continuously running devices. Instead of rewriting everything, the developers introduce Rust gradually — one module at a time. Typical starting points include telemetry handlers, utility modules, or background services that can be tested in isolation. The existing C/C++ software keeps running while new Rust components are integrated and validated. Over time, as more modules are proven stable, Rust takes over a larger share of functionality. 2. Full Rewrite If you’re launching a new product or phasing out outdated firmware, a full rewrite may be the most efficient route. The development starts with Rust from day one. It covers firmware logic, communication layers, and testing infrastructure. This scenario removes legacy constraints entirely and lets your team build clean architectures without legacy dependencies, design for memory safety from the start, and use Rust’s concurrency model to its full advantage. A full rewrite requires a skilled team, proven toolchains, and time for onboarding, but it also gives you complete control over the system’s design. Yalantis supports such transformations through end-to-end Internet of Things solutions, helping businesses develop secure, scalable, and future-ready connected products. 3. Hybrid Approach Many embedded systems can’t be fully rewritten due to vendor SDKs, licensed components, or hardware dependencies. In those cases, Rust and C/C++ coexist. Engineers connect both through FFI, so each language can do what it does best. C++ handles hardware-level operations, while Rust manages business logic, safety-critical processes, or communication layers. This model works particularly well for medical devices, industrial equipment, and long-lifecycle IoT platforms, where reliability and compliance matter just as much as modernization speed. Challenges and Best Practices by Yalantis for C/C++ to Rust Migration Before starting a full-scale migration, it’s important to understand the technical and architectural risks involved. Each challenge that appears during the embedded C++ to Rust migration can be handled through structured engineering practices. Below is an overview of key challenges in migrating embedded firmware from C++ to Rust and the Rust migration best practices Yalantis uses to keep modernization predictable and low-risk. Challenge 1: Maintaining Compatibility With Existing C/C++ Code Legacy firmware often has C++ modules woven tightly together. You can’t rewrite everything at once without risking device instability or unexpected regressions. Challenge 2: Operating Under Hardware and Microcontroller Limitations Most embedded devices run on minimal RAM, limited storage, and low CPU frequencies. A rushed migration can overload the hardware or increase latency. Challenge 3: Ensuring Deterministic Behavior in Real-Time Systems IoT and embedded applications often have strict timing and reliability requirements. Even slight delays in execution can affect safety, user experience, or device coordination. Challenge 4: Integrating With External Libraries and APIs Embedded firmware relies on vendor drivers, communication stacks, and low-level APIs. Poor integration planning can cause instability or break device compatibility. Conclusion Modernizing embedded and IoT systems requires predictable performance, strong security guarantees, and code that stays maintainable as the product evolves. Traditional C/C++ firmware often struggles with these demands. Rust solves many of these challenges upfront with built-in memory safety, reliable concurrency, deterministic execution, and a modern development ecosystem that reduces defects and speeds up delivery. A well-planned C/C++ to Rust migration strengthens product stability, lowers lifecycle costs, and gives you a foundation that scales with new device architectures and connectivity needs.

Every Kafka tutorial covers idempotency the same way: "Store the event ID. Before processing, check if it exists. If it does, skip. Done." That advice is correct. It is also incomplete in a way that will surface only under load — and only in production, at the moment you least want a surprise. Here is what actually happens inside a multi-replica consumer group when two delivery attempts arrive at the same instant. The Pattern You Know A Kafka topic configured with acks=all and enable.idempotence=true on the producer side still delivers messages at least once. Network partitions, broker leader elections, retry storms — any of these can cause a consumer to receive the same message twice. The standard fix is to deduplicate on the consumer side using a stable event ID included in every message payload. The implementation looks like this: Java // Application-layer idempotency check if (processedEventRepository.existsByEventId(eventId)) { log.warn("Duplicate event — skipping | eventId: {}", eventId); return false; } // Process and record processedEventRepository.save(ProcessedEvent.builder() .eventId(eventId) // ... .build()); Under normal operation, this is correct. One consumer thread, one message, one check, one save. The second delivery hits the existsByEventId guard and is dropped cleanly. The problem is that a Kafka consumer group under load does not serialize delivery at the event level. The Race Condition You Don't See in the Tutorial Kafka's partition assignment guarantees that only one consumer in a group receives messages from a given partition at a given time. What it does not guarantee is that the same event ID cannot arrive via two different partitions — for example, during a producer retry that sends to a different partition — or that two messages with the same semantics cannot appear if upstream logic emits duplicates before the broker sees them. More concretely: if you run two replicas of your consumer service (and in Kubernetes, you will), two pods can receive the same event in the same wall-clock window. The sequence looks like this: Plain Text T=0 Pod A receives event e1, calls existsByEventId("e1") → false (not yet in DB) T=0 Pod B receives event e1, calls existsByEventId("e1") → false (not yet in DB) T=1 Pod A passes the guard, begins processing T=1 Pod B passes the guard, begins processing T=2 Pod A saves ProcessedEvent("e1") → INSERT succeeds T=2 Pod B saves ProcessedEvent("e1") → ??? Both pods read false from the application-layer check before either has written. Both proceed. Without a database-level guardrail, both complete, and you have sent the welcome email — or charged the card, or incremented the counter — twice. The Fix: The Database as the Serialization Point The application-layer check is an optimization — it avoids a write attempt for the obvious duplicate case. It is not, and cannot be, a correctness guarantee, because it is not atomic with the write. The correctness guarantee has to live at the layer that is atomic: the database. The schema includes a UNIQUE constraint on event_id: SQL CREATE TABLE processed_events ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), event_id VARCHAR(255) NOT NULL UNIQUE, -- the guarantee is here event_type VARCHAR(100) NOT NULL, processed_at TIMESTAMP NOT NULL DEFAULT now(), topic VARCHAR(255) NOT NULL, partition_number INT NOT NULL, offset_number BIGINT NOT NULL ); That UNIQUE constraint is enforced at the storage engine level — not at the application level, not at the ORM level. No two rows with the same event_id can be committed, regardless of how many concurrent writers attempt it. The database serializes the race. What changes in the application is how you handle the loser of that race: Java @Transactional public boolean sendWelcomeNotification(UserCreatedEvent event, ...) { String eventId = event.getEventId().toString(); // Fast path: skip if already recorded (optimization, not guarantee) if (processedEventRepository.existsByEventId(eventId)) { log.warn("Duplicate event detected — skipping | eventId: {}", eventId); notificationMetricsService.incrementDuplicate(); return false; } try { // ... process the event ... processedEventRepository.save(ProcessedEvent.builder() .eventId(eventId) .eventType("USER_CREATED") .processedAt(LocalDateTime.now()) .topic(topic) .partitionNumber(partition) .offsetNumber(offset) .build()); return true; } catch (DataIntegrityViolationException ex) { // Two threads passed the application check simultaneously. // The DB constraint caught one of them. This is correct behavior. log.warn("Duplicate event caught at DB level — skipping | eventId: {} | userId: {}", eventId, event.getId()); notificationMetricsService.incrementDuplicate(); return false; } } The DataIntegrityViolationException is Spring's translation of a constraint violation from any supported database (PostgreSQL, MySQL, Oracle). Catching it and suppressing it is not an error condition — it is the correct outcome of a race that the DB resolved correctly. Note that the exception is caught in the service layer, not in the Kafka listener. If it propagated up to the listener, Spring Kafka's retry mechanism would re-deliver the event, triggering more unnecessary attempts. Catching it at the service boundary terminates the race cleanly. Two Layers, One Correct Answer The full pattern has two distinct roles: LayerRoleWhat it stopsApplication check (existsByEventId)OptimizationThe non-concurrent duplicate: the obvious second delivery hours or days laterDB UNIQUE constraint + DataIntegrityViolationExceptionCorrectness guaranteeThe concurrent duplicate: two threads, two replicas, same event, same instant Removing either layer has a cost. Without the application check, every retry hits a write attempt that will fail, generating unnecessary DB load and constraint violations in your logs. Without the DB constraint, the application check is a promise with no enforcement — and distributed systems do not honor promises. The right posture is to treat the application check as a performance optimization and the database constraint as the correctness invariant. What This Looks Like Under Fault Injection I verified this pattern against deliberate duplicate delivery in a Kafka consumer running two Spring Boot replicas. Injecting two simultaneous deliveries of the same eventId produced exactly one SENT notification log and one SKIPPED log (the DataIntegrityViolationException path). No double side effects, no exceptions visible to the retry mechanism, no compensating logic required. The database caught the race that the application check could not. Key Takeaways Application-layer idempotency checks are not atomic with the writes they guard. Under concurrent delivery, both threads can pass the check before either writes.The database's UNIQUE constraint is the actual serialization point. It is enforced at a level that the application cannot race.DataIntegrityViolationException is a success path, not an error path. It means the DB caught a concurrent duplicate — suppress it, do not propagate it.You need both layers. The application check eliminates unnecessary write attempts. The DB constraint enforces the invariant. Neither is sufficient alone. If your Kafka consumer only has the application check, you have an idempotency aspiration, not an idempotency guarantee. The race condition described here may never surface on a single-replica dev environment. It will surface on a multi-replica production deployment, under the load conditions that make that second replica necessary in the first place. The implementation described in this article is part of a production-grade event-driven microservices platform built with Spring Boot 4, Java 21, and Kafka. The full source is available at github.com/Rummy43/ai-microservices-platform.

Kestra is an open-source workflow orchestration platform that unifies IT scheduling, data pipelines, business processes, infrastructure automation, and AI agents under one declarative, event-driven control plane. As enterprises modernize legacy systems and adopt agentic AI, unified enterprise workflow orchestration is becoming increasingly important. Here is why I joined Kestra as Global Field CTO and why I believe the orchestration market is approaching a turning point. Orchestration is the connective tissue of the modern enterprise, and right now nobody owns it. That is the gap I am joining Kestra to help close. The market is at the same inflection point Apache Kafka was at when I joined Confluent in 2017. The technology is mature, already in production at Apple, JPMorgan, Toyota, and Xiaomi. The commercial story is under-narrated. And there is a two-to-three-year window to define who owns the category before it closes. If you want the backstory on the nine years that led here, including the journey from a Kafka startup to an IBM acquisition, I wrote about it separately in My Confluent Chapter: From Apache Kafka Startup to $11 Billion IBM Acquisition. Enterprise Workflow Orchestration: The Problem With Disconnected Tools Every large enterprise runs four kinds of orchestration tools that do not talk to each other. Legacy IT schedulers like Control-M, ActiveBatch, and cron run date-driven batch jobs with no event awareness. Data pipeline tools like Airflow, Prefect, and Dagster move data, but stay scoped to Python DAGs. Business process platforms like Camunda, Appian, Pega, and UiPath handle approvals and human steps, but stop at the process boundary. Infrastructure automation tools like vRA, ServiceNow, and Terraform provision resources, but stay locked into ITSM lifecycles. Each tool owns one slice. None of them owns the full picture. Running these tools separately was manageable when each stayed in its lane. Two forces are changing that: enterprises are actively modernizing these tools, and agentic AI is forcing them to work as one. Why Enterprise Orchestration Matters for Agentic AI The first force is modernization, and it is where most enterprises actually start. They are retiring brittle, expensive legacy schedulers. Consolidating data pipeline sprawl across teams. Replacing rigid BPM suites that stop at the process boundary. Each of those is a funded project on its own. And once you are modernizing one category, the obvious question follows: why run four engines when one declarative platform covers all four? The second force is agentic AI, and it removes any remaining reason to keep the tools apart. AI workflows refuse to stay inside a single tool. One execution can touch a data pipeline, trigger a process step, call a model, and provision infrastructure. When it fails, it often fails quietly in the gap between tools. The CTO sees governance risk. The CIO sees infrastructure that cannot keep up. The CDO sees stale data. The COO sees a process that stalled for no visible reason. Four executives, four symptoms, one root cause: no single layer governs the full mix of workloads. Kestra takes all four categories onto one declarative platform, under one governance model. Instead of a patchwork of disconnected tools, one engine orchestrates across the domains an enterprise depends on. Deterministic workflows and autonomous agents, running side by side, under one control plane. Kestra: Open-Source Orchestration for Cloud, On-Premises, and Edge Kestra is open source under the Apache 2.0 license, the same permissive license as Apache Kafka. Developers and data engineers can start in minutes, building workflows in YAML, through APIs, or with the no-code editor, with a plugin ecosystem of 1,500+ integrations and tasks that run code in any language (Python, Node.js, R, Go, Shell, and more). Workflows can even be generated by an LLM via MCP, so you describe the orchestration in natural language and let the model build the flow. The platform is language-agnostic and event-driven by design. For production, there is an enterprise edition and a fully managed cloud, with the governance, scale, and SLAs that regulated organizations require built in rather than bolted on, including air-gapped deployment for those that cannot let a single byte leave their perimeter. Proven in Production at Apple, JPMorgan, Toyota, Xiaomi, and Many Other Enterprises Across Public and Private Industries Kestra runs in production well beyond early adopters. Apple's machine learning team uses it to orchestrate large-scale data pipelines. A senior engineering manager there put it plainly: "I want to highlight their robustness, which is crucial at our scale." Few orchestration tools earn that kind of statement from a team operating at Apple's level in AI and ML. The enterprise logos span industries and regions: Bloomberg, Crédit Agricole, JPMorgan, Toyota, T-Systems, and Xiaomi among them. Behind those production deployments sits a fast-growing open-source community, with execution volume up roughly 20x year over year. The growth curve is the same shape I watched at Confluent. Community adoption first, enterprise gravity following close behind. Enterprise Workflow Orchestration in the Age of AI I do not join companies for incremental bets. I expect Kestra to be a high-growth startup operating at enterprise scale, with the largest companies in the world adopting it across business units for a widening set of use cases. That is not optimism. It is where the demand is already heading. Open, scalable, reliable, and flexible workflow orchestration is needed everywhere. Every business unit runs processes that have to be triggered, coordinated, retried, audited, and governed. Finance, operations, data, platform engineering, customer-facing teams: each has its own workflows, and today each reaches for a different tool. One platform that serves all of them is a large and durable market, not a niche. Agentic AI raises the stakes further. It does not reduce the need for orchestration; it multiplies it. Every agent that takes an action needs a governed, observable workflow underneath it, and those workflows cut across exactly the business-unit boundaries that today's fragmented orchestration tools cannot span. The more AI an enterprise adopts, the more it needs one control plane across every unit. That is the wave Kestra is built to ride. Kestra Architecture: Database and Apache Kafka Backends Start simple. Run Kestra on a standard database and never think about Kafka at all. For many workloads, that is all you need. The platform is event-driven from the ground up, and the database backend handles low- and medium-scale with no extra moving parts. After nine years close to Apache Kafka, people expect me to lead with the Kafka angle. So let me deal with it directly, because there is a real and underappreciated opportunity here. But it is an option, not a requirement. I have seen too many teams force the wrong tool into orchestration. Some use Kafka Streams or Flink to orchestrate workflows. In most cases, that is an anti-pattern. Neither was built for it. Others bolt on yet another complex system just to get scale. Others hack together Python scripts and hope they hold. Kestra takes a cleaner path. Run it on a traditional database for low and medium scale. Switch to an existing Kafka deployment when you need larger throughput. Light when you want it light, Kafka-native when you need the scale. For the thousands of enterprises that already run Kafka as their nervous system, that is a natural fit. Everyone else can stay on the database. Today that means database or Kafka. With the re-architected core in Kestra 2.0 (coming soon), queue and database become fully independent and pluggable, adding support for Redis, AMQP, and cloud-native messaging from AWS, GCP, and Azure. The orchestration layer adapts to your infrastructure rather than forcing you onto theirs. But Kafka is one entry point, not the destination. The bigger opportunity sits in everything beyond the place I came from: legacy scheduler modernization, data and AI pipelines, and business process orchestration across the enterprise. Why I Joined Kestra as Global Field CTO Global Field CTO is the external technology and business voice of Kestra to customers, partners, analysts, press, and the practitioner community. The job is to help Kestra grow across these orchestration domains, regions, and the community building on it. I have written before about the daily life of a Field CTO, and the shape of the role carries over directly. In practice, that means three things. Working with enterprise CIOs and CTOs to figure out where orchestration fits in their architecture, in briefings and roundtables. Briefing analysts like Gartner, Forrester, and IDC on the customer patterns and market shifts I see. And doing the public work: reference architectures, conference keynotes, competitive positioning, and content that maps the technology to concrete use cases across financial services, insurance, manufacturing, automotive, telco, retail, logistics, and technology. That last part is where I have spent years building signature assets. The landscape reports, the industry guides, the books. I will keep producing them, now with orchestration as the connecting layer across data integration, process intelligence, and agentic AI. I Recommend the Right Architecture, Including When It Is Not Kestra One thing will not change from how I worked before. I recommend the right architecture, even when Kestra is not the answer. Orchestration rarely lives alone. A company might run Celonis for process mining to understand its operations, and Kestra for process orchestration at scale once agentic AI starts acting on those insights. Another might keep Camunda for human-centric, BPMN-modeled approval workflows, and use Kestra for the technical orchestration underneath: the data pipelines, integrations, and agent steps those processes depend on. The two are complementary in both cases. My job is to be clear about where Kestra fits, where it does not, and how it works alongside the rest of the stack. Credibility is what makes this role effective. I intend to keep mine. Europe Today, the US Next Kestra has its deepest enterprise traction in Europe today. The next move is the United States, where the company is expanding strategically over the coming months and opening a New York presence. I will support that push heavily. Customers in Asia and Australia are already in production, and the Middle East is a deliberate next step. Let's Build This Together There is a lot of ground to cover, and a lot of flights ahead to meet customers, partners, and the broader open-source community in person. I am excited to get started. If you are wrestling with orchestration across data, processes, infrastructure, or agentic AI, I want to be in that conversation. The simplest way: pull me into your architecture discussions. Reach out, and let's work through the hard problems together.

By Kai Wähner CORE

If you are a data engineer, you have probably come across this situation where you have to decide between two approaches for refreshing enterprise data. One approach focuses more on speed and less on compute, while the other focuses more on reliability and data correctness. Usually, when we talk about data refresh, the two options that come to mind are incremental load and full load, and the question becomes which one we should choose. In my opinion, this is not necessarily a choice where we have to pick one approach over the other. Both approaches have advantages and disadvantages, and rather than relying on one approach, I believe we can combine them and use each where it makes the most sense. I learned this firsthand when I was tasked with building a data warehouse from scratch for my organization, and one of the most important decisions I had to make, apart from the data model and schema, was the refresh strategy. Why I Didn't Want to Pick Only One Approach Incremental loading is the modern approach and is commonly used in enterprise environments. Instead of loading the entire table every time the refresh runs, incremental loading focuses only on records added or modified since the previous refresh. This significantly reduces the data to process, resulting in faster refreshes and lower compute usage. However, incremental loading also has disadvantages, especially for historical data. Incremental loading assumes we can correctly identify all changes in the source system. For example, if a new record is added or an existing record is updated, we can generally identify those changes using a timestamp, watermark, or change data capture mechanism. The problem comes when a historical record is deleted from the source system. If our incremental process is only looking for new and updated records, then the deletion may never be captured. The source system may no longer have that record, but the data warehouse will continue to have it. This can result in the data warehouse slowly becoming different from the source system without necessarily causing the ETL process to fail. I consider this one of the biggest weaknesses of an incremental approach. The process can run successfully, but the data can still be incorrect. A full load, on the other hand, addresses this problem more simply. We can essentially reload the data from the source system and make sure that the warehouse reflects what is currently available in the source. This makes full load more reliable for historical changes and deletions. However, full load has its own problem. As the number of records increases from thousands to millions or even hundreds of millions, loading the entire table every time becomes expensive and time-consuming. A process that initially took a few minutes can eventually take hours, and running it every night may not be practical from a compute or maintenance-window perspective. So, in my opinion, neither approach is perfect. Where I Ran into This Problem Myself I was tasked with creating a data warehouse for my organization. When I investigated the existing environment, I found that the data I would be working with spanned millions of records across multiple tables, with a large number of columns. This is common when working with enterprise-level data, but it becomes an important consideration when designing a data warehouse and deciding how to refresh it. Prior to the launch of this data warehouse project, the data was stored in normalized tables coming directly from the ERP database. The data itself was good, but it was not very usable for a business intelligence developer or a data analyst. The main problem was that there was no one common place where an analyst could get all the information they needed. They had to work with multiple normalized tables and understand how those tables were related before they could even start building their analysis or dashboards. This was one of the main reasons the data warehouse project mattered. Once we kicked off the project, one important decision I had to make, besides the data model and schema, was the refresh strategy. I had two conventional options. The first option was incremental load, which is the modern approach where the primary focus is on faster refreshes and less compute. The second option was the traditional full load, where the primary focus is on reliability and data correctness, but it takes more time and compute because all the records are loaded every time the refresh runs instead of only loading the new and updated records. Now, in my opinion, both approaches have their own advantages and disadvantages. The modern approach is to employ incremental refresh, and this is what most enterprises prefer because of the faster refresh times and reduced compute. But I thought a little differently. Although incremental refresh has an advantage when it comes to speed and compute, the traditional full load also has some important advantages. It provides more reliability when it comes to data correctness, especially when historical records are deleted from the source system. Full load is also conceptually simpler because you do not have to build and maintain complex logic to identify every possible type of change that can happen in the source system. So instead of choosing between the two, I thought of employing a third approach. The Hybrid Approach The approach I decided to use was a hybrid approach, which meant combining incremental and full load refreshes instead of relying completely on either one. The idea was relatively simple. On weekdays, I used incremental refresh because business users and analysts are actively working with the data, and faster refreshes matter more. The incremental process would capture the latest and most frequently changing data without having to reload millions of records every time. Then, during the weekend, I would run a full load or reconciliation process to capture changes in historical data that the incremental process may not have been able to identify, especially deletions. This allowed me to achieve both objectives. I could have faster refreshes during the business days while still having a mechanism that periodically verifies that the data warehouse is aligned with the source system. In other words, instead of asking whether incremental or full load is better, I looked at where each approach was better and used both approaches accordingly. How the Hybrid Approach Works The approach can be generalized into a few steps. 1. Monday to Friday, Run Incremental Loads During weekdays (Monday to Friday), the ETL process retrieves only the records that have changed since the previous refresh. This can be done using Change Data Capture, a watermark column, or another reliable mechanism available in the source system. For example, if the source system has millions of records but only a few thousand records change during the day, there is no reason to process all the millions of records again. We can identify the changed records and process only those records. 2. Make the Process Idempotent Another important consideration is making sure that the ETL process can safely be executed more than once. For example, if an ETL job fails halfway through and we have to restart it, we should not end up inserting duplicate records into the data warehouse. This is where approaches such as MERGE-based upserts can be useful, although they need to be implemented carefully, especially in environments where multiple processes can update the same records concurrently. 3. Run a Full Reconciliation During the Weekend On Saturday or Sunday, when activity is generally lower, we can run a more comprehensive reconciliation against the source system. For smaller dimension tables, a complete truncate-and-reload may be perfectly reasonable. For very large fact tables, however, I would not necessarily recommend blindly truncating and reloading the entire table. Instead, we can compare the warehouse with the source and identify differences between the two. This is especially important for identifying historical records deleted from the source system. 4. Track What the Weekend Process Had to Correct I also think it is important not to treat the weekend reconciliation simply as another ETL process. Every time the full reconciliation finds a difference, that difference tells us something about our incremental process. For example, if the weekend process finds 100 deleted records that the incremental process did not capture, that is useful information. If that number continues increasing every week, it may indicate that there is a problem with the incremental logic or the way changes are being captured from the source. Therefore, the full load can also act as a validation mechanism for the incremental process. 5. Communicate the Refresh Window Finally, communicate when the full reconciliation will happen. If reports or dashboards are connected to the warehouse, users should know that the data may temporarily look different while the reconciliation or reload is happening. This should be treated as a planned maintenance activity rather than something unexpected. What This Approach Delivered Going into the project, I initially thought that I would have to accept one weakness or the other. Either I could use incremental refresh and get faster refreshes but accept that some historical changes, especially deletions, might not be captured. Or I could use full load and have better data correctness but accept longer refresh times and higher compute costs. The hybrid approach allowed me to avoid that trade-off every day. The warehouse could remain fast and relatively inexpensive to refresh during the business days, while the weekend reconciliation provided an additional layer of reliability and data correctness. Most importantly, historical changes that an incremental-only approach could potentially miss were given another opportunity to be captured during the weekend process. This made the refresh strategy more resilient because I was not relying on one mechanism to solve every possible problem. Where Would I Go From Here? In my opinion, the question should not be whether incremental load or full load is better. The better question is how we can use both approaches together to get the advantages of each. If your data warehouse currently relies completely on incremental refresh, you don't necessarily have to redesign the entire architecture. Start small. Take your highest-risk fact table and introduce a weekend reconciliation process alongside the existing incremental process. Then track how many records that reconciliation has to correct. That number can tell you a lot about the reliability of your incremental process. If the number is close to zero every week, that gives you confidence that your incremental process is working well. If the number keeps increasing, you have identified an area where the incremental logic needs improvement. For me, this is what made the hybrid approach interesting. It was not simply about making the data warehouse faster or cheaper. It was about finding a practical balance between performance, compute, reliability, and data correctness. Rather than choosing one approach and accepting its weaknesses, I chose to use each approach for what it was actually good at.

A Kafka recovery often looks successful before it actually is. The replay job exits. Consumer lag drops. Error rate returns to normal. The dashboard is green enough for everyone to want the incident closed. Then somebody checks the downstream projection, account balance, inventory state, enrichment table, or audit report and finds the real problem: data moved, but state did not converge. That is the failure this article is about. Consumer lag is a progress signal, not a recovery verdict. It proves position movement. It does not prove terminal outcomes, idempotent side effects, owned failures, or source-to-target convergence. If an incident review ends with “lag went back to zero,” the team has proved movement, not recovery. Declare the Recovery Cohort First The first observability decision is not which metric to graph. It is what work the recovery run is supposed to prove. A useful cohort names: the topic, partitions, and offset rangethe event types included in the runthe business-key scopethe expected record countthe recovery reasonthe ownerand the stop conditions A compact contract object makes that scope reviewable: YAML recovery_run: id: rec-2026-09-05-01 reason: projection_drift_after_consumer_restart owner: billing-platform-oncall run_type: incident_repair source: topic: billing-events event_types: - InvoicePosted - CreditApplied offsets: - partition: 8 from: 492100 to: 492881 - partition: 9 from: 388010 to: 388522 business_scope: key_type: account_id selector: incident_account_hashes_2026_09_05 expectations: expected_record_count: 1294 reconciliation: source_of_truth: billing_ledger checks: - account_day_balance_matches - no_missing_decision_events - no_unowned_final_failures safety: dry_run_required: true max_retry_attempts: 3 stop_on_reconciliation_mismatch: true The object does not have to be beautiful. It has to be unambiguous. During an incident, ambiguity is expensive: “replay the billing topic” is not a scope, and “check the dashboard” is not evidence. The cohort also forces a design decision: do you need partition/offset proof, business-key proof, or both? Offsets are excellent for Kafka lineage. Business keys are what downstream systems usually reconcile. A serious recovery plan connects them instead of pretending one replaces the other. Count Terminal Outcomes, Not Processed Records Processed count is a dangerous comfort metric. A record can be processed and still be wrong. It can be skipped correctly. It can be suppressed as a duplicate. It can be retried. It can land in a dead letter queue. It can be quarantined because replaying it would violate a business rule. The recovery run should account for outcomes explicitly: OutcomeTerminal?MeaningAPPLIEDYesThe expected side effect was performed.UPDATEDYesExisting target state was corrected or brought current.SKIPPEDYesThe record was intentionally ignored under a documented rule.DUPLICATE_SUPPRESSEDYesIdempotency prevented a repeat side effect.FAILED_RETRYABLENoProcessing failed and remains inside retry policy.FAILED_FINALYesProcessing failed and needs owner action.QUARANTINEDYesThe record is retained but should not be replayed without domain review. The key equation is simple: expected = applied + updated + skipped + duplicate_suppressed + failed_final + quarantined Retryable failures should not count as terminal until the retry policy is exhausted or the run is stopped. If the math does not close, the recovery has not been proven. Keep Metrics Bounded Recovery runs create tempting labels: run IDs, account IDs, order IDs, offset ranges, incident IDs. Put those blindly into metrics and the observability system becomes the next incident. Use metrics for bounded dimensions and put high-cardinality detail in decision logs, traces, audit tables, or exemplars. MetricBounded labelsPurposekafka_recovery_records_expectedtopic, event_type, run_typeDefines the denominator.kafka_recovery_records_unaccountedtopic, event_type, run_typeShows records not yet terminal.kafka_recovery_records_terminal_totaloutcome, event_type, run_typeShows how records completed.kafka_consumer_duplicate_suppressed_totalconsumer_group, event_type, run_typeProves replay did not duplicate side effects.kafka_recovery_reconciliation_mismatchescheck_name, run_typeBlocks false recovery success. That split matters. Metrics tell you that recovery is unhealthy at a class level. The decision record tells you which run, offsets, business keys, and owners are involved. Emit Decisions You Can Inspect A useful recovery log does not just say that a handler ran. It says why a record took a path. JSON { "recoveryRunId": "rec-2026-09-05-01", "eventId": "evt-1001", "businessKeyHash": "bkh_7f3a", "topic": "billing-events", "partition": 8, "offset": 492881, "eventType": "InvoicePosted", "outcome": "DUPLICATE_SUPPRESSED", "reason": "idempotency_key_exists", "schemaVersion": "v4", "consumerGroup": "usage-aggregator", "runType": "incident_repair", "decidedAt": "2026-09-05T18:42:31Z" } Notice what is missing: raw payloads, customer identifiers, tokens, and private business data. Recovery logs should explain decisions without becoming a second data-leak surface. The Java side can stay small. The following Java 17/Spring/Micrometer sketch is illustrative, not a complete production service. It assumes repository, meterRegistry, objectMapper, and log are class fields: Java record RecoveryDecision( String recoveryRunId, String eventId, String businessKeyHash, String topic, int partition, long offset, String eventType, RecoveryOutcome outcome, String reason, String consumerGroup, String runType, Instant decidedAt ) {} void emit(RecoveryDecision decision) { repository.insert(decision); Tags tags = Tags.of( "event_type", decision.eventType(), "run_type", decision.runType(), "outcome", decision.outcome().name() ); meterRegistry .counter("kafka_recovery_records_decision_total", tags) .increment(); if (decision.outcome() != RecoveryOutcome.FAILED_RETRYABLE) { meterRegistry .counter("kafka_recovery_records_terminal_total", tags) .increment(); } log.info("kafka_recovery_decision={}", toJson(decision)); } String toJson(RecoveryDecision decision) { try { return objectMapper.writeValueAsString(decision); } catch (JsonProcessingException error) { return "{\"serialization_error\":\"recovery_decision\"}"; } } That emitter is not the recovery system. It is the instrumentation boundary. It makes the record decision visible in the durable decision log, the metric stream, and the incident timeline. Reconciliation Is the Verdict Layer The hardest part of recovery observability is deciding what “correct” means. For some domains, a count is enough. For most serious domains, it is not. A count can match while the wrong keys, amounts, versions, or timestamps remain wrong. The source of truth must be named. The tolerance window must be named. Late arrivals, clock skew, eventually consistent sinks, and manual exceptions must be handled explicitly. Otherwise reconciliation becomes another dashboard panel that looks precise while hiding the policy decision. For example, if the recovery cohort is supposed to repair account/day billing state, compare the source ledger to the target projection for the same cohort. This is PostgreSQL-style SQL; use an equivalent pattern for your database: PLSQL WITH expected AS ( SELECT account_hash, business_date, SUM(amount_cents) AS expected_amount_cents FROM billing_ledger WHERE recovery_run_id = :recovery_run_id GROUP BY account_hash, business_date ), actual AS ( SELECT account_hash, business_date, amount_cents AS actual_amount_cents FROM billing_daily_projection WHERE recovery_run_id = :recovery_run_id ) SELECT COALESCE(e.account_hash, a.account_hash) AS account_hash, COALESCE(e.business_date, a.business_date) AS business_date, COALESCE(e.expected_amount_cents, 0) AS expected_amount_cents, COALESCE(a.actual_amount_cents, 0) AS actual_amount_cents FROM expected e FULL OUTER JOIN actual a ON e.account_hash = a.account_hash AND e.business_date = a.business_date WHERE COALESCE(e.expected_amount_cents, 0) <> COALESCE(a.actual_amount_cents, 0); Also check whether every selected event produced a decision: PLSQL SELECT c.event_id FROM kafka_recovery_cohort_event c LEFT JOIN kafka_recovery_decision_log d ON c.recovery_run_id = d.recovery_run_id AND c.event_id = d.event_id WHERE c.recovery_run_id = :recovery_run_id AND d.event_id IS NULL; Those queries are intentionally plain. Recovery confidence should not depend on a heroic debugging session. It should depend on executable checks that can run before anyone is asked to close the loop. Alert on Evidence Failure, Not Just Kafka Symptoms Recovery alerts should catch bad recovery, not only broken Kafka. Page when terminal outcomes do not close after the deadline, reconciliation fails after the check has completed, or an active recovery run stops making progress. Do not page simply because an in-progress recovery still has work remaining. YAML groups: - name: kafka-recovery rules: - alert: KafkaRecoveryUnaccountedRecords expr: | kafka_recovery_records_unaccounted > 0 and on(topic, event_type, run_type) kafka_recovery_deadline_reached == 1 for: 15m labels: severity: page annotations: summary: "Kafka recovery has overdue unaccounted records" - alert: KafkaRecoveryRunTooOld expr: | time() - kafka_recovery_last_progress_timestamp_seconds > 1800 and on(topic, event_type, run_type) kafka_recovery_run_active == 1 for: 5m labels: severity: ticket annotations: summary: "Kafka recovery run has stopped making progress" The alert is not the verdict. It tells the operator where to look. The verdict should still come from the recovery evidence record. Test the Evidence Path Recovery safety should not only be described. It should be testable. At minimum, test the evidence path with a small matrix: TestInjectionExpected proofDuplicate replayProcess the same event_id twice.One APPLIED, one DUPLICATE_SUPPRESSED, no second side effect.Missing decisionSelect an event into the cohort but skip emission.no_missing_decision_events fails.Projection driftChange target state after replay.Reconciliation returns mismatched keys.Poison recordSend a record that violates a domain rule.Outcome is QUARANTINED with owner and reason.Stalled runStop progress after partial terminal outcomes.Unaccounted-record or stale-progress alert fires. Run these tests in two places. First, keep a small contract-test suite around the recovery library or consumer adapter so decision emission cannot regress. Second, run a replay drill in a non-production environment with a deliberately small cohort. The drill should prove that the metrics, decision log, reconciliation query, alerts, and final verdict all agree before the same pattern is trusted during an incident. Recovery Definition of Done Before closing a Kafka recovery run, require evidence for each item: CheckEvidenceCohort declaredTopic, partitions, offsets, event types, expected count, owner, and stop conditions are recorded.Terminal math closesExpected records equal terminal outcomes, excluding retryable in-flight failures.Idempotency is visibleDuplicate suppressions and side-effect decisions are emitted as bounded metrics and durable decision records.Failure ownership existsFinal failures and quarantined records have reasons, owners, and next actions.Reconciliation passedThe named source of truth matches the target state for the approved scope, or exceptions are explicitly owned.Verdict recordedThe run ends as RECOVERED, RECOVERED_WITH_EXCEPTIONS, BLOCKED, QUARANTINED, or FAILED. This full pattern is not free. Structured decision logs add write volume. Reconciliation consumes compute. Audit tables need retention and access control. Quarantine creates human workflow. There are also cases where the full pattern is too much. A stateless consumer that only updates an internal cache may not need decision tables, reconciliation jobs, and formal verdicts. Start with the recovery definition of done, then choose the minimum evidence that proves the business risk you actually have. Kafka gives teams a powerful recovery primitive because the log can be replayed. That does not make recovery safe by itself. Lag tells you the consumer moved. Recovery evidence tells you whether the system became correct again.

Kafka is a distributed event streaming platform that enables real-time data ingestion, storage, and processing of continuous event streams. To do that, Kafka has a complex internal structure. In this part, I will try to cover an overview of the full internal structure. Overview Kafka uses a publish-subscribe model where producers produce events and consumers consume messages. Kafka producers publish events to the broker, and consumers consume the events from the broker. In the meantime, the broker has a different layer of architecture that allows Kafka to be an event streaming platform. The producer, broker, and consumer-related metadata are managed by the Kafka controller, which is elected by ZooKeeper or KRaft. Key Terms Topic: A topic is a logical name or category where Kafka stores a stream of related events/messages. Think of a topic as a database table for events or a channel where producers publish messages and consumers subscribe to read them. Partition: A partition is a physical subdivision of a Kafka topic that stores messages in an ordered, immutable sequence. It is the basic unit of parallelism and scalability in Kafka. Segment: A segment is a physical file inside a Kafka partition where messages are stored on disk. A partition is not stored as one huge file. Kafka divides it into multiple smaller segment files. Offset: An offset is a unique sequential number assigned to each message within a Kafka partition. It represents the position of a message in that partition. Metadata: Metadata is information about the Kafka cluster, not the actual message data. It describes the structure and state of the Kafka cluster. Controller: The Kafka Controller is a special broker responsible for managing cluster-level operations. In a Kafka cluster, one broker is elected as the controller. ZooKeeper (OLD): ZooKeeper is an external distributed coordination system used by older Kafka versions to manage cluster metadata and controller election. Kraft (Modern): KRaft (Kafka Raft metadata mode) is Kafka's new architecture that removes ZooKeeper and manages metadata internally using the Raft consensus algorithm. Leader: A leader is the broker that owns the primary copy of a partition and handles all read and write operations for that partition. Only one broker can be the leader for a partition at a time. Replica: A replica is a copy of a partition stored on another broker to provide fault tolerance. Internal Details With the Kafka Library At first, download Kafka from this link. After extracting, we will get the following directory structure: Here, each folder has its own functionality. The bin folder contains all the executable scripts. These scripts can be used from the command line to manage Kafka components and perform various tasks, including those listed below: Start serversStop serversTopic managementCluster managementAct as a publisher/consumer Another important folder is the config folder, where we can configure the broker and controller nodes. Initially, this folder contains the default configuration files for the Kafka components. Here we find property files named controller.properties, server.properties, broker.properties, consumer.properties, producer.properties, and some other configuration files. In a Kafka installation, these .properties files are configuration files used by different Kafka components and client applications. However, not all of them are used by the Kafka server itself. We can think of the configuration files in three major categories: Servers - server.properties - broker.properties - controller.propertiesClients - producer.properties - consumer.propertiesTools - other properties file 1. server.properties This is the main configuration file for a Kafka broker. It controls how the Kafka broker runs, including: Broker ID/node identityListener and network configurationLog directoriesNumber of network and I/O threadsDefault number of partitionsReplication settingsRetention policiesMessage size limitsController configuration in KRaft modeCluster-related settings For example: Properties files node.id=1 process.roles=broker,controller listeners=PLAINTEXT://:9092,CONTROLLER://:9093 controller.quorum.voters=1@localhost:9093 log.dirs=/tmp/kafka-logs num.partitions=1 log.retention.hours=168 In modern Kafka using KRaft, server.properties can configure both the broker and controller when a combined node is used. 2. broker.properties This is generally used as a separate broker configuration file when you want to maintain configuration for a broker independently. For example, you might create: Plain Text config/ ├── broker.properties ├── controller.properties and start Kafka with a particular configuration: Properties files kafka-server-start.sh config/broker.properties The exact files supplied with Kafka can vary depending on the Kafka version/distribution. broker.properties is not a universal mandatory filename in Kafka; Kafka mainly cares about the properties file you pass to the startup command. 3. controller.properties This is used for configuring a KRaft controller. Kafka's modern architecture doesn't require ZooKeeper. Instead, Kafka uses the KRaft consensus mechanism. A controller is responsible for managing cluster metadata, such as: Which brokers are part of the clusterTopic metadataPartition informationLeader assignmentsCluster configurationController elections A dedicated controller configuration could contain things such as: Properties files process.roles=controller node.id=1 listeners=CONTROLLER://:9093 controller.quorum.voters=1@localhost:9093 controller.listener.names=CONTROLLER A production Kafka cluster may use dedicated controller nodes rather than running broker and controller roles together. 4. producer.properties This is not a broker configuration. It contains configuration for a Kafka Producer client — the application that sends messages to Kafka. For example: Properties files bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer Other producer settings can control: acksretriesbatch.sizelinger.mscompression.typebuffer.memoryenable.idempotencehghg For example: Properties files acks=all retries=10 compression.type=snappy A producer application reads these settings when connecting to Kafka. 5. consumer.properties This is also a client configuration, not a broker configuration. It controls how a Kafka Consumer reads messages. Example: Properties files bootstrap.servers=localhost:9092 group.id=my-consumer-group key.deserializer=org.apache.kafka.common.serialization.StringDeserializer value.deserializer=org.apache.kafka.common.serialization.StringDeserializer auto.offset.reset=earliest Important consumer settings include: bootstrap.serversKafka broker addressesgroup.idConsumer group identitykey.deserializerConverts message key bytesvalue.deserializerConverts message value bytesauto.offset.resetWhat to do when no valid offset existsenable.auto.commitControls automatic offset commitsmax.poll.recordsMaximum records returned per pollfetch.min.bytesMinimum data fetched from broker 6. Other Configuration Files Depending on your Kafka version, you may also find files such as: Plain Text config/ ├── connect-distributed.properties ├── connect-standalone.properties ├── connect-log4j.properties ├── tools-log4j.properties └── ... Some are for Kafka's additional components. For example: Kafka Connect Plain Text connect-distributed.properties connect-standalone.properties These configure Kafka Connect workers. Logging Plain Text connect-log4j.properties tools-log4j.properties These configure logging behavior for particular Kafka tools/components. So, server.properties/broker.properties/controller.properties configure Kafka infrastructure, while producer.properties and consumer.properties configure applications that communicate with Kafka. Step by Step: Test Producer and Consumer From CLI We will discuss the Kafka cluster later, but for now we will check how to run the Kafka controller, broker, and an example producer and consumer from the CLI. 1. Initialize Kafka Storage For a new Kafka installation running in KRaft mode, first generate a cluster ID. If we move to the Kafka bin directory, we can find a file called kafka-storage.sh Shell ./kafka-storage.sh random-uuid Copy the generated UUID. Then format the Kafka storage: Shell ./kafka-storage.sh format \ -t <CLUSTER_ID> \ -c ../config/server.properties For example: Shell ./kafka-storage.sh format \ -t abc123xyz \ -c ../config/server.properties You normally perform this initialization only once for a fresh Kafka storage directory. Don't repeatedly format an existing cluster unless you intentionally want to initialize/reset it. 2. Start the Kafka Server Because we are now inside the bin, specify the configuration file relative to the parent directory: Shell ./kafka-server-start.sh ../config/server.properties Or we can run it in the background: Shell ./kafka-server-start.sh -daemon ../config/server.properties The important relationship is: Plain Text bin/ └── kafka-server-start.sh config/ └── server.properties So the command is Shell ./kafka-server-start.sh ../config/server.properties 3. Verify That Kafka Is Running We can also check the Kafka port: Plain Text lsof -i :9092 4. Create a Topic Still inside the bin directory: Shell ./kafka-topics.sh \ --create \ --topic test-topic \ --bootstrap-server localhost:9092 \ --partitions 1 \ --replication-factor 1 List topics: Shell ./kafka-topics.sh \ --list \ --bootstrap-server localhost:9092 5. Start a Consumer Open another terminal and go to Kafka's bin directory: Shell ./kafka-console-consumer.sh \ --topic test-topic \ --bootstrap-server localhost:9092 The consumer will wait for messages. 6. Start a Producer Open another terminal: Shell ./kafka-console-producer.sh \ --topic test-topic \ --bootstrap-server localhost:9092 Now type: Plain Text Hello Kafka The message should appear in the consumer terminal. This is the introduction to this series. We will discuss more details in the upcoming articles.

A few years back, I sat through a production incident review where the root cause turned out to be a single stored procedure nobody on the team had touched in four years. The original author had left the company. Nobody knew why a particular join filtered out cancelled orders, only that removing it broke three downstream reports. That afternoon summed up, for me, everything wrong with how a lot of organizations still build their data pipelines. Legacy ETL was never really about "extract, transform, load" as much as it was about hiding logic somewhere convenient at the time and hoping the convenience lasted. It usually didn't. Why ETL Started Creaking Traditional ETL tools like Informatica or SSIS made sense when compute was expensive and storage was the constraint you designed around. You transformed data before it landed in the warehouse because you couldn't afford to store raw, messy versions of everything. That constraint shaped an entire generation of architecture decisions, and honestly, it wasn't a bad approach for its time. The trouble is that constraint mostly disappeared. Cloud storage is cheap now. Compute is elastic. Yet a lot of teams are still running the same transform-before-load pattern, just on newer infrastructure, which is a bit like buying a Tesla and still driving it like it takes ten minutes to warm up. Here's roughly what that legacy shape looks like in practice: Notice where the pain actually concentrates. It's not really the source systems or the warehouse; it's that middle stretch where an ETL server holds all the transformation logic, usually in a proprietary format, usually without meaningful version control, and usually without anyone testing whether a schema change upstream will quietly break a job three hops downstream. What ELT Changes, and What It Doesn't Moving to ELT isn't a magic fix, and I'd push back a little on anyone who frames it that way. What it does is separate two things that legacy ETL bundled together: moving data and transforming data. You load raw data first, cheaply, and you transform it later using the compute and tooling that's actually good at transformation, which for most teams now means dbt on top of a lakehouse like Databricks. This matters more than it sounds like on paper. When transformation logic lives in dbt models rather than a GUI-based ETL tool, it becomes something you can review in a pull request, test automatically, and trace through version history. A junior engineer six months from now can open the repo and actually see why cancelled orders get filtered, because it's sitting there in a documented model, not buried in a stored procedure that requires archaeology to understand. A Governed Architecture, Not Just a Faster One The part people skip, though, is governance. Plenty of "modernization" projects swap Informatica for dbt and call it done, then wonder why data quality issues and access sprawl show up six months later, just in a different shape. Speed without governance is still going to bite you; it just takes longer. Here's the architecture I'd actually recommend, built around the medallion pattern with governance woven through it rather than added on afterward: A few things worth calling out here, because the diagram alone won't tell you everything: Bronze is genuinely raw. No transformation, no filtering, just an append-only landing zone. This is your safety net. If a transformation bug corrupts your Silver layer, you can always rebuild from Bronze because nothing was thrown away. Silver is where dbt earns its keep. Cleaning, deduplication, conforming dimensions, basic business rules. This layer should be boring and predictable, which is exactly what you want from something everything else depends on. Gold is opinionated on purpose. These are business-facing marts, built for specific consumption patterns, whether that's a finance dashboard or a feature store feeding a fraud model. Unity Catalog isn't decoration. Row and column level access control, PII tagging, and lineage tracking across the whole Bronze to Gold journey happen here. When an auditor asks who touched customer PII and when, you actually have an answer instead of a guess. dbt tests gate the pipeline, not the postmortem. Schema tests, not-null checks, referential integrity, and increasingly, formal data contracts between teams run before code merges rather than after something breaks in production. I'll admit the CI and testing layer is the part teams most often underinvest in, mostly because it's less visible than a shiny new lakehouse. But it's the difference between catching a schema drift issue in a pull request versus catching it because a VP asks why the revenue dashboard looks wrong on a Monday morning. Where This Still Falls Short I don't want to oversell this. Governance frameworks like Unity Catalog add real operational overhead, and smaller teams sometimes find the setup cost disproportionate to their scale, at least initially. dbt also assumes a certain SQL-first maturity in your team; if most of your transformations genuinely need Python or Spark-native processing, you're looking at a hybrid setup rather than a pure dbt one. And migration itself is rarely a clean cutover. Most organizations run legacy ETL and modern ELT in parallel for a stretch, which is its own kind of operational headache. Still, for teams dealing with genuine scale, audit requirements, or just the accumulated technical debt of a decade of ad hoc SQL scripts, this pattern tends to hold up. Not because it's trendy, but because it separates concerns that never should have been tangled together in the first place, and it gives you a paper trail for decisions that used to live only in someone's memory. That's really the underlying shift here. Legacy ETL asked you to trust the person who wrote the job. Governed ELT asks the system to show its work.

ChatGPT responses in Europe may soon carry a signal users cannot see. OpenAI is rolling out invisible text watermarks for eligible ChatGPT and Codex output in the European Union across all plans. The provenance system is designed to help identify AI-generated text as transparency requirements under the EU AI Act take effect. Users will not need to enable anything themselves, but the watermark will not apply equally to every response — and editing, translation or paraphrasing can significantly weaken detection. How OpenAI's invisible watermark works OpenAI calls its technology textGrain. The system slightly changes how its models choose among possible next words or word pieces. Across a long enough passage, those choices create a statistical pattern that compatible detection software can recognize. No hidden characters, extra spaces, or unusual punctuation need to be inserted. Copying a response into another document therefore does not automatically strip the watermark because the wording itself carries the detectable pattern. EU provenance rules form part of the EU AI Act, which sets transparency obligations for providers of general-purpose AI systems. Code gets different treatment. Fewer acceptable choices may exist for the next piece of code, limiting how much textGrain can alter model output without affecting correctness. EU guidance also excludes code snippets and responses below 200 tokens, roughly 150 English words, from text-watermarking expectations. Codex is included in the rollout, but its inclusion does not mean every generated code snippet will contain a detectable watermark. Detection varies with length and editing Longer passages give the detector more material to examine. Testing produced the following results across several evaluations. Test Reported detection rate 200-token passage at a 1% false-positive rate About 80% 400-token passage at a 1% false-positive rate About 95% 400-token passage after 10% of words were replaced with synonyms About 66% 400-token passage after 25% of words were replaced 17% Editing can weaken the pattern considerably, and translation or substantial paraphrasing can also interfere with detection. Highly constrained material such as mathematics produced lower rates because the model has fewer acceptable wording choices. A match does not identify a user, reveal the original prompt or prove authorship. A negative result is also inconclusive because editing, paraphrasing or translation can weaken or remove the detectable signal. Similar debates around AI-generated content provenance have grown as technology companies experiment with machine-readable markers. Detector access will initially be limited to approved researchers and qualified expert organizations. Ordinary ChatGPT users will not receive a public detector as part of the launch. ChatGPT users in the EU should keep a disclosure trail Anyone using ChatGPT for work governed by an AI disclosure policy should continue following that policy after watermarking starts. An invisible marker cannot document why AI was used, how a draft changed afterward, or whether its use complied with an employer, publisher, client, or school rule. Users and organizations can take a few additional steps. Keep your own record of AI use. Version history can preserve changes made after a ChatGPT draft, and a short note documenting permitted AI assistance can provide context if a finished document is questioned. Avoid treating a detector match as proof of authorship. A positive result can justify reviewing how AI was used, but it should not settle an authorship dispute on its own. Organizations using automated checks should define how results fit into their AI governance policies before relying on them in employment, academic, or publishing decisions. EU users may never have direct access to OpenAI’s detector, which makes their own records more important when AI use needs to be explained later. OpenAI's watermark may give researchers and organizations another way to identify AI-generated text, but for users, it is still only a provenance signal. When AI use needs to be explained or disclosed, ordinary documentation remains more reliable than an invisible marker. Editor’s note: This article originally appeared on our sister publication, TechRepublic.

Apache IoTDB is well-suited for environments where time-series data from connected devices and industrial systems is generated continuously and requires efficient querying. Typical use cases include monitoring temperature, pressure, vibration, energy usage, machine status, and device telemetry. This approach also applies to manufacturing, smart infrastructure, fleet monitoring, utilities, and edge computing. IoTDB stands out for its focus on large-scale time-series workloads from devices and industrial systems. It is purpose-built for high-frequency data ingestion, historical analysis, and time-based queries. This makes it ideal for applications that require insight into both the current state and historical trends of devices or processes. Why Apache IoTDB Matters Apache IoTDB excels at managing large volumes of device-generated data over time. It addresses issues beyond storage, such as continuous data ingestion, efficient organization by device and timestamp, and fast queries for both recent and historical values. This makes IoTDB well-suited for systems with temporal, device-oriented data models. Common use cases include industrial IoT, smart factories, energy monitoring, connected vehicles, building automation, predictive maintenance, and edge computing. For example, manufacturers can track machinery data, utilities can analyze energy consumption across many meters, and fleet operators can monitor location and engine telemetry. In each scenario, IoTDB enables applications to determine current status, review device behavior over time, and detect measurements outside expected ranges. Hands-On: Java With Apache IoTDB We will build a simple Java application using Apache IoTDB. For simplicity, IoTDB will run locally in Docker. Shell docker run -d \ --name iotdb-instance \ -p 6667:6667 \ -e dn_rpc_address=0.0.0.0 \ apache/iotdb:2.0.11-standalone Next, create the database using IoTDB’s Table SQL dialect: Shell docker exec -it iotdb-instance \ /iotdb/sbin/start-cli.sh \ -h 127.0.0.1 \ -p 6667 \ -u root \ -pw root \ -sql_dialect table \ -e "CREATE DATABASE IF NOT EXISTS jnosql" For this local example, we use the default root/root credentials and disable client redirection. These settings are suitable for single-node development only. In production, use secure credentials and configure redirection based on your IoTDB cluster topology. Configuring the Application Next, configure the connection in microprofile-config.properties: Properties files jnosql.timeseries.database=jnosql jnosql.iotdb.host=localhost jnosql.iotdb.port=6667 jnosql.iotdb.username=root jnosql.iotdb.password=root jnosql.iotdb.enable.redirection=false Eclipse JNoSQL is built on Jakarta APIs such as CDI, JSON-B, and Eclipse MicroProfile Config. These APIs are supported by popular runtimes including Helidon, Quarkus, Open Liberty, and other Jakarta EE-compatible environments. Then add the Apache IoTDB driver: XML <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-iotdb</artifactId> <version>${jnosql.version}</version> </dependency> Modeling Sensor Data A sensor-oriented model aligns well with IoTDB: Java @Entity public class SensorReading { @Id private Instant id; @Column private String sensor; @Column private double temperature; @Column private double humidity; // constructors, getters, and setters } The Instant field records the timestamp of the reading. The other fields capture the measurements at that time. The entity can also be exposed through Jakarta Data: Java @Repository public interface SensorReadingRepository extends BasicRepository<SensorReading, Instant> { List<SensorReading> findBySensorOrderByIdDesc( String sensor, Limit limit); } Using TimeSeriesTemplate With the infrastructure in place, insert several readings and retrieve both the latest value and recent history: Java "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { TimeSeriesTemplate template = container.select(TimeSeriesTemplate.class).get(); template.insert(firstReading); template.insert(secondReading); template.insert(latestReading); var currentReading = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .limit(1) .singleResult(); System.out.println( "Current sensor reading: " + currentReading ); var history = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .skip(1) .limit(10) .result(); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } The first query answers a common IoT question: what is the latest reading from this sensor? The second retrieves its recent history, excluding the latest observation. Using Jakarta Data The same use case can be implemented using the repository: Java public class App2 { public static void main(String[] args) { var firstReading = new SensorReading( Instant.parse("2026-09-20T08:00:00Z"), "sensor-01", 21.4, 45.0 ); var secondReading = new SensorReading( Instant.parse("2026-09-20T09:00:00Z"), "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { SensorReadingRepository repository = container.select(SensorReadingRepository.class).get(); repository.save(firstReading); repository.save(secondReading); repository.save(latestReading); var currentReading = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.of(1) ) .stream() .findFirst(); System.out.println( "Current sensor reading: " + currentReading ); var history = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.range(2, 10) ); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } Both approaches model the domain in terms of latest state and historical observations. This is where a time-series database like Apache IoTDB is more effective than using a general-purpose database with timestamps as regular fields. Why Apache IoTDB Is More Than Sensor Storage While Apache IoTDB is often associated with IoT, its capabilities go well beyond basic sensor data storage. It excels when organizations must manage large volumes of device-centric data over time while retaining relationships among measurements, devices, and timestamps. As a result, IoTDB is valuable for industrial systems, utilities, transportation, smart infrastructure, manufacturing, and edge computing. For example, in predictive maintenance, a machine may continuously report vibration, temperature, pressure, energy consumption, and operating condition. The true value is not only found in the latest readings, but in studying how these measurements change together over time. Maintenance systems may compare recent data with historical trends, spot anomalies before failures, or correlate changes across multiple devices. This requires more than simple sensor data storage. This approach also benefits energy systems, where smart meters, solar panels, batteries, and grid equipment generate continuous measurement streams that require time-based analysis. In transportation, vehicles produce ongoing data such as speed, location, fuel consumption, battery status, and engine telemetry. Similarly, smart buildings rely on time-series data from HVAC systems, occupancy sensors, and energy meters. IoTDB is especially effective when system architecture is organized around devices and their measurements. It can serve as the historical data layer for operational systems, while other technologies manage transactional or business information. For example, a relational database might store details about a machine, its owner, contracts, or maintenance schedules, while IoTDB manages the extensive measurement data the machine generates. This separation lets enterprise applications treat telemetry as a primary workload, rather than forcing high-frequency device data into models meant for business entities and transactions. Conclusion Apache IoTDB is a strong fit for applications that need to ingest and analyze device-generated data over time, especially in IoT, industrial, energy, and edge scenarios. With Eclipse JNoSQL 1.1.18, Java developers can access these capabilities through familiar Jakarta APIs, keeping the application model consistent while still leveraging IoTDB’s time-series specialization.

By Otavio Santana CORE