Big data comprises datasets that are massive, varied, complex, and can't be handled traditionally. Big data can include both structured and unstructured data, and it is often stored in data lakes or data warehouses. As organizations grow, big data becomes increasingly more crucial for gathering business insights and analytics. The Big Data Zone contains the resources you need for understanding data storage, data modeling, ELT, ETL, and more.
How (and Why) to Build Dynamic Data Pipelines
When to Use AMQP, JMS, Kafka, or MQTT: Trade-Offs, Not a Winner
Everyone is rushing to build AI agents right now, but GenAI integration really just puts a stronger spotlight on solid system design and backend reliability. An AI agent relies entirely on the APIs and data pipelines underneath it. If your backend drops events or double-processes payments during a network blip, even the best agent will just execute a broken workflow faster. To build resilient systems, you need to see how they break. Reading Kafka documentation on fault tolerance helps, but intentionally introducing network chaos shows how the underlying mechanics actually behave. In this post, we will set up a local testing environment using Spring Boot, Kafka, and Toxiproxy. We'll configure producer properties for topic-level idempotence and walk through common network simulation pitfalls that lead to duplicate data. 1. Setting Up the Chaos Test Architecture To test network issues accurately, stopping the Kafka broker isn't enough. We need to simulate realistic network conditions like latency spikes, packet drops, and bandwidth throttling. Toxiproxy acts as a proxy between our producer and broker to simulate these conditions. The Network Architecture Creating a Local/Dev Kafka Cluster We start by spinning up the Kafka cluster and Kafka UI using Docker. YAML # docker-compose-kafka.yml version: "3.8" #avoid this services: kafka: image: apache/kafka:latest container_name: kafka ports: - "9092:9092" environment: CLUSTER_ID: "4L62xdwCQKWvZs96A39vGA" KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: "broker,controller" KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093" KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL" # Configure distinct internal/external routing ports KAFKA_LISTENERS: "INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" # We will bookmark this line, dear reader! KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:19092" KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_LOG_DIRS: "/var/lib/kafka/data" volumes: - kafka_data:/var/lib/kafka/data kafka-ui: image: provectuslabs/kafka-ui:latest container_name: kafka-ui ports: - "9080:8080" depends_on: - kafka environment: DYNAMIC_CONFIG_ENABLED: "true" KAFKA_CLUSTERS_0_NAME: "local-cluster" KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:29092" volumes: kafka_data: To spin it up, we use Docker Compose: docker compose -f "docker-compose -kafka.yml" up Both your local Kafka cluster and the kafka-ui should spin up. Note that KAFKA_ADVERTISED_LISTENERS includes the external port localhost:19092; we'll see why. A Local Proxy to Sit Between Kafka Broker and Producer We use Docker for this, too! YAML # docker-compose-toxiproxy.yml version: "3.8" # Don’t use this version - it’s deprecated services: toxiproxy: image: shopify/toxiproxy:latest container_name: toxiproxy ports: - "8474:8474" # Toxiproxy Management API - "8000:8000" # Example port exposed for a proxied application service - "19092:19092" networks: - toxiproxy-net toxiproxy-ui: image: buckle/toxiproxy-frontend:latest container_name: toxiproxy-ui ports: - "8080:8080" # We don’t use UI in this article environment: - TOXIPROXY_URL=http://toxiproxy:8474 depends_on: - toxiproxy networks: - toxiproxy-net networks: toxiproxy-net: driver: bridge docker compose -f docker-compose-toxiproxy.yml up Once running, create a proxy tunnel via the Toxiproxy CLI to listen on port 19092 and forward traffic to Kafka on port 9092: ./toxiproxy-cli create -l 0.0.0.0:19092 -u host.docker.internal:9092 kafka-proxy Note: We use the above CLI inside the toxiproxy container. So, you must use docker exec. In some versions, the CLI tool might not be included in the $PATH, and you might need to switch the current directory: Shell /go/bin # ls toxiproxy toxiproxy-cli If you're using this toxiproxy tool for the first time, you can refer to the official Toxiproxy documentation or this starter guide we created. 2. The Kafka Client Producer Code We create a basic application using Spring Boot 4.1 (though we do not have any version restriction until now) and use the following dependency: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-kafka</artifactId> </dependency> We'll keep the Spring Boot service simple. We define a basic payload configuration and direct messages to topic T3 via KafkaConfig.java. Config: Java // KafkaConfig.java package com.soham.selfteaching.kafka.kafkadeepdive.config; import lombok.Data; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Configuration; @Configuration @Data public class KafkaConfig { @Value("${app.topic.name:T3}") private String topic; } Our application.yml is minimal, accepting defaults provided by the Kafka library. Properties files # application.yml server: port: 8082 spring: application: name: kafka-deep-dive kafka: bootstrap-servers: "localhost:19092" producer: acks: "1" # Required to bypass idempotence shield key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: request.timeout.ms: 3000 delivery.timeout.ms: 120000 enable.idempotence: false # We disable it first before we explore producer level idempotence The domain event is defined as a Java record. Java // EventDto.java package com.soham.selfteaching.kafka.kafkadeepdive.dto; public record EventDto(String id,String payload) { } Producer Code The core logic sits in ProducerService.java. We use CompletableFuture to handle delivery timeouts asynchronously without blocking thread execution. If the broker is unreachable initially, max.block.ms handles the immediate block on the main thread while background retries handle transient issues. Java // ProducerService.java package com.soham.selfteaching.kafka.kafkadeepdive.service; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.springframework.kafka.core.KafkaTemplate; import org.springframework.kafka.support.SendResult; import org.springframework.stereotype.Service; import java.util.concurrent.CompletableFuture; @Service @Slf4j @RequiredArgsConstructor public class ProducerService { private final KafkaTemplate<String, Object> kafkaTemplate; public void sendMessage(String topic, String key, Object payload) { log.info("Attempting to send message to topic: {} with key: {}", topic, key); CompletableFuture<SendResult<String, Object>> future = kafkaTemplate.send(topic, key, payload); future.whenComplete((result, exception) -> { if (exception == null) { log.info("Message successfully delivered. Topic: {}, Partition: {}, Offset: {}", result.getRecordMetadata().topic(), result.getRecordMetadata().partition(), result.getRecordMetadata().offset()); } else { log.error("Fatal delivery failure for key {}. Exhausted delivery.timeout.ms.", key, exception); } }); } } 3. Configuring Topic-Level Idempotence During a network timeout, the producer cannot verify whether the message failed to reach the broker or if the acknowledgment (ACK) was lost on the return path. By default, the producer retries the request, which can produce duplicate records without proper safeguards. Exactly-Once Semantics (EOS) and topic-level idempotence address this by assigning a producer ID (PID) and sequence number to each message batch. If the broker receives a retried batch with a sequence number it has already processed, it ignores the duplicate write. Disabling Idempotence to Force Duplicates To demonstrate retry behavior, we can explicitly disable these safety settings in application.yml. Since Kafka 3.0, enable.idempotence defaults to true. To allow duplicates during testing, set acks: "1" and explicitly set enable.idempotence: false. Setting request.timeout.ms to 3000 (3 seconds) creates a tight window to trigger network retries during latency testing. 4. Running the Kafka Producer and Verify We run the Spring Boot application: Java @SpringBootApplication public class KafkaDeepDiveApplication { public static void main(String[] args) { SpringApplication.run(KafkaDeepDiveApplication.class, args); } } And hit a REST API (Hold on! It's a simple HTTP API which may NOT quite follow REST paradigms). curl -X GET http://localhost:8082/api/send Java @RestController @RequestMapping("/api") @Slf4j @RequiredArgsConstructor public class ProducerController { private final ProducerService service; private final KafkaConfig kafkaConfig; @GetMapping("/send") public String send() { long ts = System.currentTimeMillis(); Object payload = new EventDto("p1.t1." + UUID.randomUUID().toString(), ("Hello message " + i)); service.sendMessage(kafkaConfig.getTopic(), "k1", payload); return "Process took :%s ms".formatted(System.currentTimeMillis() - ts); } } In the case when idempotence is disabled, it's fair to expect that duplicate messages will arrive on a topic if a producer does not receive timely ACK from the broker leaders in alignment with the ack flag. But we do not yet see any duplicate messages at all. This is fairly explainable, since you're all set up locally, and there's usually no lost acknowledgment. Introduce Chaos to the Local Network Add 5000ms latency to the proxy. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy Send a message to the endpoint. After an API hit, wait for 2–3 seconds and remove the toxic Remove the Toxic While the Retry Is in Flight Instantly run this command to remove the toxic while the retry is in flight: ./toxiproxy-cli toxic remove -n false_positive kafka-proxy While this should have produced duplicates due to retries by the producer following a timed-out broker ACK, we find NO DUPLICATE MESSAGES even after repeating the process above many times. 5. Let Us Debug Our Disconnect Common Pitfalls During Network Injection Simply starting the proxy and adding latency doesn't always trigger duplicates as expected. Below are two common issues related to how Kafka clients handle metadata and connections. Issue 1: Advertised Listeners Bypass If Toxiproxy is running on port 19092 and a 5000ms latency is emulated, messages sent through the proxy might still complete in milliseconds. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy This occurs because of how Kafka advertises broker addresses during metadata discovery. If you look closely at the internal Kafka NetworkClient logs during startup, you will find out why: Plain Text brokers=[MetadataResponseBroker(nodeId=1, host='localhost', port=9092)] When the producer connects to port 19092 for initial cluster metadata, the broker returns its configured advertised listener (e.g., localhost:9092). The producer client caches this response and routes subsequent produce requests directly to port 9092, bypassing the proxy entirely. We must ensure that our KAFKA_ADVERTISED_LISTENERS explicitly broadcasts EXTERNAL://localhost:19092 so the producer continues routing traffic through Toxiproxy. If updated, clear existing Docker volumes (docker-compose down -v) to avoid using cached listener configurations. Issue 2: Initial Metadata Fetch Blocking If high latency (e.g., 15,000ms) is applied before sending any messages, the application may time out after 60 seconds with a TimeoutException: Topic T3 not present in metadata, rather than failing on delivery.timeout.ms. The delivery.timeout.ms timer only applies once a record enters the producer's accumulator buffer. If the client hasn't fetched topic metadata yet, the call blocks on max.block.ms (defaulting to 60 seconds) while attempting to retrieve metadata through the degraded network path. We apply the fix: Send an initial message without active toxics to populate the local metadata cache before injecting network latency. Finally With latency removed, the retried request reaches the broker successfully. Because acks=1 and enable.idempotence=false, the broker writes the payload again without verifying duplicate sequence numbers. Checking Kafka UI confirms two identical records are present in the topic partition. Simulating network edge cases directly demonstrates why configurations like advertised listeners, metadata timeouts, and idempotence defaults exist in modern Kafka clients. Now we get duplicate messages across a topic partition. Duplicate Messages We can now enable back idempotence flag to see that no duplicate messages are created repeating the above steps with enable.idempotence:true, ensuring that idempotence is preserved enable.idempotence= true only ensures idempotence across a partition only and not the entire topic. We shall explore it in a next article.
If you've worked with a relational database for any length of time, you know the drill: state lives in a row, and every UPDATE overwrites whatever was there before. It's simple, it's fast to reason about, and it throws away history by design. Update a bank balance from $1,000 to $900, and the fact that a withdrawal happened and why is gone unless some separate audit log happened to catch it. Event sourcing flips that model on its head. Instead of storing current state, you store the sequence of events that produced it, and you derive state on demand by replaying the log. History isn't a side effect you bolt on afterward; it is the write path. This isn't a niche idea anymore. Netflix runs event-sourced pipelines that process over a billion viewer events a day for personalization and health monitoring. Uber pushes millions of events per minute ride requests, location updates, pricing signals through Kafka-backed streams with exactly-once semantics for fraud detection. LinkedIn built Kafka in the first place to serve as an append-only log for activity data. Slack pipes billions of daily messages through event-driven pipelines that carry full metadata so services don't have to make synchronous calls to each other. What's harder to find is a clear-eyed look at what actually breaks in production and how teams fix it. That's what this article is about: the data model, the CQRS split, the consistency trade-offs, and the part that tends to get glossed over: how to keep read models correct when the same event might get delivered twice, three services need to stay in sync, and any one of them might be down when the message arrives. Events as First-Class State In a CRUD system, a withdrawal looks like this: SQL -- Traditional CRUD (destructive) UPDATE accounts SET balance = 900 WHERE id = 123; In an event-sourced system, nothing gets overwritten. You append a new fact: SQL -- Event sourcing (additive) INSERT INTO events (stream_id, event_type, payload, version) VALUES ('account-123', 'MoneyWithdrawn', '{"amount": 100}', 42); Current state is reconstructed by folding over every event in a stream, in version order. It's a pure function — same events in, same state out, and it maps directly onto a left-fold/reduce if you've ever written one: Python def reconstruct_balance(account_id): events = db.query( 'SELECT * FROM events WHERE stream_id = %s ORDER BY version', [f'account-{account_id}'] ) state = {'balance': 0} for event in events: if event.type == 'AccountOpened': state['balance'] = 0 elif event.type == 'MoneyDeposited': state['balance'] += event.payload['amount'] elif event.type == 'MoneyWithdrawn': state['balance'] -= event.payload['amount'] return state['balance'] The Event Store Schema The event store itself is a boring relational table with one architectural rule bolted on: nothing is ever updated or deleted. SQL CREATE TABLE events ( id BIGSERIAL PRIMARY KEY, stream_id UUID NOT NULL, event_type VARCHAR NOT NULL, payload JSONB NOT NULL, version INT NOT NULL, occurred_at TIMESTAMPTZ NOT NULL DEFAULT now(), UNIQUE (stream_id, version) ); That UNIQUE (stream_id, version) constraint is doing more work than it looks like. It's your optimistic concurrency control: if two processes both try to append version 5 to the same stream, the second one gets a unique-constraint violation instead of silently clobbering the first. It's compare-and-swap at the stream level, for free. And to be clear — this table can live in Postgres, MySQL, Oracle, whatever you're already running. Event sourcing is a write strategy, not a database product. Don't let anyone tell you that you need a specialized event store to do this. Snapshots Replaying thousands of events every time you want to read an entity's state gets expensive fast. The standard fix is snapshotting: periodically materialize current state and store it next to the log. On read, load the latest snapshot and replay only what's happened since. SQL SELECT * FROM snapshots WHERE stream_id = 'account-123' ORDER BY version DESC LIMIT 1; -- Then replay only events after snapshot.version SELECT * FROM events WHERE stream_id = 'account-123' AND version > :snapshot_version ORDER BY version; The CQRS Boundary In practice, event sourcing almost always shows up paired with CQRS (Command Query Responsibility Segregation) — splitting the write path (commands that produce events) from the read path (queries against a denormalized read model). This isn't just architectural tidiness. The event log is optimized for append throughput and sequential replay; it's a bad fit for ad-hoc queries, joins, sorting, or full-text search. ComponentRoleTypical technologyCommand handlerValidates commands, appends eventsApplication service layerEvent storeAppend-only source of truthPostgreSQL, EventStoreDB, KafkaProjectionReads events, builds the read modelBackground worker, Kafka consumerRead modelDenormalized query targetPostgreSQL, Redis, ElasticsearchQuery APIServes reads to clientsREST / GraphQL Consistency Models and the Lag Problem Projection Lag Is Real, Design For It Because the projection runs asynchronously, there's a window between an event landing in the store and the read model reflecting it. A user withdraws money and immediately checks their balance and might see the stale number. This is the central consistency problem in any CQRS/event-sourcing setup, and it needs an explicit answer, not a hope that it won't matter. In healthy systems on Kafka or a comparable bus, projection lag typically sits around 50–300ms. Under load spikes, partitions, or consumer restarts, it can stretch to seconds or minutes. Your design needs to be correct under the bad case, not just the median. Pick Your Battles: Selective Consistency The real design question isn't "How do I eliminate eventual consistency?" It's "Where do I actually need strong consistency, and where can I let the lag ride?" Those two buckets look pretty different: Data typeConsistency requiredWhyAccount balance post-withdrawalStrongFinancial correctnessPayment confirmationStrongRegulatory, user expectationRecommendation feedEventualA few seconds of lag is invisibleNotification badge countEventualApproximate is fineProduct inventory on a listing pageEventualMinor lag is tolerableSearch index for a new entityEventualUsers already expect crawl delay Read-Your-Own-Writes For the strong-consistency bucket, the most common mechanism is read-your-own-writes (RYOW). The command API hands back the version number it just appended; the client sends that version back on its next read; and the query service refuses to answer until the read model has caught up. Markdown # 1. Command returns the appended event version POST /accounts/123/withdraw {amount: 100} → HTTP 200 {status: 'ok', event_version: 42} # 2. Client sends version as constraint on next read GET /accounts/123/balance Headers: X-Min-Version: 42 # 3. Query service checks projection position SELECT last_processed_version FROM projection_checkpoints WHERE name = 'balance_projection'; -- If < 42, wait up to 500ms then retry or return 202 Accepted When You Just Bypass the Read Model For a small set of high-stakes flows such as payment confirmation screens, dispute resolution, regulatory audits, plenty of systems skip the read model entirely and rebuild state straight from the event log. It's always consistent, at the cost of latency. Snapshots (see above) keep that latency bounded even for entities with long histories. Three Ways to Keep Projections in Sync The projection needs to process each event exactly once and update the read model atomically with its own checkpoint. In practice, that comes down to three strategies, each suited to a different infrastructure setup. 1. Same-Database Transaction The simplest option: put the read model and the checkpoint table in the same relational database, and wrap the update and the checkpoint advance in one transaction. SQL BEGIN; UPDATE account_balances SET balance = 900 WHERE account_id = 123; UPDATE projection_checkpoints SET last_event_id = 42 WHERE name = 'balance_projection'; COMMIT; Either both writes land or neither does, which kills the two classic failure modes: processing an event without advancing the checkpoint (you'll reprocess it on restart) and advancing the checkpoint without writing the data (permanent, silent data loss). Great when your whole read model fits in one database. 2. Kafka + Idempotent Consumers Once Kafka is your event bus, consumer offsets live inside Kafka and can't be committed in the same transaction as your read-model write. Kafka only promises at-least-once delivery, so on a consumer restart, some events may show up twice. The fix is to make the consumer idempotent, in which case it's safe to run twice on the same event. SQL INSERT INTO account_balances (account_id, balance, last_event_id) VALUES (%s, %s, %s) ON CONFLICT (account_id) DO UPDATE SET balance = EXCLUDED.balance, last_event_id = EXCLUDED.last_event_id WHERE account_balances.last_event_id < EXCLUDED.last_event_id; That WHERE clause is the whole trick. If event version 42 gets replayed after it's already been applied, 42 < 42 is false and the update is a no-op. ScenarioDB state beforeIncoming eventConditionResultNormal processinglast_event_id = 41v = 4241 < 42 → TRUEUpdate appliedCrash replay (duplicate)last_event_id = 42v = 4242 < 42 → FALSESkipped safelyOut-of-order deliverylast_event_id = 42v = 4142 < 41 → FALSEStale event ignored The INSERT branch only ever fires once, when the entity's stream first shows up. Every event after that takes the ON CONFLICT path, and the version guard is what makes it idempotent. 3. The Transactional Outbox Pattern Things get harder when a projection needs to fan out to several different stores at once, say a relational read model, an Elasticsearch index, a Redis cache, and a webhook receiver. No single database transaction can span all of those. This is where the transactional outbox pattern earns its keep. The projection writes its read-model update and an outbox row in one local transaction. A separate relay process picks up pending outbox rows and delivers them to each downstream target, retrying until each one confirms. Because the outbox write is atomic with the read-model write, the relay always knows exactly what still needs to go out. SQL -- Atomic: both rows commit or neither does BEGIN; UPDATE account_balances SET balance = 900 WHERE account_id = 123; INSERT INTO outbox (id, payload, es_sent_at, redis_sent_at, webhook_sent_at) VALUES (gen_random_uuid(), '{"account_id": 123, "balance": 900, "event_version": 42}', NULL, NULL, NULL); COMMIT; Making the Outbox Relay Fault-Tolerant The relay delivers to each target independently and tracks confirmation per target. The scenario worth designing for: the relay writes successfully to target A, then crashes before marking the row delivered, so on restart, target A gets the same write again. Every target needs to handle that gracefully. Elasticsearch: Let the Document ID Do the Work Elasticsearch indexing is idempotent as long as you supply a stable document ID yourself. Index the same ID twice and you just overwrite the first doc with identical content, having no duplicates. Python # Non-idempotent: ES generates a random ID es.index(index='accounts', body={'balance': 900}) # → Two calls produce two documents # Idempotent: stable ID derived from the outbox row es.index( index='accounts', id=outbox_row.id, # same on every retry body={'balance': 900, 'event_version': 42} ) Redis: Absolute Values, Not Deltas SET in Redis is naturally idempotent — setting a key to the same value twice is the same as setting it once. The catch: your relay has to write absolute values, never deltas. INCRBY and LPUSH are not idempotent and have no business being in a relay path. Python # Non-idempotent: applying a delta twice corrupts state redis.incrby('account:123:balance', -100) # Idempotent: writing the derived absolute value redis.set('account:123:balance', 900) Webhooks: Idempotency Keys External webhook receivers are the trickiest case, because you don't control their implementation. The standard answer is the one Stripe documents publicly: an Idempotency-Key header. The relay sends the outbox row's UUID as that key on every attempt; the receiver keeps a table of keys it's already seen and just replays the original response for a repeat. Getting this right on the receiver side means handling three edge cases: Concurrent duplicate requests: Two requests with the same key arrive before either has been saved. Fix it with an atomic INSERT under a unique constraint; the second insert fails, and that thread waits on the first.Key reuse with a different body: Same key, different payload. Reject with a 422 after comparing a hash of the stored body to the incoming one.Key expiry: Keys can't live forever. Stripe uses a 24-hour TTL, which means your relay's retry window (and dead-lettering) has to finish well inside that boundary. The key record and the business-logic effect have to commit in the same transaction. If the business logic runs but the key never gets saved, a retry re-executes the logic and that's not a missed optimization, that's data corruption. Python def handle_request(request): key = request.headers['Idempotency-Key'] req_hash = sha256(request.body) with db.transaction(): row = db.execute(''' INSERT INTO idempotency_keys (key, request_hash, created_at, expires_at) VALUES (%s, %s, now(), now() + interval '24 hours') ON CONFLICT (key) DO UPDATE SET key = EXCLUDED.key RETURNING key, request_hash, response_status, response_body ''', [key, req_hash]).fetchone() if row.request_hash != req_hash: return 422, 'Key reused with different request body' if row.response_status is not None: return row.response_status, row.response_body result = run_business_logic(request) db.execute(''' UPDATE idempotency_keys SET response_status = %s, response_body = %s WHERE key = %s ''', [result.status, result.body, key]) return result.status, result.body Dead-Letter Escalation Transient failures — timeouts, a target being briefly down — get exponential backoff with per-target retry counters on the outbox row. Permanent failures — a schema mismatch, a decommissioned endpoint — shouldn't be allowed to block the relay forever. After a configurable number of attempts (10 is a common default), move the row to a dead-letter table for someone to look at. Who's Actually Running This in Production CompanyEvent volumePatternOutcomeNetflix>1B events/dayEvent sourcing + CQRS on KafkaPersonalization and monitoring for 260M+ subscribersUberMillions/minuteKafka + Flink, exactly-once semanticsFraud detection and dynamic pricing at global scaleLinkedInOrigin of Kafka (2010)Append-only activity stream logUnified data pipeline, replacing point-to-point integrationsSlackBillions of messages/dayKafka event propagation with full metadataDecoupled delivery and indexing under heavy loadStripeDocumented idempotency-key APITransactional idempotency at the webhook layerFormally specified 24-hour idempotency window with body-hash validation Takeaways A few things worth internalizing if you're building (or maintaining) one of these systems: Eventual consistency is a contract, not a bug. RYOW turns the consistency requirement into an explicit versioned header instead of an assumption. Idempotency keys turn "don't double-process this" into a first-class API contract. The outbox pattern turns delivery into a durable, queryable table instead of a fire-and-forget call. Once it's explicit, you can test it and monitor it — that's the whole point.Idempotency is the load-bearing primitive. It's standing in for distributed transactions, which are either unavailable, too slow, or too fragile for high-throughput event-driven systems. At-least-once delivery plus idempotent receivers gets you effectively-once behavior without any distributed coordination.The pattern underneath is always the same: a unique key derived from something stable (an event version, an outbox row's ID), stored atomically next to the effect it protects, checked before that effect is applied, and expired on a TTL that bounds how long you need to keep retrying.None of this locks you into specific technology. Event store in Postgres, EventStoreDB, or Kafka's compacted log — doesn't matter. Read model in MySQL, Redis, or Elasticsearch — doesn't matter. The outbox just needs to live in the same ACID store as the read model it's paired with. That's a low bar, and it's why teams can adopt this incrementally next to whatever they're already running. The hard part of event sourcing was never the event log where appending immutable rows is the easy 20%. The real engineering work is keeping derived state correct across multiple heterogeneous targets when any one of them can fail at any moment. Idempotency, applied consistently, is what makes that tractable without reaching for distributed transactions.
I built a pipeline framework on top of a convenience feature, and for about a year it looked like the best decision in the design. The framework generates analytics pipelines from declarative configuration: a user describes the events, stages, and metrics they care about, and the framework produces the Spark jobs, output tables, and dashboards. To make those pipelines incremental, I required every upstream source to be a table with change data feed enabled. The table format's changelog told us exactly which rows had been inserted, updated, or deleted since the last run, so every generated pipeline processed only deltas, for free, with no bookkeeping code of its own. The fragility hidden in that decision took longer to surface than it should have because it never showed up as a framework bug. It showed up as other people's schema changes. The Coupling Nobody Signed Up For When your only supported source type is a physical table with a change feed, every pipeline you generate is bound directly to that table's physical schema. Not to the concept the table represents, and not to some resilient subset of columns. It is bound to the table itself, exactly as its owning team defined it this week. Upstream teams in a large organization evolve their tables constantly, and they are right to. A column gets renamed to match a naming convention. A type gets widened. A table gets rebuilt as part of a storage migration and picks up a new layout. From the owning team's perspective, these are internal changes to an asset they own. From my framework's perspective, each one was a breaking change delivered without notice to every downstream pipeline reading the table. Because the framework made pipelines cheap to create, there were a lot of them. One upstream rename could break a dozen production pipelines in a single morning, none of which had changed at all. The standard answer to this problem is an isolation layer: put a view between the physical table and its consumers, and absorb schema changes in the view definition. That is exactly the layer my design made impossible. Change data feeds are a property of physical tables. A view does not emit a changelog. By requiring change-data-feed sources, I had not just failed to build an isolation layer. I had structurally forbidden anyone from adding one. The convenience feature and the fragility were the same feature. It is worth being precise about why the usual mitigations do not work here. Asking upstream teams to freeze their schemas does not scale past two or three teams, and should not, because it converts your fragility into their paralysis. Materializing private copies of every upstream table restores isolation but doubles storage and adds a freshness lag you now have to monitor. Contract tests detect the breakage earlier, which is worth something, but detection is not absorption. The pipelines still break; you just get a nicer error message. The only structural fix is to let pipelines depend on an interface rather than an implementation, which meant the framework had to stop requiring physical tables at all. Re-Architecting for Views and Arbitrary SQL Supporting views and arbitrary SQL queries as first-class upstream sources sounds like adding a source type. It is actually a rewrite of the framework's incremental-processing model, because the thing you lose when you give up the change feed is the framework's entire notion of "what's new." The replacement is less elegant and more honest: date-field-based processing. Every view or SQL source declares a date or timestamp field that partitions its data by processing period. In the declarative config, the difference between the old world and the new one is small on the surface: YAML sources: # Old style: physical table, incremental via change data feed orders_cdf: type: table name: prod.orders_v2 # New style: view or SQL, incremental via a declared date field orders_view: type: view name: analytics.orders_stable date_field: order_date sessions_sql: type: sql query: > SELECT s.session_id, s.user_id, s.started_at, d.channel FROM prod.sessions s JOIN prod.session_details d ON s.session_id = d.session_id date_field: started_at Underneath, everything changes. Instead of asking the storage layer which rows changed, the framework tracks which date ranges each pipeline has processed and generates predicates that read exactly the unprocessed ranges: SQL SELECT * FROM analytics.orders_stable WHERE order_date >= '{{ watermark }' AND order_date < '{{ run_end }' This is watermark-based incremental processing, one of the oldest patterns in data engineering, and there is a reason it has survived: it depends only on the logical content of the data, never on any physical property of how the data is stored. A view can answer a date-range predicate. A three-table join can answer it. A changelog can only be asked for by the physical table's direct reader. Moving to date-range semantics forced two other pieces of the re-architecture. The first was backfill handling. Change-feed processing has no natural story for reprocessing history; the feed only moves forward, and most implementations retain limited history anyway. Date-range processing makes backfill a first-class operation. Reprocessing means re-running a set of date ranges, and correctness means each range must be idempotent, overwriting that period's output atomically rather than appending to it. Building that idempotency into every generated pipeline, instead of leaving it to each pipeline author, was most of the work. It also required adding a true batch mode, because a three-year backfill and a daily increment are the same operation at different scales only if the execution engine treats them that way. The second piece was a source-contract layer, and this is the part I would build first if I were starting over. Each pipeline declares the columns it needs from each source, with expected types and nullability, and the framework validates every source against its contract before any run: Python def validate_contract(source_schema, contract): failures = [] for col in contract.columns: actual = source_schema.get(col.name) if actual is None: failures.append(f"missing column: {col.name}") elif not compatible(actual.dtype, col.dtype): failures.append( f"type drift on {col.name}: " f"expected {col.dtype}, found {actual.dtype}" ) elif col.required and actual.nullable: failures.append(f"nullability drift on {col.name}") if failures: raise SourceContractError(source=contract.source_name, failures=failures) When an upstream schema changes now, one of two things happens, and both are enormous improvements over silent breakage. Either the view definition absorbs the change, and the contract still validates, in which case downstream pipelines never know anything happened; or validation fails before the run starts, producing a named, actionable discrepancy (this column, this type, this source) instead of a runtime failure deep inside a generated Spark job at 3 a.m. What Actually Changed The measurable effect was the disappearance of a whole failure category. Schema drift incidents went from a recurring operational tax, each one a cross-team scramble to identify what changed and patch every affected pipeline, to a non-event absorbed at the view layer, usually by the upstream team itself as part of making their change. The less measurable effect was on how teams model their data. Once views were first-class sources, upstream teams could treat their physical tables as private implementation details and publish stable view interfaces on top, which is how table ownership should work anyway. The framework had previously punished that discipline. Now it rewarded it. The lesson I keep returning to is about where convenience features belong in a design. Change data feeds are a genuinely good optimization. My mistake was promoting an optimization into an interface. The framework's contract with its sources became "you are a physical table with a changelog" when it should have been "you can answer a date-range query and honor this column contract." Optimizations belong behind interfaces, where you can keep them for the sources that support them and fall back cleanly for the sources that do not. When an optimization becomes the interface, every one of its physical preconditions becomes a coupling point, and you inherit fragility from parts of the system you never intended to depend on. The tables were never supposed to be the dependency. The data was.
If you are a data scientist, data analyst, or anyone who works with big data, you have likely used SQLite, or even use it daily. Introduced in 2000, SQLite quickly became a popular choice for database-related tasks. Whether you used it in an introductory database module, built a local data pipeline, or managed local metadata in production, SQLite has long been the gold standard for a zero-configuration database, with more than 25 years of widespread use. But in recent years, DuckDB has emerged as a more powerful alternative, designed specifically for analytical workloads. For tasks involving Data analysis, analytics, and ETL (Extract, Transform, and Load), DuckDB is quietly gaining traction. The developer community today is increasingly migrating to DuckDB, especially when running analytical aggregations across gigabytes of local Parquet or CSV files, where it shows its speed advantage. So the question that arises is: Should DuckDB Replace SQLite? In this article, we will explore the fundamental design differences between the two libraries and determine whether DuckDB should replace SQLite. The Evolution of SQLite and DuckDB Evolutionary timeline comparing the hardware eras, workload requirements, and engine architectures of SQLite and DuckDB from 2000 to the present. (Image by the Author) SQLite was created by American software engineer D. Richard Hipp, whose original motivation for developing the library stemmed from his work with U.S. Navy destroyers. While working on Navy software, he wanted a database that could work reliably without a complex client-server system. As a result, SQLite was designed with the Navy’s specific requirements in mind. It was easy to use, with zero administrative overhead, and had excellent fault tolerance by eliminating the client-server architecture, providing continuous uptime. However, as datasets scaled to millions of records and multicore CPUs became the norm, some of SQLite’s original design choices became its core bottlenecks. Modern CPUs have multiple cores for better speed, but SQLite processes data row by row on a single thread. Moreover, it requires users to manually import files before querying them, which can result in slow import and export processes. Then came DuckDB in 2019, designed for fast analytical processing on local data. Although it was not intended to be a competitor or direct replacement for SQLite, it effectively solved the problem of single-core processing, prompting developers to use it more frequently for building data pipelines. The Developer Interface: Similar Syntax, Different Engines When it comes to the interface, there is no noticeable difference between the two libraries. If you are already familiar with SQL, then querying data with DuckDB requires virtually no learning curve compared to SQLite. SQLite Python import sqlite3 conn = sqlite3.connect("sales.db") cursor = conn.cursor() cursor.execute( "SELECT region, SUM(sales) FROM transactions WHERE sales > 1000 GROUP BY" " region" ) results = cursor.fetchall() DuckDB Python import duckdb # Querying local disk files directly with standard SQL results = duckdb.sql( "SELECT region, SUM(sales) FROM 'sales.csv' WHERE sales > 1000 GROUP BY" " region" ).fetchall() They may look similar at first glance, but the differences become clear when you look deeper into their architectures. Architectural comparison between SQLite’s sequential row-store processing and DuckDB’s optimized columnar vector execution pipeline. (Image by Author) Query Engines Processing Across CPU Cores SQLite uses the Volcano Iterator Model, where a single core processes queries sequentially on a single CPU thread. If you sort 10 million rows, a single CPU thread handles the entire task while the rest of the CPU sits idle, increasing processing time. On the other hand, DuckDB uses morsel-driven parallelism, partitioning the dataset into smaller chunks and processing them across available CPU threads through independent task queues. A large operation is split into uniform data chunks called morsels. Idle worker threads grab any available morsels from the queue, keeping the CPU cores fully utilized until execution finishes. Deferred Execution and Logical Plan Optimization SQLite executes queries in an eager streaming fashion where the rows are read immediately from disk or B-Trees and pushed through the expression evaluator step without global transformations. DuckDB enforces a deferred pipeline with an explicit optimization pass before the data is even touched. [ Raw SQL Query ] →[ Unoptimized Logical Plan ] → [Optimizer Pass ] → [ Physical Execution Plan ] DuckDB optimizes the query to eliminate non-essential work. During optimization, unused columns are dropped entirely, and conditions like WHERE sales > 1000 are pushed to the file reader level, skipping non-matching data blocks. Additionally, various query operations are arranged dynamically based on cardinalities to reduce intermediate memory allocation. Memory Efficiency and Data Storage Structures At last, we will look at the memory architecture, as the data transfer between RAM and CPU cache lines is frequently the primary bottleneck in data applications. Both libraries use fundamentally different data storage structures. SQLite uses a row-oriented storage layout, where it stores every feature of a record in a single row. Combined with an extremely minimal baseline footprint of just 2–5 MB, this design is optimal for retrieving an individual row (eg: loading a single user’s profile), but not for running an aggregation like AVG(sales), as it forces SQLite to load every unused attribute into cache along with the target values. DuckDB, on the other hand, uses a columnar format. Each feature is stored in a separate column, with the column data stored as a contiguous block in memory. This approach is preferred for analytical queries because DuckDB reads only the specific columns, maximizing CPU cache efficiency. The columnar structure also enables zero-copy data sharing with tools such as Apache Arrow, PyTorch, and Polars without expensive memory serialization steps. To maintain its multi-threaded vector channels, DuckDB operates with a higher baseline memory footprint of 20–30 MB. Final Thoughts: Should DuckDB Replace SQLite? After comparing the architectures of the two libraries, DuckDB may appear to be the better alternative. So, coming to our final question: Should DuckDB replace SQLite? The answer is: No, not quite. After looking at the architecture of the two database engines, the core trade-offs become clear: SQLite uses a row-oriented layout and a single-threaded engine to maintain a minimal memory footprint of just 2–5 MB. It is optimized for transactional (OLTP) operations, securing its position as a universal component across mobile and desktop devices. Conversely, DuckDB’s columnar architecture with multi-core parallel processing makes it a great choice for analytical (OLAP) queries and scanning external .csv, .parquet, or .json files directly without any manual imports. It turns out to be a better option for modern data science and AI pipelines. SQLite was engineered at a time when embedded storage was expected to handle lightweight applications and reliable local commits. In contrast, DuckDB was built to function as an analytics engine for large datasets. So which library you use for what task depends entirely on your use case. Choose SQLite if you need a zero-maintenance, embedded database to store app configuration or handle fast single-row lookups with minimal RAM overhead. Choose DuckDB if you are building a professional data or AI pipeline, running aggregations (GROUP BY, SUM, AVG) across millions of records, extracting features from .csv files, or processing local data in Python notebooks without a complex setup. Overall, it is better to see both as libraries designed for different purposes rather than as competitors. If you like the article, connect with me on https://medium.com/@rakshathnaik62 for more articles. I mainly write about Science, math, and AI.
What started as a recovery tool for streaming pipelines ended up as a full-fledged self-service platform, with scheduled watermark-based incremental ingestion, ad-hoc/one-time data replay, and throughput control built in along the way. The origin was narrow: streaming pipelines fail. A downstream consumer has an outage, a schema changes, a job crashes partway through, and, let's say, Kafka partitions are already gone due to max retention; when that happens, someone needs a way to re-publish a window of historical data without working with the source system to republish. We built a small tool to do exactly that: point it at a backup table (that held raw events), give it a date range, and it would replay that data back to a Kafka topic. As more teams started using this, they began asking questions like: can this be scheduled for a regular stream? Can we configure the TPS (Events Published/Sec)? Those questions are what turned a one-off recovery script into shared infrastructure. This article covers that evolution: the two design problems that mattered most in getting there — turning replay into a first-class, self-service capability, and making throughput control work correctly across a distributed Spark job, the harder of the two — plus the operational lessons that came from running it as infrastructure other teams now depend on. High-Level Architecture At its core, a user calls the Replay Service with a source table, a target Kafka topic, required TPS, and other optional parameters. The Replay Service kicks off a Spark job on serverless EMR through a Scheduler Service (Internal), which hands back a pipeline ID. The user uses that for a status request from the Replay Service, which parses the underlying job's logs and returns status - running, completed with a record count, or failed with a reason. Replay Service – Sequence Diagram Once the job is running, it reads from the source table and streams records to the target Kafka topic, where any number of downstream consumers can subscribe. The Spark job runs under its own AWS IAM role, separate from any team's own credentials, so onboarding a new table or topic is a matter of granting that role read access to the source table and write access to the target topic, not provisioning per-team infrastructure. Replay as a Self-Service Capability Replay ended up covering a few distinct shapes of the same underlying need: One-time replay: An ad hoc, manually triggered backfill for a specific date range, used for onboarding, backfill, or incident recovery.Scheduled replay: A recurring stream from a data lake table that's typically populated in batch, where the data needs to be shared with other systems, customer support, marketing, etc. The capability also allows customers to define a watermark (CDC) column, so subsequent pushes send only incremental data to the topic, improving performance and reducing cost.Auto topic management: Replay can provision its own topic to avoid colliding with the live feed, or, if needed, publish into the same live feed (a user-provided topic) so the data flows through the usual downstream setup. Keeping Throughput Steady Across a Distributed Job Often teams want to dictate TPS (Events Published/Sec) because the downstream system may or may not handle the volume the replay pipeline produces. Also, TPS needs to be deterministic to reliably estimate how long the backfill will take, or simply how long the job will run. The core challenge: a distributed job, a single target rate. A Spark job doesn't execute as one process. It runs as many parallel tasks spread across executor cores, each independently reading and publishing its own slice of data. If you want the whole job to publish at, say, 10,000 events per second in aggregate, you can't just tell every task "publish as fast as you can"; you have to divide that target across however many parallel task slots are actually running, and get every one of them pacing independently toward its share of the total. Here's the shape of the approach: Scala // NOTE: // sourceDataFrame is repartitioned to a multiple of available workers. For e.g. if total // number of table partition is 100 and available workers are 56, the data will be repartitioned // to 112 (RoundingWithCeil(100/56) ) so that every node gets fair share of records to publish val totalExecutorCount = spark.sparkContext.getExecutorMemoryStatus.size - 1 val coresPerExecutorCount = spark.conf.get("spark.executor.cores", "1").toInt val totalWorkers = totalExecutorCount * coresPerExecutorCount val tps = jobConfig.argumentConfig.tps val batchSize = math.max(1, tps / totalWorkers) val eventsPublished: LongAccumulator = spark.sparkContext.longAccumulator("eventsPublished") sourceDataFrame.foreachPartition((partition: Iterator[Row]) => { var partitionCount = 0L partition.grouped(batchSize).foreach { batch => val startTime = System.currentTimeMillis() batch.foreach { record => kafkaSink.value.send(targetTopic, headers, value) partitionCount += 1 } // Pace each batch to a 1-second window: if sending the batch took less // than a second, sleep out the remainder before starting the next one. val elapsed = System.currentTimeMillis() - startTime val sleepTime = 1000 - elapsed if (sleepTime > 0) Thread.sleep(sleepTime) } // flush() blocks until Kafka acknowledges all buffered records for this partition. // Only count after confirmed delivery, so the metric reflects events that // actually landed on the topic, not events that were merely handed to the producer. kafkaSink.value.flush() eventsPublished.add(partitionCount) }) A few things worth unpacking here: totalWorkers counts task slots, not executors. In Spark terms, it's executor count multiplied by spark.executor.cores (set at spark-submit time), the actual number of parallel task slots available, since each core runs its own independent task. Dividing tps by executor count alone would under-throttle any executor running more than one core.The rate limiter is per-task, not global. There's no central coordinator throttling the job as a whole. Each partition's task independently batches its records and sleeps out the remainder of each second. The aggregate rate emerges from every task pacing itself to the same per-slot target simultaneously.This only holds on a stable, dedicated cluster. totalWorkers is calculated once at the start of the job, so the whole scheme assumes the cluster isn't dynamically scaling mid-run. If executors are added or removed while the job is running, the fixed batchSize no longer matches the actual number of active task slots, and the real aggregate rate drifts from the configured target. It also assumes the job has the cluster's resources to itself: running on a shared, transient cluster where other jobs are competing for the same executors introduces exactly the kind of resource contention this mechanism is trying to prevent in the first place. In practice, this means running on a dedicated, statically-sized cluster per job rather than a shared or autoscaling one.This only produces a uniform aggregate rate if the data is repartitioned into a multiple of totalWorkers first. If partitions aren't evenly distributed across the available slots, some cores end up handling more partitions than others. Every task still paces to the same batchSize, but a core doing more partition-passes ends up contributing more to the real aggregate throughput than a core doing fewer. The per-task math is correct, but the cluster-wide result skews unless the upstream repartitioning step guarantees an even split. Getting this repartitioning right was the actual key to making the aggregate number line up with the configured target. The batching-and-sleep loop is straightforward once the data is distributed evenly going in.The count only advances on confirmed delivery. flush() blocks until Kafka acknowledges every buffered record in that partition, and only then does the accumulator increment. Without this, a job that crashes mid-run could report a published count higher than what Kafka actually has, which quietly breaks trust in the metric during incident investigations.Low TPS with many workers hits a floor. math.max(1, tps / totalWorkers) guarantees at least one message per task per second, so a target of, say, 10 TPS spread across 56 task slots still lets every task send 1 msg/sec, overshooting the aggregate target by up to 5.6x. This scheme is tuned for the large-backfill case it was built for; it under-throttles at very low target rates. At a high level: each partition publishes only its allocated number of events per second, then sleeps for the rest of the second. A fixed cluster size, repartitioning data to a multiple of total workers, and a batch size derived from the TPS target are what guarantee the aggregate throughput. Each streaming job also runs on its own dedicated serverless EMR instance, so there's no contention with other jobs for the same resources. A Few Supporting Capabilities A couple of other features round out the platform, worth a brief mention even though they weren't the hardest problems to solve: Custom headers on published messages, letting downstream consumers route or filter messages without inspecting payloads.On-the-fly decryption of encrypted (PII) columns, so sensitive source data can be safely streamed without requiring every downstream consumer to independently implement decryption. Running It as Shared Infrastructure Building the capability was one problem. Running it as infrastructure other teams depend on surfaced a couple of costs that don't show up until the platform has real usage. Dedicated clusters aren't free. The TPS guarantee described above only holds because every job gets its own statically-sized, serverless EMR cluster, with no sharing and no autoscaling mid-run. That isolation is what makes the throughput math work, but it also means the platform's infrastructure cost scales roughly linearly with the number of concurrent jobs, not with total data volume. A dozen small scheduled jobs cost as much in cluster overhead as a few large ones. It's the direct tradeoff for the guarantee: correctness under load, in exchange for giving up the efficiency of a shared pool. Self-service cuts both ways at off-boarding. Making it easy for a team to schedule a recurring replay made it just as easy to forget it exists. A scheduled pipeline is driven by its upstream dependency, so when the upstream stops running, the pipeline stops too. It doesn't keep quietly publishing after a team moves on. But the pipeline's configuration doesn't go away on its own: teams that deprecated the downstream consumer rarely came back to tell us, so the definition just sits there, cluttering inventory and skewing usage metrics even though nothing was actually running. This demands auto de-provisioning based on its last run check (i.e., this pipeline didn't run in the last X days) and clean up periodically. Summary A recovery tool and a self-service streaming platform look similar on the surface: both move data from a table to Kafka. What separates them is whether replay works as a capability that teams can self-serve rather than something an engineer has to trigger by hand, whether throughput holds steady when a dozen teams are running jobs against the same infrastructure at once, and whether the platform can be operated (onboarded, monitored, and off-boarded) without someone having to remember every job by hand.
The Problem A batch Spark job on YARN can show memory nearly full and CPU half idle at the same time, and you pay for both. Memory looks full because YARN reports the memory allocated to containers, not the memory in use. CPU looks idle because the executors that fit on each node don't add up to the node's cores. The cluster is billed for every vCPU and uses a fraction of them. Background: How Spark Sizes Itself on YARN Spark requests one YARN container per executor [3]. The container size is the executor heap (spark.executor.memory) plus off-heap overhead (spark.executor.memoryOverhead), which defaults to 10% of executor memory with a 384 MiB minimum [1]. The heap divides into a small reserved area, a unified region shared by execution and storage (60% of the usable heap by default, set by spark.memory.fraction), and the remainder for user data structures [2]. YARN also rounds each request up to its allocation increment, so real fits can differ slightly from paper math [4]. Two limits decide how many executors land on a node: the vCPUs and the memory that the NodeManager advertises. Whichever runs out first sets the count. Python executors_per_node = min( floor(node_vcores / cores_per_executor), floor(node_memory_mb / ((executor_memory_gb + overhead_gb) * 1024)) ) The Packing Model in Practice Take a worker that exposes 16 vCPUs and 112 GB (114,688 MB) to YARN, running executors with 5 cores each. By cores alone, the node fits three executors. Memory decides whether it actually does. Executor memory + overheadFit by memoryFit by coresExecutors per nodeBusy vCPUsCore utilization64 + 2 GB131531%48 + 2 GB2321062.5%32 + 1 GB3331594%24 + 1 GB4331594% The first two rows are the trap. A large executor looks generous, yet it strands cores that you still pay for. The third row is the sweet spot: the core limit and the memory limit meet. The last row shows that shrinking memory further gains nothing, because the core limit now binds. The overhead setting changes the packing too. With the default 10% factor, a 48 GB executor requests 4.8 GB of overhead. Setting it explicitly to a smaller value can raise the number of executors per node, and setting it too low can get containers killed for exceeding their memory limit. The misleading dashboard follows from the same model. YARN's memory-used metric counts allocated container memory. A node full of reservations looks 100% used even when the executors touch only part of their heap. High memory use plus low CPU use is therefore a signature of oversized executors, not a contradiction. The Approach The fix is to shrink the executor until the core limit binds instead of the memory limit. Keep the cores per executor fixed so that memory is the only variable. Five cores is the common starting point, a heuristic that grew out of HDFS client throughput [7][8]. Leave one or two vCPUs per node free for the NodeManager and the operating system. Then set the shuffle partition count. The default of 200 leaves a large cluster mostly idle during shuffles. A common rule is 2 to 3 times the total core count, checked against a partition size near 128 MB. When adaptive query execution is on, which is the default from Spark 3.2, Spark can coalesce small shuffle partitions at runtime, and the configured number sets the ceiling [6]. Plain Text spark.executor.cores 5 spark.executor.memory 32g spark.executor.memoryOverhead 1g spark.sql.shuffle.partitions 4800 # about 3x total cores (1,600 in this example) spark.dynamicAllocation.enabled true spark.shuffle.service.enabled true spark.speculation true spark.speculation.multiplier 3 spark.speculation.quantile 0.9 spark.executor.extraJavaOptions -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 A few of these deserve a comment. Executor autoscaling on YARN is usually paired with the external shuffle service so that shuffle files outlive their executors. Speculation relaunches tasks that run more than three times slower than the median once 90% of a stage's tasks have finished, which trims straggler tails at the price of some duplicate work. The G1 occupancy threshold of 35 starts concurrent marking earlier than the default of 45, trading some CPU for a lower risk of full garbage collections on large heaps. The Failure Mode: Local Disk Fixing CPU utilization can expose a limit you never reached before. More active cores per node means more concurrent shuffle writers on the same local disks. When a disk passes the NodeManager health checker's utilization threshold, which defaults to 90% and applies to both the local and log directories, YARN marks the node unhealthy and stops scheduling on it [4][5]. In a bad case, a large share of the cluster drops out mid-job, and the run has to be killed. The remedies are mechanical. Give workers larger local disks, size shuffle partitions so each one spills less, and watch the unhealthy-node count during any resizing experiment. Run the experiment at the quietest and busiest hours of the day, because the busy hour is where disk pressure shows up. Query-Level Tuning Still Matters Cluster sizing only goes so far. Find data skew first, since it is the most common cause of stuck stages. Prefer broadcast joins when one side is small enough to fit in executor memory, use equi-joins where possible, and always filter on the partition key. Look for joins between two very large datasets that a pre-aggregation could replace. Removing one such join can shrink the cluster a job needs far more than any configuration change. Trade-Offs Smaller executors fit more tasks per node, but each task gets less memory. Skewed tasks are more likely to hit out-of-memory errors, and broadcast variables are copied once per executor. Larger executors share broadcast data across more cores and carry less per-JVM overhead, but they can strand cores and tend to have longer garbage collection pauses. More active cores per node means more concurrent shuffle writes, so size local disks for the peak and not the average. The five-core rule is a heuristic. Test it against your own storage layer, whether that is HDFS or an object store. A Sizing Routine You Can Reuse Read the NodeManager memory and vCore limits from the ResourceManager UI. Fix the cores per executor. For each candidate memory size, compute executors per node with the function below. Pick the largest memory size that still reaches the core limit. Set shuffle partitions to 2 to 3 times the total cores, then check the partition size. Test at a quiet hour and at the peak, and watch NodeManager disk health. Python def executors_per_node(node_vcores, node_mem_mb, cores_per_exec, exec_mem_gb, overhead_gb): by_cpu = node_vcores // cores_per_exec by_mem = node_mem_mb // int((exec_mem_gb + overhead_gb) * 1024) return min(by_cpu, by_mem) for mem, ovh in [(64, 2), (48, 2), (32, 1), (24, 1)]: n = executors_per_node(16, 114688, 5, mem, ovh) print(mem, ovh, n, n * 5) # executors per node, busy vCPUs The Result Applied to a production batch job, this model explained a large gap between the cores the cluster paid for and the cores it used. Resizing the executors moved the busy cores close to the node's usable limit and shortened the run. The same job later needed fewer workers. The practical rule is simple: count the containers that fit on a node before changing any setting. Vikrant Bachhav is a Staff Software Engineer specializing in large-scale data infrastructure, real-time and batch ML feature pipelines, and cloud cost optimization. References [1] Apache Spark, Configuration [2] Apache Spark, Tuning Spark [3] Apache Spark, Running Spark on YARN [4] Apache Hadoop, yarn-default.xml [5] Amazon EMR, Cluster terminates with NO_SLAVE_LEFT and core nodes FAILED_BY_MASTER [6] Databricks, Adaptive Query Execution: Speeding Up Spark SQL at Runtime [7] Distribution of Executors, Cores and Memory for a Spark Application [8] PayPal Tech Blog, Tuning Spark Applications to Efficiently Utilize Dataproc Cluster
The message left the first service at 2:14 PM and never reached the last one. Between those facts: four pipeline stages, six topics, a pub/sub API moving every record. The engineer on call had Kafdrop open, the offsets looked fine, and nobody could say which stage had eaten the record. That gap is why we built message search for Kafdrop at a hackathon. The setup was familiar from my day job: a platform moving data between pipeline stages over a pub/sub API, with no way to ask where a message went when something breaks. Kafdrop showed partitions and offsets. It couldn't search inside the messages. Kafka has no index. That is the whole problem in one sentence. A topic is an append-only log split across partitions, and the consumer API reads by offset: hand it 41,207, and it returns record 41,207. There is no "find the record containing order-88412." To search, you pick a starting point and read forward until you find what you want or stop. So that is what our feature does, with guardrails that keep the search from hurting the cluster. The interesting part was the caps, the consumer setup, and the SRE agreement. Why a Full Scan Is Not an Option The obvious approach is to read the whole topic and grep. Topics hold days of data, so the whole thing doesn't fit in memory. Reading every record on every search hammers the brokers, and during an incident that is the worst time to add load. Most searches end with a few matches, so nearly all of that work is wasted. The other approach is to mirror every topic into a search engine. That works, and plenty of teams do it. It also means running a second distributed system and explaining index lag during every incident. For a hackathon debugging tool, that was too much machinery. The Design: Seek, Scan, and Stop The feature takes four inputs: a topic, a query string, a start time, and two caps. The start time bounds how far back the search goes. Kafka's offsetsForTimes maps a timestamp to the first offset at or after it per partition, or null when the partition holds nothing that new. The user picks "the last hour" or "since the deploy." The result cap bounds memory: return the first N matches and stop. Debugging usually needs one record, not ten thousand. The scan cap bounds broker load: stop after M records even with no match. Every search becomes a fixed amount of work, whatever the user types. A typo costs M reads. The scan can never grow beyond that. Polls use bounded batches, so the tool never holds more than one page of records at a time. Nothing loads a whole topic. Safety: Assign, Don't Subscribe The consumer setup is where a search tool can quietly cause damage. Three decisions were non-negotiable. First, the search consumer uses assign, not subscribe. Subscribe joins a consumer group, bringing coordination and rebalances. A debugging tool has no business in any of that. Assign hands the consumer a fixed partition list and stays out of group coordination entirely. Second, the consumer never commits offsets. Auto-commit is off, and the code never calls commit. The default config commits automatically, so this is the one that bites: a search tool that advances a production group's offsets is a horror story, and the fix is one config line. Third, the scan cap doubles as a safety control: every search reads at most M records, a worst case SRE can read before approving. The Search Loop Here is the core in Java. The three constants are the whole design: matches to return, records to read in the worst case, records per poll. The values are illustrative; tune them to your topics. Java private static final int RESULT_CAP = 25; private static final int SCAN_CAP = 50_000; private static final int POLL_BATCH = 500; public List<FoundMessage> search(String topic, String query, long startTimeMs) { Properties props = new Properties(); props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, brokers); props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, String.valueOf(POLL_BATCH)); List<FoundMessage> matches = new ArrayList<>(); try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) { List<TopicPartition> partitions = new ArrayList<>(); for (PartitionInfo p : consumer.partitionsFor(topic)) { partitions.add(new TopicPartition(topic, p.partition())); } consumer.assign(partitions); Map<TopicPartition, Long> times = new HashMap<>(); for (TopicPartition tp : partitions) times.put(tp, startTimeMs); for (Map.Entry<TopicPartition, OffsetAndTimestamp> e : consumer.offsetsForTimes(times).entrySet()) { if (e.getValue() != null) consumer.seek(e.getKey(), e.getValue().offset()); else consumer.seekToEnd(Collections.singletonList(e.getKey())); } long scanned = 0; while (scanned < SCAN_CAP && matches.size() < RESULT_CAP) { ConsumerRecords<String, String> batch = consumer.poll(Duration.ofMillis(500)); if (batch.isEmpty()) break; for (ConsumerRecord<String, String> r : batch) { if (++scanned > SCAN_CAP) break; if (r.value() != null && r.value().contains(query)) { matches.add(new FoundMessage(r.topic(), r.partition(), r.offset(), r.timestamp(), r.value())); if (matches.size() >= RESULT_CAP) break; } } } } return matches; } From the top: the config disables auto-commit and fixes the poll batch size. Assign takes a fixed partition list. offsetsForTimes turns the start time into a starting offset per partition; the null branch seeks to the end for partitions with nothing that new. The loop reads forward and stops at whichever cap hits first. Nothing commits, and try-with-resources closes the consumer. Matching is a substring check on the record value. Tombstones carry nulls, so the null check is load-bearing. Shipping It: A Container, Kubernetes, and the SRE Agreement The app ships as a container image: the search service and its config, nothing else. It runs on Kubernetes as a small deployment with tight resource limits, since the result cap already bounds memory. One replica is plenty. YAML apiVersion: apps/v1 kind: Deployment metadata: name: kafdrop-search spec: replicas: 1 selector: matchLabels: app: kafdrop-search template: metadata: labels: app: kafdrop-search spec: containers: - name: search image: registry.example.com/kafdrop-search:1.0.0 resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" env: - name: KAFKA_BROKERS value: "kafka:9092" The manifest is deliberately boring: one replica, tight memory limits, brokers passed as config. The SRE conversation was short, because the design answered their questions first. The tool only reads: assign instead of subscribe, no offset commits, read-only Kafka credentials. Every search is bounded by the scan cap, so the worst case was a number everyone could read. That agreement mattered more than the code. The caps were the trust. A Debugging Routine That Works First, search the topic for the message using something unique: an order ID, a request ID. Found it? Note the offset and timestamp. Second, check consumer group lag for the downstream stage. If the group consumed past your message's offset, the stage read it, and the problem is in the consumer's logic. If lag is stuck or growing, the consumer is behind, crashed, or wedged on a poison record. Third, look at the consumer: logs around the message's timestamp, error handling, the dead-letter path. The order matters. It separates "the message never arrived" from "the message arrived and something went wrong." Before, step one was guesswork. Now it takes minutes. What We Kept Kafka gives you ordering and durability, not search. Bolting search onto an append-only log means accepting the scan. Make it bounded and boring: seek to a timestamp, read forward in small batches, stop at the first cap. Keep the three decisions that made it safe. Assign instead of subscribe, so the tool never joins a consumer group. Never commit offsets, so reading changes nothing. A scan cap that turns every search into a known quantity of broker work. Get those right, and SRE will let you run it against production. Get them wrong, and you have built a very efficient way to cause your next incident. After the hackathon, the team kept the tool for one job: finding where a message stopped. A search, a lag check, and a look at the consumer, done in minutes. Diagnosis time during incidents fell.
If Your Fabric Capacity Is Throttling, Do Not Start With the SKU A report takes twenty seconds to open. A pipeline that normally finishes before breakfast is still running. Someone sees the capacity above 100% and proposes the obvious fix: move from F64 to F128. That might be the right decision. It should not be the first one. Microsoft Fabric brings data engineering, warehousing, real-time analytics, data science, and Power BI onto shared capacity. That is one of its strengths. It also means one inefficient item can consume compute that another workload needs. A poorly shaped warehouse query, an oversized Spark session, an hourly refresh of data that changes once a day, or a report page with too many expensive visuals can all surface as the same complaint: “Fabric is slow.” The complaint does not tell you the cause. The useful operating model is: Confirm the capacity is throttling. Isolate the operation creating pressure. Optimize that workload. Scale only when the remaining demand is legitimate. This distinction matters because a larger SKU gives inefficient work more room to run. It does not make the work efficient. First, Separate Slow Performance From Throttling Slow performance and capacity throttling can look similar to a user, but they are different problems. A report visual can be slow because its DAX is expensive. A warehouse query can be slow because it scans far more data than it needs. A Spark job can wait because it requested more executors than are available. Network latency, gateway pressure, cold cache, locks, and source-system performance can also add delay. Throttling is more specific. It happens when billable operations consume more Capacity Unit seconds than the capacity can supply, including the future capacity Fabric uses to absorb smoothed usage. When that budget is exhausted, Fabric progressively delays or rejects new work. This leads to the first rule: Do not diagnose throttling from a slow screen or a utilization spike. Confirm it in the Capacity Metrics app. Microsoft explicitly notes that utilization above 100% does not immediately mean throttling. Fabric allows bursting, smooths usage, and provides limited overage protection before user operations are affected. Use the Throttling view and System events on the Compute page to confirm whether the capacity entered interactive delay, interactive rejection, or background rejection. Rejected requests can also surface errors such as CapacityLimitExceeded. If the app shows no throttling during the incident, investigate the item itself. Buying capacity to fix a bad query is an expensive way to avoid reading its execution plan. What Fabric Is Protecting You From Fabric does not force every operation to fit inside the CU budget of the current second. It uses two mechanisms to absorb variability. Bursting lets an operation temporarily use more compute than the provisioned SKU provides so it can finish quickly. Smoothing spreads the CU accounting for that operation across future 30-second timepoints. Interactive operations are typically smoothed over five to 64 minutes, depending on consumption. Background operations are typically smoothed over 24 hours. Smoothing is not a performance optimization. It does not make a query cheaper or slow a job down to fit the capacity. It changes how consumed compute is applied against the capacity limits. When smoothed usage continues to accumulate faster than idle capacity can pay it down, Fabric moves through progressive protection stages: Future capacity consumedProtection stageUser impactUp to 10 minutesOverage protectionNew operations continue without throttlingMore than 10 minutes, up to 60 minutesInteractive delayNew interactive operations receive a 20-second submission delayMore than 60 minutes, up to 24 hoursInteractive rejectionNew interactive operations are rejectedMore than 24 hoursBackground rejectionNew interactive and background operations are rejected Operations already running are allowed to complete. The capacity recovers when unused CUs burn down the accumulated carryforward. That delayed accounting explains a common mystery. The item users see failing at 10:00 AM might not be the item that created the pressure at 10:00 AM. A large background operation completed earlier can still be consuming future capacity through smoothing. Diagnose From the Capacity Down, Not From the complaint up When a capacity incident occurs, use the same investigation path every time. 1. Pin Down the Affected Window Record the time, capacity, workspace, reported symptom, and whether the failure affected interactive work, background work, or both. Do not begin with a full month of data. Start with the narrowest window that contains the incident. The Capacity Metrics app shows the previous 14 days of compute usage and displays hourly usage in the multimetric ribbon. Select the affected day and hour. When filters are applied, the utilization and throttling visuals can show 30-second timepoints rather than only periodic peaks. 2. Confirm the Protection Stage Open the Throttling tab and check which threshold crossed 100%: Interactive delay means the capacity consumed more than 10 minutes of future capacity.Interactive rejection means it consumed more than 60 minutes.Background rejection means it consumed more than 24 hours. Then review Overage (Carryforward). The add, burndown, and cumulative values tell you whether the capacity experienced a short surge that quickly recovered or sustained demand that kept adding debt. Minutes to burndown is especially useful. It estimates how long recovery would take if no new operations entered the capacity. 3. Find the Items That Consumed the CUs In the matrix by item and operation, sort by CU(s) for the selected window. Expand the highest consumers to see the operation type. Duration alone is not enough. A long-running operation can use modest compute, while a short burst can consume many CUs. Add the optional columns that help explain the incident: Rejected countFailed countSuccessful countIn-progress countOverloaded minutesItem sizePerformance delta Hovering over matrix values exposes the operation-level breakdown. Capture the workspace, item, operation, user or service principal, CU seconds, operation count, and time. 4. Classify the Shape of the Problem Most incidents fall into one of four patterns: PatternWhat the evidence looks likeLikely responseOne expensive operationA single run dominates CU secondsOptimize code, query, model, or data layoutExcessive frequencyA modest operation runs far more often than its data changesReduce refresh or execution frequencyHigh concurrencyMany acceptable operations overlapControl concurrency, isolate workloads, or scaleSustained business demandOptimized work remains consistently highScale up or scale out This classification prevents random tuning. It tells you whether to reduce cost per operation, reduce operation count, change where the work runs, or provide more compute. Reducing frequency and rescheduling are not the same optimization. Changing an hourly refresh to a daily refresh removes unnecessary executions and lowers total consumption. Moving the same 24 hourly refreshes to a different part of the day does not make them cheaper. It only changes when they compete with other work. Optimize the Workload That Actually Caused the Pressure Fabric workloads share capacity, but they do not share one tuning playbook. Once you know which operation consumed the CUs, focus on that workload instead of trying to optimize everything. Warehouse and SQL Analytics Endpoint Start with Query Insights to identify completed queries using the most CPU. Microsoft's Query Insights documentation includes this query: SQL SELECT TOP 100 distributed_statement_id, query_hash, allocated_cpu_time_ms, label, command FROM queryinsights.exec_requests_history ORDER BY allocated_cpu_time_ms DESC; If a query stands out, check how much data it scans, whether filters are applied early, and whether joins or statistics need attention. Compare repeated executions, not just the first run, because cold starts can distort the picture. The goal is fewer resources consumed for the same result, not simply a faster first response. Spark, Notebooks, and Lakehouse Processing Use Spark monitoring and the Tune tab to look for heavy shuffles, skewed stages, spills, or executors that sit underused. Filter data early, avoid repeated actions, and review Delta file layout when small files become a recurring problem. Also check session lifetime. Leaving compute running after the work is done is easy to overlook. If bursty Spark workloads regularly compete with reporting, consider whether separately billed on-demand Spark is a better fit. Measure the cost and isolation trade-off before making the switch. Data Factory Pipelines and Dataflow Gen2 A pipeline can orchestrate several expensive operations, so the pipeline itself may not be where the CUs are going. Check the notebook, copy, refresh, or transformation activity behind it. The simplest savings often come from removing unnecessary runs, using incremental loads, and preserving query folding in Dataflow Gen2 where supported. Moving the same work to another hour might reduce contention, but it does not reduce total CU consumption. Removing work you do not need does. Power BI Semantic Models and Reports Separate refresh pressure from interactive report pressure. For refreshes, focus on incremental processing, folding, and schedules that match how often the source changes. For reports, use Performance Analyzer to find expensive visuals and DAX measures, then compare those findings with the semantic model's operations in Capacity Metrics. A page with too many costly visuals can feel like a capacity problem even when the better fix is in the report design. Real-Time Intelligence For KQL databases and Eventhouses, inspect wide queries, repeated aggregations, retention and caching choices, and ingestion patterns. Limit scans to the data actually needed and consider pre-aggregation when queries repeatedly calculate the same result. Keep the freshness requirement in view. Lower CU consumption is not a win if the data now arrives too late to be useful. Real-Time Intelligence also has different throttling behavior, so do not assume every workload uses the same interactive delay stage. Reduce Blast Radius When Optimization Is Not Enough Some workloads are efficient and still should not share a failure boundary. An executive reporting estate, a self-service experimentation workspace, and a large overnight engineering platform can have different service expectations. If one workload is allowed to consume the headroom needed by another, the architecture is coupling them operationally even if their data products are separate. Fabric gives you several options: Scale up when optimized demand legitimately exceeds the current SKU.Scale out by moving workspaces to another capacity when isolation, administration, or service levels differ. This adds capacity cost and operational ownership, so the service-level benefit must justify it.Use on-demand billing for Spark when dynamic Spark demand should not consume the shared capacity.Configure surge protection to limit the background compute that can consume capacity and make interactive throttling less likely.Enable capacity overage billing when continuity is worth the additional cost and the feature fits your operating model. Microsoft documents overage billing at three times the normal capacity rate, so estimate the cost first. Surge protection is a guardrail, not free capacity. When it activates, background jobs can be rejected. Microsoft recommends correctly sizing and isolating critical solutions rather than treating surge protection as a replacement for optimization or capacity design. Prove That the Optimization Worked A workload is not optimized because its code changed or one test completed faster. Compare the same business operation before and after the change. Track: MeasureWhy it mattersCU seconds per operationShows whether the work became cheaperOperation countExposes unnecessary frequency and retriesTotal CU secondsShows the net capacity effectDurationMeasures user or batch completion timeData scannedReveals pruning, cache, and query-shape improvementsRejected and failed countConfirms whether capacity pressure still disrupts workCarryforward and minutes to burndownShows whether the capacity recovers fasterInteractive latencyConfirms that users experience the improvement Run the comparison over a representative period. A change that improves one query but shifts expensive work into refresh, maintenance, or another engine is not an end-to-end optimization. For recurring operations, calculate both cost per run and daily cost: Plain Text Daily CU-seconds = CU-seconds per operation × operations per day Reducing a query from 1,000 to 800 CU seconds saves 20% per run. If it runs ten times more often after the redesign, total consumption still increases substantially. Efficiency and frequency belong in the same calculation. A Practical Throttling Response Worksheet Use this during the next capacity incident: Plain Text Incident window: Capacity and SKU: User-visible symptom: Confirmed throttling stage: Carryforward and minutes to burndown: Top workspace: Top item: Top operation: Interactive or background: CU-seconds: Operation count: Rejected count: Problem shape: [ ] One expensive operation [ ] Excessive frequency [ ] High concurrency [ ] Sustained business demand Optimization applied: Before CU-seconds per operation: After CU-seconds per operation: Before daily CU-seconds: After daily CU-seconds: Problem shifted to refresh, maintenance, or another engine: Decision: [ ] Keep current SKU [ ] Scale up [ ] Isolate on another capacity [ ] Move Spark to on-demand billing [ ] Add a protective capacity control This turns a vague performance complaint into a traceable engineering decision. The Goal Is Not Zero Utilization A well-used Fabric capacity should do work. The goal is not to keep every chart comfortably low or eliminate every temporary excursion above 100%. The goal is to ensure that each operation earns the compute it consumes, critical workloads retain the service level they need, and sustained business demand fits inside a deliberate capacity envelope. When throttling appears, ask these questions in order: Did the capacity actually throttle, or is one item simply slow?Which item and operation consumed the CUs?Is the problem cost per operation, frequency, concurrency, or real growth?Can the workload be made cheaper without reducing the business outcome?If not, should the answer be more capacity or better isolation? That is the difference between reacting to throttling and operating Fabric well. Sources This article is grounded in current Microsoft documentation. Product behavior and preview features can change, so verify the latest guidance before making production capacity decisions. Understand capacity throttling and smoothing (Microsoft Learn)Capacity Metrics app compute page (Microsoft Learn)Evaluate and optimize your Microsoft Fabric capacity (Microsoft Learn)Performance guidelines for Fabric Data Warehouse (Microsoft Learn)Query Insights in Fabric Data Warehouse (Microsoft Learn)On-demand billing for Apache Spark in Microsoft Fabric (Microsoft Learn)Optimization guide for Power BI (Microsoft Learn)Manage surge protection for Fabric capacities (Microsoft Learn)Data clustering in Fabric Data Warehouse (preview) (Microsoft Learn)Partitioning for Delta tables (Microsoft Learn)
Edge to cloud is not one integration problem. It is four: telemetry going up, control going down, sites syncing sideways, and data reaching people. Each has different physics. Most failed projects picked one technology for all four. Every enterprise with factories, stores, vehicles, hospitals, or devices faces the same architectural question. How does data move between the edge and the data center or cloud, in both directions, reliably, securely, and at acceptable cost? The vendor landscape answers with dozens of overlapping options: MQTT platforms, hyperscaler services, event streaming, event mesh, messaging fabrics, delivery platforms, and protocol proxies. This article structures the option space by traffic pattern instead of by product category, maps the vendor categories onto those patterns, and closes with a decision framework you can apply to your own architecture. The categories here sit alongside the wider integration landscape rather than inside it. For that broader map across event streaming, API, and batch, see the Data Integration Landscape 2026. What Does Edge-to-Cloud Communication Mean? Edge-to-cloud communication is the movement of data between distributed physical locations and centralized computing environments, in both directions, across networks the architect does not control. The word edge hides four very different tiers: The device edge means microcontrollers, sensors, and actuators with kilobytes of memory.The gateway edge means industrial PCs, vehicle computers, and protocol converters.The site edge means a factory, store, hospital, or ship with local compute and its own network boundary.The near edge means regional data centers close to many sites. A technology that fits one tier often fails at another. Edge to cloud splits into four traffic patterns: northbound telemetry, southbound control, site-to-site sync, and outbound delivery to people. The traffic crossing these tiers falls into four patterns: Northbound telemetry moves sensor readings, machine states, transactions, and logs from the edge toward central platforms for analytics, AI, and business systems.Southbound control moves commands, configuration, software updates, and increasingly AI models from central platforms down to machines and gateways.Site-to-site traffic synchronizes data between sites, between a site and a core data center, or between clouds.Outbound delivery pushes live data from central platforms to the applications people use: dashboards, mobile apps, and trading screens. Direction plus consumer type plus constraint determines the right technology. Vendor preference does not. Why Do Network Constraints Decide the Edge-to-Cloud Architecture? The network between edge and cloud is the one component you cannot refactor. Its properties dictate the architecture before any product evaluation starts. Connectivity is unreliable. Ships, vehicles, and remote sites go offline for hours or days. Any viable architecture is offline first: local operation continues, data buffers locally, and synchronization resumes when the link returns. This is why store and forward is the single most important capability on any edge evaluation checklist. Inbound connections are forbidden. Industrial sites, retail networks, and vehicle fleets sit behind NAT, firewalls, and cellular carriers. Security teams do not open inbound ports for messaging. Every serious edge technology therefore uses the same trick: the edge dials out. MQTT clients dial out to brokers. NATS leaf nodes dial out to hub clusters. Azure IoT Operations dials out from the factory to the cloud broker. Different stacks, identical answer, because the constraint is identical. Hardware is constrained and heterogeneous. An 8-bit microcontroller, a fanless gateway PC, and a site server need different footprints. Some technologies scale down to bare microcontrollers. Others assume a Kubernetes cluster. Matching footprint to tier is half the selection process. Footprint is not the only axis. In fixed installations like plants and warehouses, the device edge can stay thin, because the uplink is reliable and most intelligence lives upstream. In the field, agriculture, cranes, rural grid, remote or contested sites, the uplink is wireless and often degraded, saturated, expensive, or absent for days. There, the edge cannot just buffer and forward. It has to run enough local intelligence, pattern matching, image recognition, autonomous decisions, to operate correctly on its own and sync opportunistically when the link returns. That pushes compute onto the device tier for reasons that have nothing to do with protocol size, and it changes the pattern, not just the product. Data residency is now an architecture requirement, not a legal afterthought. European manufacturers increasingly must prove what data leaves the site, country, or jurisdiction. Topic design, routing rules, and retention policies become compliance controls. And one constraint is usually invented rather than measured: latency. Teams assert real-time requirements that the business case does not contain. A large share of edge traffic is perfectly fine as periodic batch over a metered link. Keep that thought. It returns in the site-to-site section. Northbound: How Does Telemetry Get from OT and Devices into the Core? Northbound telemetry is the pattern the industry knows best, and the one with the most settled consensus. At the machine level, OPC UA provides the semantic information model. It defines what a temperature, a batch, or a machine state means. MQTT provides the efficient transport, and Sparkplug adds topic structure, birth and death certificates, and state awareness on top of MQTT. The two standards stopped being rivals. OPC UA Pub/Sub over MQTT is the convergence point: OPC UA semantics carried over MQTT infrastructure. Edge gateways and industrial DataOps products, for example HighByte or Ignition with its MQTT modules, convert brownfield protocols like Modbus and S7 into that stack. The Unified Namespace (UNS) pattern then organizes all site data as a live, hierarchical topic structure on a central broker. I covered UNS design and its trade-offs in depth in Unified Namespace vs. Data Product in IT/OT for Industrial IoT and the broader MQTT market in MQTT Market Trends: Cloud, Unified Namespace, Sparkplug, Kafka Integration, so this section stays short. The real decision is the broker and its operating model. Self-managed platforms carry the northbound load for most industrial estates: HiveMQ and EMQX at the enterprise end, Mosquitto and NanoMQ where a lightweight or embedded footprint matters. They give you deployment control, protocol depth, and edge variants that run on gateways. Managed cloud services remove the operations burden and integrate natively with cloud analytics. The dividing line is usually sovereignty and site autonomy: if the broker must run inside the factory, on the ship, or in a specific jurisdiction, self-managed wins. If all sites have reliable connectivity and the data may leave, managed wins on cost of operations. The hyperscalers themselves tell an asymmetric story. Google retired Cloud IoT Core in 2023 and exited the category. AWS IoT Core remains the steady incumbent for managed device connectivity. Azure went furthest architecturally: Azure IoT Operations runs a distributed MQTT broker on Arc-enabled Kubernetes at the site edge and bridges it to the Event Grid MQTT broker in the cloud, which now supports Sparkplug natively. That is a symmetric edge plus cloud MQTT pair from a single vendor, and it is a new pattern. One hyperscaler retreated, one held, one doubled down. The lesson for architects: managed convenience is real, but so is the risk that a cloud provider deprecates your ingestion layer. The open standards underneath, MQTT and OPC UA, are the insurance policy. Choose products that speak them, and migration remains possible. Southbound and Outbound: How Do Data and Commands Flow Back to Machines and People? Almost all edge content covers ingestion. The return direction is harder, and it is where architectures break, because the return direction has two completely different consumers. Machines need commands and state. Request-response over pub/sub topics handles commands: reboot this gateway, close this valve, change this setpoint. Retained messages and device twins handle state: a device that reconnects must immediately learn its desired configuration without waiting for the next command. Fleet-wide rollout needs staged delivery, and the payloads keep growing. Configuration files became firmware images, and firmware images are becoming AI models. Shipping a quantized model to ten thousand gateways is a southbound data movement problem, not an MLOps footnote. The same fabric increasingly carries traffic for agents at the edge: AI agents that perceive local data, act locally, and coordinate through the messaging layer. NATS and the MQTT platforms are both positioning for exactly this workload. People need the current picture. A plant manager opening a dashboard, an operator reconnecting after a network drop, and a trader loading a screen mid-session all have the same requirement: current state on connect, then live updates. A log is not a user interface. Machines just consume the stream. People do not. This is the last-mile delivery problem, and it has its own vendor category: Lightstreamer, Ably, and PubNub sit between the central platform and many concurrent browsers and mobile apps, handling fan-out, per-client throttling, and entitlement checks. The category is maturing fast. For instance, Lightstreamer's Kafka Connector 2.0, released in July 2026, moved snapshot management into the connector itself, so every reconnecting client receives materialized state followed by the live stream. Delivery to humans deserves a dedicated article. Here it earns its place as the fourth traffic pattern that most edge architectures forget until the first user complains. Site-to-Site: How Do Sites, Data Centers, and Clouds Stay in Sync? The third direction is sideways: keeping data consistent between many sites and one core, or between multiple clouds. For durable event streams, Kafka-style replication is the default. Open source MirrorMaker 2 or vendor-specific cluster linking copies topics between an edge cluster and the central cluster, preserving ordering and history. Apache Kafka is the open-source option; Confluent, Amazon MSK, Aiven, and other solutions are the managed ones. This suits sites large enough to operate a streaming cluster, such as a factory with local analytics that also feeds the global platform. For thousands of small sites, the streaming cluster per site model collapses under its own operational weight. This is where NATS is architecturally distinct rather than just another broker. The entire server is a single small binary with no external dependencies. Each store, vehicle, or clinic runs a leaf node that dials out to the central cluster, keeps serving local applications when the WAN drops, and uses JetStream mirroring to synchronize persisted streams when connectivity returns. Retail chains run this pattern across thousands of locations. The same design serves both the intermittent case, hours or days offline with a bulk sync on reconnect, and the scheduled-batch case where sites replicate on a timer rather than continuously. Its own maintainers warn against one trap: do not stretch a single stream cluster across unstable edge links, because the cluster's consensus breaks every time the network does. Instead, let each site persist its data locally and keep serving through an outage, then mirror to the center asynchronously when the link returns. For enterprises whose sites run applications rather than device fleets, an event mesh covers the same ground at the application layer. Solace links its brokers across data centers and clouds so an event published anywhere reaches any interested subscriber, with topic routing and governance built in. It is the natural fit when the endpoints are SAP, mainframes, and microservices rather than sensors. And a large share of site-to-site traffic does not need streaming, even if streaming is often the stronger long-term architecture. Nightly file synchronization, batch uploads over metered links, and change data capture from edge databases remain perfectly valid when there is no real-time requirement. This is especially true when the existing OT and IT systems on both ends are batch-oriented anyway, and no business case justifies the cost and effort of making them real time. Real time is a requirement, not a default. Stream when you need fresh data, or when you build greenfield and carry no legacy batch systems to preserve. An architecture that streams everything pays streaming costs for batch problems. Which Category Serves Which Edge to Cloud Pattern? Seven vendor categories map onto the four patterns. No single category covers all of them, and messaging fabrics, event mesh, and protocol proxies deliberately span several. No single vendor covers every direction. Seven categories divide the field, and the real picture is that some sit in one pattern while others deliberately span several. MQTT platforms (EMQX, HiveMQ, Mosquitto, NanoMQ) serve both northbound telemetry and southbound control, because MQTT is bidirectional by design. Hyperscaler services (AWS IoT Core, Azure IoT Operations) do the same for the managed-cloud case, ingesting telemetry and pushing desired state through device twins. Event streaming (Apache Kafka, Confluent, Aiven, and more in the data streaming landscape) owns site-to-site replication. Last-mile delivery(Ably, Lightstreamer, PubNub) owns outbound delivery to people, and it is the only pattern with a category built solely for it. Three categories span multiple patterns. An event mesh (Solace) routes site-to-site and out to applications. Messaging fabrics (NATS, Zenoh) carry northbound, southbound, and site-to-site in a single fabric. Protocol proxies (Zilla) translate across all four, replacing several specialized components with one layer. The underlying distinction is worth stating plainly: a protocol standard is not a deployment architecture. MQTT standardizes the wire, which is why so many products speak it and your devices stay portable. NATS and Zenoh standardize the topology, which reduces moving parts and increases commitment. Zenoh, born from DDS research and adopted by automotive initiatives including General Motors' uProtocol and by the ROS 2 robotics ecosystem, goes furthest: it routes named data along the shortest path between producer and consumer, aiming to make the cloud versus edge placement debate irrelevant. These spanning options are not equivalent. A fabric like NATS or Zenoh reduces moving parts and unifies security by being one technology end to end, but that reach is also the lock-in: the more it carries, the more expensive it is to leave. A protocol proxy like Zilla does the opposite. It spans the patterns by translating between existing technologies, which keeps your exit options open at the cost of another component in the path. The right choice depends on how many patterns you actually need and how much lock-in you can accept, not on which vendor markets hardest. How to Choose: Five Questions That Pick Your Edge-to-Cloud Architecture Product evaluations start too late. Answer five questions first, and the shortlist mostly writes itself. Five-question decision framework for choosing an edge-to-cloud architecture, from consumer type to data residency Who consumes the data, machines or people? People need current state on connect, then live updates, which means a last-mile delivery layer over the backbone. Machines just consume the stream.Does this flow actually need real time? Measure before you architect. If a fifteen-minute batch satisfies the business case, use file sync or change data capture. Stream when you need fresh data, because streaming infrastructure you did not need is the most expensive kind.What is the most constrained node in the path? A microcontroller means MQTT or Zenoh, the two with clients small enough for the device itself, not a JVM. A gateway opens the field to lightweight NATS or full brokers, and a site server adds event meshes or a small streaming cluster.How many sites, and can each one operate a cluster? A few large sites with local analytics justify edge Kafka clusters with MirrorMaker or cluster linking. Hundreds or thousands of small sites demand hub-and-spoke: NATS leaf nodes or hierarchical MQTT bridges, with store and forward at every spoke. Enterprises whose sites run applications rather than devices fit an event mesh.Where must the data live? If residency or sovereignty rules apply, the broker runs on site or in jurisdiction, self-managed, and routing policy becomes a compliance control. If data may leave and connectivity is solid, managed cloud services win on operations cost. Two checks apply on almost every branch, at every step. The edge must dial out and never accept inbound connections to the site. And local operation must survive offline, with a buffer sized deliberately, not discovered in the first outage. Five questions, four patterns, one map. The technologies then sort themselves into it. Moving Data Is Not Automating the Business Every pattern and category in this article solves the same problem: moving data between edge, data center, and cloud. That movement is necessary but not sufficient. When telemetry arrives, something must decide what happens next: trigger the maintenance workflow, roll back the failed firmware deployment, replicate the dataset, open the ticket, retrain the model. Deciding what happens next is workflow orchestration. It spans the same edge-to-cloud boundary as the data movement beneath it. Data integration moves the events. Orchestration, with the process intelligence to see how those flows actually run, decides what the business does about them. And increasingly, agentic AI acts on those decisions at the edge. These are not three separate topics. They are one stack, and edge to cloud is where they meet the physical world. Architectures that move data and improvise everything above it end up with the automation logic scattered across broker plugins, cron jobs, and cloud functions. That is the integration spaghetti this whole discipline set out to remove. Edge to cloud is not one problem. It is four. Architect for the pattern, not the product.
Making a Databricks-owned Delta table queryable from Snowflake without copying it is one of those things where every individual piece — Delta Lake UniForm, Unity Catalog, private connectivity — is documented somewhere on its own, but nobody writes up how they actually fit together end to end. If you've been asked to give Snowflake read access to a dataset that Databricks owns, without standing up a nightly export job, this is for you. Why the Two Platforms Don't Share Data Natively Delta Lake and Apache Iceberg are both open table formats, but they track a table's files, schema, and change history differently. Delta Lake is Databricks' native format. Snowflake has strong native support for Iceberg. A table written as Delta Lake isn't something Snowflake can open directly, and a table written as Iceberg by some other engine isn't something Databricks reads as Delta. Historically, the fix for this mismatch was a second copy: export the Databricks table, land it somewhere Snowflake could read, and run a job to keep the copy in sync. That's double the storage, a sync job that needs monitoring, and a copy that's stale the moment the source table changes. None of that gets billed as "vendor lock-in" on any invoice, but that's exactly what it is. What UniForm Changes Delta Lake UniForm, generally available since Delta Lake 3.2 (Databricks Runtime 14.3+), writes Iceberg metadata alongside the Delta transaction log on every commit. Same Parquet files on disk, one additional metadata layer. Enabling it is a table property: SQL ALTER TABLE my_table SET TBLPROPERTIES ( 'delta.universalFormat.enabledFormats' = 'iceberg' ); Once that's set, Databricks keeps writing Delta exactly as before. Snowflake reads the same table as Iceberg, through Unity Catalog's Iceberg REST Catalog endpoint. No export step, no second storage bill. The Setup, End to End Getting from "UniForm is on" to "a Snowflake analyst can query the table" is four pieces. Piece 1: Enable UniForm on the source table. Covered above. One property, per table, Runtime 14.3 or later. Piece 2: Scope a service principal at the Unity Catalog level, not the storage account. Register a service principal and grant it Unity Catalog privileges — EXTERNAL USE SCHEMA specifically, which isn't part of ALL PRIVILEGES and is easy to grant around by accident. There's no separate storage role to assign. Unity Catalog holds the storage credential itself and vends short-lived, scoped credentials to Snowflake at query time, so the service principal only ever authenticates against the catalog API, never against the storage account directly. Piece 3: Put the connection on private networking. This is the piece that involves the most moving parts, and it's worth walking through because the Snowflake side does the provisioning, not Databricks. Snowflake account needs to be on Business Critical edition or higher for this, and both accounts need to be in the same Azure region. Provision an outbound private endpoint from Snowflake toward the Databricks workspace: SQL SELECT SYSTEM$PROVISION_PRIVATELINK_ENDPOINT( '<databricks_resource_id>', 'databricks_ui_api' ); That endpoint sits in a Pending state until someone with access to the Azure resource approves it in the Azure Portal — a step that's easy to lose a day to if you don't know to go looking for it. Then register the catalog integration: SQL CREATE OR REPLACE CATALOG INTEGRATION uc_catalog_int CATALOG_SOURCE = ICEBERG_REST CATALOG_API_TYPE = PRIVATE CATALOG_URI = 'https://<databricks-workspace-url>' ... ENABLED = TRUE; The detail that trips people up: use the standard public hostname for the workspace URL, not the PrivateLink endpoint's own hostname. Snowflake routes traffic through the private endpoint automatically once it exists — pointing the catalog integration at the PrivateLink hostname directly just breaks it. A second private endpoint, and an external volume with USE_PRIVATELINK_ENDPOINT = TRUE, does the same thing for the storage account so file reads stay off the public internet too. Piece 4: Link the catalog — don't hand-create Iceberg tables. The last step is a catalog-linked database, not individual CREATE ICEBERG TABLE statements: SQL CREATE DATABASE uc_linked_db LINKED_CATALOG = ( CATALOG = 'uc_catalog_int' ) EXTERNAL_VOLUME = 'uc_external_vol'; Once this exists, whatever is in Unity Catalog — including every UniForm-generated Iceberg table — shows up in Snowflake automatically, and stays in sync on its own. New tables and schemas created later in Databricks appear without touching Snowflake again. It's a refresh on an interval under the hood, not a live push, so don't oversell the freshness to whoever's asking for "real-time." Writes Are Newer, and Direction Still Matters Everything above is Snowflake reading a Databricks-written table. Writing the other way is newer, and support isn't symmetric yet. Snowflake now supports writing to Databricks-managed Iceberg tables in Unity Catalog — GA on Azure as of earlier this year. Snowflake Horizon's federation is bidirectional by design, using vended short-lived credentials rather than long-lived IAM roles in either direction. Unity Catalog's federation is still mostly inbound: strong at letting other engines read and write into it, weaker when Databricks itself needs to reach out to a catalog it doesn't own. If a design depends on writes going both directions, check current support for the specific cloud and catalog combination before committing to it — this part is moving fast. Where It Still Falls Short Row- and column-level masking is GA for Spark and expanding to other Iceberg REST-compatible engines, but Snowflake's Scan Plan API — cross-engine, attribute-based access control for federated Iceberg data — was still in private preview as of mid-2026. Don't assume a masking policy enforces the same way on both sides until it's been tested. Iceberg's own spec keeps moving underneath this too. Iceberg v3 features like deletion vectors and row lineage are in public preview on Databricks now, which is good news for performance, but it means "Iceberg compatible" isn't a fixed target — check that both platforms agree on the table spec version before standardizing on it. And disaster recovery for federated catalog setups is still gated or preview-stage on the Unity Catalog side as of this writing. Don't lean on this as part of a DR plan without confirming that directly. One Thing I'd Tell Anyone Doing This Check the edition and region requirements before you design anything, not after. Business Critical or higher on the Snowflake side, same Azure region on both sides — if either of those isn't true for your accounts, the private connectivity piece doesn't work at all, and there's no workaround short of moving one of the accounts. Confirm that first, then start building.
Freelance Backend Developer,
Freelancer
Founder,
DataView
Vice President - Banking and Finance / Cloud /Big Data Analytics / AI & ML,
JPMorgan Chase & Co.
Data Analytics - Assistant Vice President,
U.S. Bank