Big Data

I work as a data analyst at a legal services company. Part of my work involves protecting sensitive data during the Test Data Management (TDM) process. Many other departments in the company need test data to develop an application. Copying the production data for test sounds like a good plan. But because the test environment usually has lower cybersecurity requirements, this will cause customer privacy data leaks. So, my job is to mask the sensitive data to protect customer privacy. When it comes to my job, the first thing that comes to many people’s minds is that my work involves masking sensitive data. For example, changing the email address from [email protected] to [email protected]. Masking data is indeed important, but before we jump to the masking step, there's one basic question: Which column contains sensitive data, and how can I find it? In this article, I will introduce a pipeline designed to identify sensitive data columns before masking steps. Structure of the Pipeline Please find the identifying sensitive data pipeline structure workflow chart below: Identifying Sensitive Data Pipeline Structure Workflow Before I start introducing each stage, I’d like to mention two points. The first point: The original intent behind this pipeline structure design was to save time spent locating sensitive data. Server usage is billed based on duration. In a perfect world, the system would balance efficiency and accuracy. However, in practice, efficiency takes precedence in order to cut costs. The second point: The pipeline also had to preserve data usability for testing. In some cases, data privacy controls must be designed in a way that does not break core application workflows. For key columns such as primary keys and foreign keys, they need to preserve join functions and application workflows. So in practice, we usually leave them unchanged. Apply Column Name and Pattern Matching First, and quite intuitively, many columns' names are really straightforward and can be easily identified. For example, full name, phone number, and email. After the very first easy screening, some columns can be identified by hardcoded Python scripts, based on the specific column name pattern. However, there is an issue at this stage. I can identify columns containing sensitive data using customer email. But if there is another column named customer email address that hasn't been included in the hardcoded script, I won't be able to detect it. Besides that, relying solely on column names isn't always reliable. Take the notes column, a free-text field, for instance. It often appears as an optional field after the main information has been entered. Most people will leave it blank or write some insignificant things. But sometimes customers do write something, such as Our CEO Everett would like you to prioritize processing the ABC document. Please email them to [email protected] as soon as you finish, and then call 123-456-7890 to notify him. If I don't mark this column as need masking, the customers' private information will be exposed. Check Historical Decisions After the initial filtering step, I will check the historical decisions database for specific columns, such as the notes column mentioned earlier. If the database indicates that the historical decision for column notes is to mask it, then that column will be masked during the current round. Even if the notes in this specific round contain no sensitive data. For example, no privacy-related information is mentioned. There is no guarantee that the data in the next refreshed cycle will remain free of sensitive information. Send Ambiguous Columns to AI for Review and Analyze Sample Values Here comes the highlight of the entire pipeline. Sometimes, column names are somewhat ambiguous. Or it's unclear whether certain rows contain sensitive data. Let's take the notes column mentioned earlier again. It might be empty. Or it could contain a message like When food is delivered, please ring the doorbell and call my wife Bobi, thereby the sensitive information gets leaked. I started using the spaCy library from Python for Natural Language Processing (I will refer to this term as NLP later in this article). While spaCy isn’t a Large Language Model (I will call this term LLM), it certainly performs NLP analysis. However, the sampling process was time-consuming. I would sample the entire dataset if it had fewer than 50,000 rows, but randomly select 50,000 rows if it exceeded that limit. In a later version of the workflow, I switched to OpenAI: this time, I just need to select a sample of 100 rows and send them via API to the AI/LLM for analysis. The AI then generates a masking recommendations database, which will undergo manual review later. Accuracy improved significantly after we began using LLMs. It rose from 80% with spaCy to approximately 93% after switching to OpenAI. This 93% figure was determined by having human analysts conduct a column-by-column analysis in parallel with my development of the pipeline and automation scripts. So the result is benchmarked against manual reviews. Furthermore, this figure represents an average obtained after two rounds of actual TDM data masking operations and several additional rounds of testing. Regarding the remaining 7% of errors, false positives accounted for about 90%, and false negatives for only 10%. This is important because missing sensitive data is much more serious than over-flagging a column for review. Compared to manually analyzing a medium-sized schema containing 100 tables for 64 hours. An automated script can complete the analysis in just 2 hours. However, please note that this 2-hour timeframe does not include the time required for subsequent manual review. Human Review and Store Recommendations and New Decisions After the AI/LLM finishes analysis, human analysts will review the mask recommendations database generated by the AI. Each row in the database generates a report containing the user ID, database name, table name, column name, masking suggestion, masking rule, and analysis date. Then, humans will review the mask suggestions and corresponding masking methods. For example, the AI-generated mask suggestion database is: AI-generated Mask Suggestion Database Example As a human analyst, at this stage, I can review the masking suggestion generated by the AI. I would agree with the suggestion to mask the data. However, regarding the masking rule, I would review it and change it to set it to a blank value. Manual review needs to randomly sample 500 rows and analyze them individually to reach a final mask decision. In this new process, human analysts only need to review a single row of AI-generated mask decisions and mask rules. The switch saves time significantly. During a new round of the TDM data masking process, some new columns will be identified by AI and flagged as requiring masking. The new masking decision will be added to the existing historical decisions database after manual review. Send to Data Governance and Send to Business Customer and Get Feedback After our TDM team identifies and masks the sensitive data columns, we submit our results to the Data Governance department for a secondary manual review. Their review process differs slightly from ours. Our team focuses on using business knowledge to determine whether a column contains sensitive data. And we’re also responsible for developing more efficient identification & masking procedures. However, the Data Governance department needs to review and provide more accurate masking decisions. Because their team members have better knowledge of how to decide whether a column should be masked and of the appropriate masking method. After our two departments conducted two rounds of manual review, we sent the masked data results to our business customers' departments. They will use this data for testing and provide us with feedback based on their specific needs. For example, we recommended masking customer_id with a generated synthetic number. But doing so will change primary and foreign keys, thereby breaking database linkages. So, our business customer departments advised us against masking those columns. Conclusion and Future Improvement Plans Successfully masking sensitive data begins with accurately identifying the columns containing such data. Many people skip this and jump straight to the more interesting masking process. In my view, however, getting this step wrong will fail the rest of the workflow as well. The pipeline I designed isn't perfect. And I have a few ideas for improving the "Apply column name and pattern matching" component in the future. Since we’ve already used OpenAI, why not let the AI detect new patterns when analyzing ambiguous columns? We could have the AI generate a dynamic pattern database that updates automatically with every refresh cycle. It would also help us continuously update and refine our historical decisions database.

Supply chains are graphs. Suppliers feed into warehouses, warehouses feed into distribution centers, and distribution centers feed into retailers. When we model them that way — as nodes and relationships rather than rows and columns — we unlock a set of tools that gives us the ability to ask questions about connectivity, paths, and the structural importance of individual nodes. In this article, we'll build a supply chain, load it into Neo4j via Apache Spark, use NetworkX to identify the most critical nodes in the network, and then simulate a real-world disruption to find alternative routes. The full source code is available on GitHub. The Stack Each tool in the stack does what it does best: ToolRoleApache Spark (local mode)Data generation, transformation, and loading into Neo4jNeo4j (remote, AuraDB)Graph storage and native variable-length path queriesNetworkXBetweenness centrality - identifying the most critical nodesPlotlyInteractive visualization throughout One tool conspicuously absent from this list is Neo4j's Graph Data Science (GDS) library. We'll come back to why and what to reach for when you outgrow the approach described in this article. Setting Up Neo4j AuraDB AuraDB is Neo4j's fully managed cloud database. A free tier is available with no credit card required. Sign up at console.neo4j.io/graphacademy.Create a new AuraDB Free instance.When the instance is created, download or note the credentials - the connection URI, username, and password. Neo4j only shows the password once, so save it somewhere safe.Once the instance is running, open the built-in Query tab and verify connectivity: cypher MATCH (n) RETURN count(n) . This should return 0. We are ready to load data. Before starting Jupyter, export the connection details as environment variables in your shell: Shell export NEO4J_URI=neo4j+s://xxxx.databases.neo4j.io export NEO4J_USERNAME=your_username_here export NEO4J_PASSWORD=your_password_here export NEO4J_DATABASE=your_database_name_here The notebook reads these at startup and raises an error immediately if any are missing. The Data Model The supply chain has four layers connected by SHIPS_TO relationships: Plain Text Suppliers -> Warehouses -> Distribution Centers -> Retailers Each SHIPS_TO relationship carries three properties: cost (shipping cost in dollars)distance (km)capacity (maximum units per shipment) We'll generate a synthetic but reproducible dataset using Faker and NumPy with a fixed random seed, giving us 20 suppliers, 12 warehouses, 10 distribution centers, and 30 retailers with 125 routes across all three layers. Loading the Graph With Spark Spark earns its place in the pipeline by handling the loading step. The Neo4j Spark Connector translates Spark DataFrames into Cypher MERGE statements under the hood, handling the graph write for us: Python spark = ( SparkSession.builder .master("local[*]") .appName("SupplyChainResilience") .config("spark.jars.packages", SPARK_CONNECTOR) .config("neo4j.url", NEO4J_URI) .config("neo4j.authentication.basic.username", NEO4J_USERNAME) .config("neo4j.authentication.basic.password", NEO4J_PASSWORD) .getOrCreate() ) The connector JAR resolves automatically from Maven Central on first run. In a real pipeline, this step would read from S3, a data warehouse, or a Kafka topic and stream records into Neo4j continuously. One important detail is that we'll clear the database before each load using Cypher's IN TRANSACTIONS syntax so each run starts from a clean slate: Cypher MATCH (n) CALL (n) { DETACH DELETE n } IN TRANSACTIONS OF 10000 ROWS We'll then confirm the database is empty before writing new data to the database. Betweenness Centrality With NetworkX Betweenness centrality answers a specific question: if we looked at every possible shortest path between every pair of nodes in the network, how often does each node appear on one of those paths? A node with high betweenness acts as a bridge through which many shortest paths pass. If it disappears, many routes break. A node with low betweenness is peripheral - the network barely notices if it goes offline. We'll pull the graph out of Neo4j via Spark into a NetworkX DiGraph and compute centrality using shipping cost as the edge weight, so the algorithm finds shortest paths by lowest cost rather than fewest hops: Python edges_sdf = ( spark.read.format("org.neo4j.spark.DataSource") .option("query", "MATCH (a)-[r:SHIPS_TO]->(b) " "RETURN coalesce(a.id, a.name) AS source, " " coalesce(b.id, b.name) AS target, " " r.cost AS cost") .load() ) edges_pd = edges_sdf.toPandas() G = nx.DiGraph() for _, row in edges_pd.iterrows(): G.add_edge(row["source"], row["target"], weight = row["cost"]) centrality = nx.betweenness_centrality(G, weight = "cost", normalized = True) Figure 1 shows the full supply chain network before any disruption. Each node type is color-coded: suppliers in blue, warehouses in orange, distribution centers in teal, and retailers in red-orange. The density of connections between layers gives a first impression of where bottlenecks might exist. Figure 1. Full Supply Chain Network Once computed, we'll write the scores back into Neo4j via Spark so Cypher queries can use centrality as a filter or sort key without recomputing it every time. Figure 2 shows the top 15 nodes ranked by betweenness centrality. The length of each bar reflects how often that node appears on a shortest path between other nodes in the network. A longer bar indicates a node that carries a disproportionate share of shortest-path traffic. Figure 2. Top 15 Nodes by Betweenness Centrality Why Not GDS? Neo4j's Graph Data Science (GDS) library has a native gds.betweenness.stream() procedure that runs the same algorithm inside the database using advanced processing. For our small-node demo dataset, NetworkX is instant and requires no additional setup. But nx.betweenness_centrality() runs in O(n * m) time and loads the entire graph into memory. At tens of thousands of nodes, both of those properties become problems. That is exactly where GDS comes in. If you are using Neo4j AuraDB, the same algorithm is available through Aura Graph Analytics — a service that connects directly to your AuraDB instance. The rest of the notebook — Spark for data loading, Plotly for visualization, native Cypher for shortest path — works identically on AuraDB without any changes. Simulating a Disruption With centrality scores computed, we'll identify the highest-scoring node that is a Supplier or Warehouse and mark it as disrupted in Neo4j: Python with driver.session(database = NEO4J_DATABASE) as session: session.run( "MATCH (n {id: $id}) SET n.disrupted = true", id=disrupted_id ) We'll deliberately restrict disruption to Suppliers and Warehouses. Distribution centers are fewer in number, and each carries more routing burden, making them more likely to be sole bridges whose removal severs the network entirely. A warehouse disruption is a more realistic scenario and produces richer alternative-route results. Finding Alternative Routes With Native Cypher With the disrupted node flagged, we'll use Neo4j's built-in variable-length path matching to find alternative routes that avoid it: Cypher MATCH (s:Supplier), (r:Retailer) WHERE s.disrupted IS NULL AND r.disrupted IS NULL MATCH path = (s)-[:SHIPS_TO*..6]->(r) WHERE none(n IN nodes(path) WHERE n.disrupted IS NOT NULL AND n.disrupted = true) WITH s, r, path, reduce( cost = 0.0, rel IN relationships(path) | cost + rel.cost ) AS total_cost ORDER BY total_cost ASC RETURN s.id AS source, r.id AS target, [n IN nodes(path) | coalesce(n.id, n.name)] AS path_nodes, round(total_cost, 2) AS total_cost, length(path) AS hops LIMIT 10 This query is available on both local Neo4j and AuraDB with no additional plugins required. A typical result looks like this: Plain Text source target total_cost hops S006 R001 94.35 3 S007 R026 148.86 3 S007 R017 155.19 3 S019 R026 165.88 3 The cheapest alternative route bypasses the disrupted node entirely at a total shipping cost of $94.35. Note that MATCH (s:Supplier), (r:Retailer) creates a cartesian product for every Supplier/Retailer pair, which is fine for our small dataset. For larger graphs, you would normally constrain the source and destination. The network after disruption is shown in Figure 3. The disrupted node is highlighted in red, and the best alternative route is shown in green, tracing the lowest-cost path from supplier to retailer that avoids the failed node entirely. Figure 3. Best Alternative Route After Disruption Figure 4 compares the top alternative routes by total shipping cost and number of hops. A route with more hops may still be cheaper - the cost comparison makes that trade-off explicit and gives logistics planners a clear basis for decision-making. Figure 4. Alternative Route Cost and Hop Comparison Gotchas and Lessons Learned This project required some debugging. Here are the issues worth knowing about before you try this yourself. Java Version Compatibility PySpark 3.5.x officially supports several versions of Java. However, Java 23 removed javax.security.auth.Subject.getSubject(), which Spark's Hadoop dependency calls during startup. On Java 23 or later, this produces a cryptic UnsupportedOperationException: getSubject is not supported error and Spark never starts. The solution is to install Java 21 LTS alongside any existing Java installation and point PySpark at it before starting Jupyter. Here, for example, using Homebrew on Apple hardware: Shell brew install openjdk@21 export JAVA_HOME=/opt/homebrew/opt/openjdk@21 export PATH="/opt/homebrew/opt/openjdk@21/bin:$PATH" Any existing Java installation is unaffected outside that shell session. The Neo4j Spark Connector 6.x support for Spark 4.x is in active development, so upgrading PySpark to avoid the Java issue is a future option. Relationship Write Deadlocks When writing relationships via the Neo4j Spark Connector with multiple Spark partitions, concurrent writes can deadlock inside Neo4j as transactions compete for the same node locks. The error looks like this: Plain Text ForsetiClient can't acquire EXCLUSIVE NODE_RELATIONSHIP_GROUP_DELETE because it would form a deadlock wait cycle The solution is to call .coalesce(1) on the DataFrame before writing relationships, which forces Spark to write them sequentially from a single partition: Python sdf.coalesce(1).write.format("org.neo4j.spark.DataSource") ... Node writes do not need this because they do not acquire the same lock types. Stale Data Between Runs In the Jupyter notebook's write configuration, the Spark Connector's Overwrite mode merges on node keys but does not remove relationships that existed in a previous run but are absent from the current one. If the dataset size changes between runs, old relationships accumulate alongside new ones, interfering with the graph structure. The solution is to clear the database at the start of every load run rather than relying on Overwrite to clean up after itself. Always confirm the clear succeeded with a node count check before writing. The none() Predicate and Missing Properties This was the subtlest issue of the project. Our disruption query used: Cypher WHERE none(n IN nodes(path) WHERE n.disrupted = true) This returned zero results even when paths clearly existed, and the disrupted node was correctly flagged. In Neo4j, when a node doesn't have a disrupted property at all, n.disrupted = true evaluates to null rather than false. The none() predicate then treats every node as potentially disrupted and filters out all paths. This is exactly how Cypher's three-valued logic works. The solution is an explicit IS NOT NULL check: Cypher WHERE none(n IN nodes(path) WHERE n.disrupted IS NOT NULL AND n.disrupted = true) shortestPath() and Alternative Routes Initially, Neo4j's shortestPath() function was used to find alternative routes. It returned zero results. The reason is that shortestPath() finds the path with fewest hops first, then applies the WHERE none(...) filter. It computes a single shortest path rather than exploring alternative candidates, and filtering on disrupted nodes can eliminate that path without considering longer valid alternatives. The solution is to use a plain variable-length path match with an explicit hop limit instead. This lets the WHERE clause filter while still returning valid results: Cypher MATCH path = (s)-[:SHIPS_TO*..6]->(r) WHERE none(n IN nodes(path) WHERE ...) Guaranteed Connectivity in Generated Data With purely random route generation, it's possible for a single node to end up as the only connection between two layers - a so-called sole bridge. Disrupting that node severs the network completely and leaves no alternative routes to find. The solution is to generate routes with a guaranteed minimum connectivity. So, every source node gets at least two outbound routes, and every target node gets at least two inbound routes before random fill: Python def make_routes(sources, targets, n_routes, min_out=2, min_in=2): # Guarantee every source has at least min_out outbound routes for s in src_ids: sample = rng.choice(tgt_ids, size = min(min_out, len(tgt_ids)), replace = False) for t in sample: if (s, t) not in seen: rows.append(make_row(s, t)) # Guarantee every target has at least min_in inbound routes for t in tgt_ids: sample = rng.choice(src_ids, size = min(min_in, len(src_ids)), replace = False) for s in sample: if (s, t) not in seen: rows.append(make_row(s, t)) # Fill remaining routes randomly ... Cypher 25 Syntax If you are running Neo4j 2025.06 or later, the CALL { WITH n ... } subquery syntax used in batch deletes is deprecated. Use the new variable scope syntax instead: Cypher MATCH (n) CALL (n) { DETACH DELETE n } IN TRANSACTIONS OF 10000 ROWS Summary We've built a supply chain resilience analysis pipeline that models a supply chain as a graph, identifies its most critical nodes using betweenness centrality, simulates a real-world disruption, and finds alternative routes using native Cypher. Each tool did what it does best: Spark handled bulk data loading, Neo4j stored the graph and answered path queries, NetworkX computed the graph algorithm, and Plotly produced interactive visualizations at every stage. The gotchas section above contains several useful engineering lessons, which should save you time and effort on your projects. The full source code is available on GitHub.

By Akmal Chaudhri CORE

The Problem: Our p99 Was 3-5 Seconds Our PyFlink pipeline was missing its latency SLO by seconds. The pipeline itself was straightforward: consume events from Kafka, transform them, serialize them as Protobuf, and write the results to downstream systems. Yet under production load, p99 end-to-end latency was consistently in the 3-5 second range. Profiling pointed us to an unexpected bottleneck: we were deserializing Protobuf messages in Python, even though the Flink runtime processing our stream was JVM-based. Every record that entered the Python path had to cross the JVM-to-Python process boundary, get parsed by a Python UDF, and then cross back. The business logic wasn't the problem. The doorway was. We moved Protobuf deserialization to Flink's JVM-side Protobuf format and kept Python for orchestration and SQL. In our environment, p99 dropped to approximately 500 milliseconds, with less code and a pipeline that is easier to reason about. Verified on AWS Managed Service for Apache Flink (formerly Kinesis Data Analytics). Why Python-Side Deserialization Is So Expensive The naive PyFlink architecture looks like this: A Kafka source table declared with a generic format (raw, json, or a SimpleStringSchema), so every record arrives as opaque bytes or a string.A Python map() or UDF that imports generated _pb2.py classes and calls ParseFromString() on every message.Downstream transforms and sinks. Two costs hide in step 2, and they compound at high throughput. The process boundary. PyFlink is not Python running inside Flink; it is a JVM runtime coordinating with a separate Python execution environment. Every record that enters the Python execution path incurs overhead associated with moving data between the JVM and Python, and depending on the operator and execution mode, that can involve serialization and inter-process communication in both directions. For a per-record deserialization UDF on a latency-sensitive pipeline, that overhead is paid before the actual business transformation begins. Per-record parse cost. Even when Python's Protobuf implementation uses its native backend, parsing in a Python UDF still requires the record to enter the Python execution path. When the workload is latency-sensitive and high-throughput, the combination of serialization, inter-process communication, Python execution, and parsing overhead can become significant. In our case, profiling showed that this path was a major contributor to our latency. In our pipeline, these two costs together accounted for the bulk of the gap between a 3–5 second p99 and the ~500ms target we needed, before the enrichment logic even began executing. The Key Realization: PyFlink Already Runs on the JVM Here's the insight that changes the architecture: if Protobuf is declared at the table DDL level, Flink's Kafka connector deserializes it with its native, optimized JVM-based Protobuf format before any data reaches the Python side. Your columns simply arrive typed and ready. Python's role shrinks to what it's genuinely good at in this stack: orchestration and SQL. No rewrite to Java. No change to how jobs are deployed. Just a different declaration of intent. The trade is that Flink's native Protobuf format needs compiled Java message classes on the classpath; it does not consume .proto files or Python _pb2 modules directly. That means adding a small build step to your workflow, which we'll cover below. Implementation The pipeline splits into two declarative jobs. Job 1: JSON In, Protobuf Out The source table reads the raw JSON topic; the sink table declares format = 'protobuf' and points at the compiled Java class. The JVM handles typed-row-to-Protobuf encoding. SQL -- SOURCE: raw JSON payload as STRING plus Kafka record timestamp CREATE TABLE source_events_json ( event_data STRING, kafka_timestamp TIMESTAMP(3) METADATA FROM 'timestamp' ) WITH ( 'connector' = 'kafka', 'topic' = '${INPUT_JSON_TOPIC}', 'properties.bootstrap.servers' = '${KAFKA_BOOTSTRAP_SERVERS}', 'scan.startup.mode' = 'latest-offset', 'format' = 'raw' ); -- SINK: Protobuf out to Kafka (JVM handles typed row to Protobuf) CREATE TABLE sink_events_pb ( id STRING, organization_id STRING, event_ts ROW<`seconds` BIGINT, `nanos` INT>, is_active BOOLEAN, event_type STRING ) WITH ( 'connector' = 'kafka', 'topic' = 'acme.events.pb.v1', 'properties.bootstrap.servers' = 'kafka:9092', 'format' = 'protobuf', 'protobuf.message-class-name' = 'com.acme.events.v1.EventOuterClass$EnrichedEvent' ); -- TRANSFORM: pure SQL, no Python UDFs INSERT INTO sink_events_pb SELECT JSON_VALUE(event_data, '$.id') AS id, JSON_VALUE(event_data, '$.organization_id') AS organization_id, ROW( UNIX_TIMESTAMP(NULLIF(JSON_VALUE(event_data, '$.after.event_ts'), '')), CAST(EXTRACT(NANOSECOND FROM CAST(NULLIF(JSON_VALUE(event_data, '$.after.event_ts'), '') AS TIMESTAMP_LTZ(9))) AS INT) ) AS event_ts, CAST(JSON_VALUE(event_data, '$.is_active') AS BOOLEAN) AS is_active, JSON_VALUE(event_data, '$.event_type') AS event_type FROM source_events_json; Note what's absent: no ParseFromString(), no _pb2.py imports, no Python deserialization loop. The Python program registers DDL and runs SQL. Job 2: Protobuf In, OpenSearch Out Downstream, the sanitized Protobuf topic becomes a typed source, using the same protobuf.message-class-name property, plus ignore-parse-errors so a malformed record can't poison the pipeline. SQL -- SOURCE: Protobuf from the sanitized Kafka topic CREATE TABLE kafka_source_pb ( id STRING, organization_id STRING, event_ts ROW<`seconds` BIGINT, `nanos` INT>, is_active BOOLEAN, event_type STRING, kafka_timestamp TIMESTAMP(3) METADATA FROM 'timestamp' ) WITH ( 'connector' = 'kafka', 'topic' = 'acme.events.pb.v1', 'properties.bootstrap.servers' = 'kafka:9092', 'scan.startup.mode' = 'latest-offset', 'format' = 'protobuf', 'protobuf.message-class-name' = 'com.acme.events.v1.EventOuterClass$EnrichedEvent', 'protobuf.ignore-parse-errors' = 'true' ); -- SINK: OpenSearch (JSON) CREATE TABLE opensearch_sink ( id STRING, organization_id STRING, event_ts TIMESTAMP_LTZ(3), is_active BOOLEAN, event_type STRING, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'opensearch-2', 'hosts' = '${OPENSEARCH_ENDPOINT}:443', 'index' = 'acme-events-v1', 'format' = 'json' ); INSERT INTO opensearch_sink SELECT id, organization_id, TO_TIMESTAMP_LTZ(event_ts.seconds * 1000, 3), is_active, event_type FROM kafka_source_pb; The Build Step: Getting Java Classes Onto Flink's Classpath The one genuinely new piece of workflow is compiling your .proto definitions to Java and packaging them into the job's fat JAR. The essential Maven pieces: XML <dependencies> <dependency> <groupId>com.google.protobuf</groupId> <artifactId>protobuf-java</artifactId> <version>3.25.5</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-protobuf</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> <version>${flink.connector.kafka.version}</version> </dependency> <!-- plus your sink connectors, e.g. flink-connector-opensearch2 --> </dependencies> Two practices that made this maintainable for us: Version-control the generated Java sources (or generate them in CI from a single canonical .proto repo) and pull them in with build-helper-maven-plugin's add-source, rather than compiling .proto files in every consuming project. One schema source of truth, many consumers.Shade everything into one JAR with maven-shade-plugin, excluding signature files (META-INF/*.SF, *.DSA, *.RSA). On AWS Managed Flink, pass it via the job's JAR configuration; on self-managed Flink, drop it in lib/ or use --classpath. The full workflow: define the .proto, compile it to Java with protoc, package the fat JAR, put it on Flink's classpath, author the PyFlink job with the DDL above, then deploy and watch end-to-end p99. How We Measured the Improvement We measured end-to-end p99 latency as the time from a record landing on the source Kafka topic to the corresponding OpenSearch write being acknowledged MetricBeforeAfterp99 latency3-5s~500msSustained throughput~5,000 events/sec~5,000 events/secFlink parallelism128Python UDF parsingYesNoJVM/Python boundary on hot pathYesNoProtobuf decodingPythonJVM Results End-to-end p99 latency around 500 milliseconds in our environment at production load, down from a 3-5 second baseline, by eliminating per-record JVM-to-Python crossings and Python-side parsing on the hot pathLess code. The deserialization UDFs, the _pb2 imports, and their error handling all disappeared. What remains is DDL plus SQLSimpler and easier to operate. The pipeline now relies on Flink's Kafka connector and Protobuf format for serialization and parsing, with built-in parse-error handling, instead of hand-rolled Python parsing When This Optimization Won't Help Moving Protobuf decoding to the JVM won't automatically solve every latency problem. If your pipeline's critical path is dominated by sink backpressure, network latency, external API calls, state access, or checkpointing overhead rather than deserialization, changing the serialization path may have little effect on end-to-end latency. This optimization is most valuable when profiling specifically shows that Python execution and JVM/Python data movement are significant contributors to the critical path, which is why we'd recommend profiling first rather than applying this as a default change. When You Should Still Use Python UDFs This pattern is not "never write Python UDFs." It's "keep them off the per-record deserialization path." Python remains the right tool when: The transformation genuinely needs Python libraries (ML feature computation, model inference, specialized parsing that has no SQL equivalent).Throughput is modest and developer velocity matters more than the last hundred milliseconds.You're prototyping. Even then, declare the format natively from day one anyway; it costs nothing and you won't have to migrate later. If a UDF is unavoidable on a hot path, at least let the JVM do the deserialization first so the UDF receives typed columns rather than raw bytes. Gotchas Worth Knowing Before You Ship Property syntax varies by Flink version. Some versions use format = 'protobuf'; newer key/value descriptors prefer value.format = 'protobuf'. Check your version's docs.Enums: surface them as STRING if you need ergonomic SQL manipulation, or keep them numeric with a lookup table.Schema evolution: favor backward-compatible, additive changes with defaults. Because the compiled Java classes are baked into the JAR, a schema change means a rebuild and redeploy, so make that a deliberate, versioned step in CI rather than an afterthought. ignore-parse-errors is your safety net during rollout windows, but monitor the drop counter so it doesn't silently eat data.Benchmark end-to-end, not just the UDF: source lag, operator latency, and sink acknowledgments under production load patterns. Deserialization wins can be masked, or dwarfed, by sink backpressure.Security: lock down OpenSearch credentials and TLS; pin Kafka client versions compatible with your Flink release. Closing Thoughts We didn't rewrite the pipeline in Java. We removed an unnecessary per-record JVM-to-Python boundary from the hot path and let Flink's JVM-native Protobuf format do the work it was designed to do. If your PyFlink job parses Protobuf messages in Python today, check whether Flink's native format support can move that work into the JVM-side execution path. For latency-sensitive pipelines, eliminating unnecessary Python boundaries may be one of the highest-leverage optimizations to investigate, especially when profiling shows that serialization and Python execution are on the critical path.

A few days ago, I set out to build a simple image classification model using convolutional neural networks (CNNs). The task itself wasn’t particularly complex, but choosing the right framework proved more challenging than expected. I found myself choosing between TensorFlow and PyTorch, two powerful frameworks for building high-performance CNNs. To explore this, I implemented the same CNN in both frameworks under identical conditions and compared them across key aspects like learning curve, flexibility, debugging, and performance. A Quick Look at the Frameworks Before deep-diving into the comparison, it’s worth briefly understanding the two frameworks used throughout this experiment. 1. TensorFlow TensorFlow is an open-source deep learning framework developed by Google. It is widely known for its strong ecosystem and production-ready capabilities. One of its key strengths is its integration with high-level APIs such as Keras, which simplifies model building and training. TensorFlow is commonly used in large-scale applications, offering tools for deployment across web, mobile, and edge devices. Overall, it is often preferred when moving models from experimentation to production environments. 2. PyTorch PyTorch is an open-source deep learning framework developed by Meta Platforms. It has gained significant popularity, especially in the research community, due to its simplicity and flexibility. PyTorch uses a dynamic computation graph, which makes it feel more like standard Python code. This makes model development more intuitive and debugging significantly easier. It is often the preferred choice for experimentation, rapid prototyping, and research-driven projects. Experiment Setup To ensure a fair and meaningful comparison between TensorFlow and PyTorch, both implementations were designed under identical conditions. 1. Dataset The models were trained and evaluated on the CIFAR-10 dataset, a widely used benchmark for image classification tasks.It consists of 60,000 color images across 10 classes, making it suitable for evaluating CNN performance.CIFAR-10 is publicly available for research purposes and is commonly distributed under a permissive academic license, allowing free use for educational and non-commercial applications. 2. Model Architecture A simple yet effective Convolutional Neural Network (CNN) architecture was used in both frameworks. The structure includes: Convolutional layers for feature extractionReLU activation functionsMax-pooling layers for dimensionality reductionFully connected layers for classification Care was taken to ensure that the architecture remained identical in both implementations. 3. Training Configuration To maintain consistency, the following hyperparameters were used across both frameworks: Optimizer: AdamLearning rate: 0.001Batch size: 64Number of epochs: 10Loss function: Cross-Entropy Loss 4. Environment All experiments were conducted using Google Colab. Both TensorFlow and PyTorch implementations were executed in the same runtime environment. The configuration used includes: Runtime Type: GPU-enabled environmentPython Version: 3.xDeep Learning Libraries: TensorFlow and PyTorch (latest stable versions) The experiments were run on the same Colab runtime session to maintain consistency in resource allocation. Implementation To ensure a fair comparison, the same CNN architecture and training configuration were implemented using both TensorFlow and PyTorch. While the underlying model remains identical, the implementation approach differs significantly across the two frameworks. 1. CNN Implementation in TensorFlow The model was first implemented using TensorFlow with its high-level Keras API, which provides a concise and structured way to define deep learning models. Model Definition Python model = models.Sequential([ layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)), layers.MaxPooling2D((2,2)), layers.Conv2D(64, (3,3), activation='relu'), layers.MaxPooling2D((2,2)), layers.Flatten(), layers.Dense(64, activation='relu'), layers.Dense(10, activation='softmax') ]) The Sequential API allows layers to be stacked in a linear fashion, making the architecture easy to read and implement. This significantly reduces boilerplate code and is especially helpful for beginners. Model Compilation and Training Python model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = model.fit(x_train, y_train, epochs=10, batch_size=64, validation_data=(x_test, y_test)) Training in TensorFlow is handled using a single high-level function. It automatically manages the training loop, backpropagation, and metric tracking, making the process highly streamlined. Observation: TensorFlow offers a compact and beginner-friendly implementation. With minimal code, it handles most of the underlying complexity, making it ideal for rapid development and production-oriented workflows. 2. CNN Implementation in PyTorch The same CNN architecture was implemented using PyTorch, which follows a more explicit and flexible approach. Model Definition Python class CNN(nn.Module): def __init__(self): super().__init__() self.conv1 = nn.Conv2d(3, 32, 3) self.pool = nn.MaxPool2d(2,2) self.conv2 = nn.Conv2d(32, 64, 3) self.fc1 = nn.Linear(64*6*6, 64) self.fc2 = nn.Linear(64, 10) In PyTorch, models are defined using Python classes. This provides greater flexibility but requires a more detailed understanding of how each component works. Forward Pass Python def forward(self, x): x = self.pool(torch.relu(self.conv1(x))) x = self.pool(torch.relu(self.conv2(x))) x = x.view(-1, 64*6*6) x = torch.relu(self.fc1(x)) x = self.fc2(x) return x The forward pass must be explicitly defined, giving full control over how data flows through the network. This makes it easier to customize and debug complex models. Training Loop Python for inputs, labels in trainloader: optimizer.zero_grad() outputs = model(inputs) loss = criterion(outputs, labels) loss.backward() optimizer.step() Unlike TensorFlow, PyTorch requires a manual training loop. While this increases the amount of code, it also provides complete transparency and control over the training process. Observation: PyTorch offers a more flexible and transparent approach. Although it requires more code, it allows finer control over model behavior, making it a preferred choice for experimentation and research. With both implementations in place, the next step is to evaluate their performance and analyze how they compare across different metrics. Results and Analysis With both implementations completed under identical conditions, we now compare TensorFlow and PyTorch using empirical results and practical observations. 1. Accuracy The image illustrates the Accuracy and Training Time for TensorFlow and PyTorch. (Image by Author) Both frameworks achieved nearly identical performance on the CIFAR-10 dataset: TensorFlow Accuracy: 68.78% PyTorch Accuracy: 68.95% The difference (0.17%) is extremely small and falls within normal training variation. When architecture, data, and hyperparameters are controlled, the choice of framework has virtually no impact on model accuracy. Additionally, both models show: Consistent improvement across epochsNo signs of severe overfittingStable generalization on test data The image illustrates the Train and Test accuracy for TensorFlow and PyTorch. (Image by Author) 2. Loss Convergence The image illustrates the Loss Convergence for TensorFlow and PyTorch in Logarithmic Scale. (Image by Author) TensorFlow exhibits a smooth and gradually decreasing loss, both for training and validation.PyTorch shows a similar downward trend, but with slightly larger values. The higher loss values in PyTorch are due to loss accumulation across batches, whereas TensorFlow reports average loss per epoch. Despite differences in scale, both frameworks demonstrate stable and consistent convergence behavior, indicating effective training. 3. Model Training Performance Training Speed TensorFlow: 715.23 secondsPyTorch: 723.31 seconds TensorFlow is slightly faster (~1% difference), but the gap is minimal For moderate-sized datasets like CIFAR-10, training speed differences are negligible and unlikely to influence framework selection, but TensorFlow provides strong tooling for large-scale deployment, while PyTorch is equally capable in training large models. 4. Scalability and Flexibility TensorFlow follows a more structured and predefined approach, but provides robust tools such as distributed training and deployment pipelines. It also holds an advantage in large-scale production environments, while PyTorch continues to close the gap. PyTorch uses a dynamic computation graph, allowing runtime modifications, which makes custom modifications easy. It is better suited for research and experimentation, where flexibility is critical. 5. Learning Curve From an implementation standpoint: TensorFlow (via Keras) allows model creation with minimal and structured code; hence, it is easier to start with.PyTorch requires explicit definitions for model architecture, forward passes, and training loops; this results in lengthier code and greater initial effort. Ultimately, the choice between TensorFlow and PyTorch is less about performance and more about how you prefer to design, experiment with, and deploy deep learning models. Choosing Between TensorFlow and PyTorch TensorFlow is better suited when working on production-ready systems, where scalability, deployment tools, and a structured workflow are important. Its high-level APIs make it easy to develop models quickly and integrate them into real-world applications, including mobile and edge environments.PyTorch is more appropriate for research and experimentation, where flexibility and control are critical. Its dynamic nature and seamless debugging experience make it ideal for testing new ideas and building custom architectures. Conclusion: Choosing the Right Framework Through this hands-on comparison of TensorFlow and PyTorch using a CNN on the CIFAR-10 dataset, one key insight becomes clear: both frameworks perform almost identically when it comes to core metrics. The experimental results showed: Nearly identical accuracy (~68–69%)Comparable training timesSimilar loss convergence patterns This highlights an important takeaway: The choice of framework has little to no impact on model performance when architecture and training conditions are kept consistent. However, the real difference lies not in performance, but in how you build, debug, and deploy models. Ultimately, the best framework is not the one that performs slightly better on benchmarks, but the one that aligns with your workflow, problem domain, and development style. Connect with me for more updates: MediumLinkedIN

A production LLM pipeline is rarely just a prompt and a response. It typically combines retrieval, prompt rendering, model inference, output shaping, validation, persistence, and downstream actions. That broader shape is why many systems look stable in a demo and then become fragile under live traffic. The model call is only one component; the operational problem is the workflow around it. Provider APIs impose rate limits, structured outputs still need application-level checks, and external calls introduce failure ambiguity that ordinary request-response code does not handle well. Where the Breakage Starts Most production failures happen between steps, not inside the prompt. A request enters an API, context is loaded, a model call is sent, the response is parsed, a downstream action is triggered, and a record is written. If the provider generated output but the network dropped before the caller saw it, the system no longer has a clean answer to whether the operation should be retried or treated as complete. Kafka’s default delivery model is at least once, and Temporal’s documentation is explicit that activities may be retried and therefore should be idempotent. That combination makes duplicate side effects the default risk unless the pipeline is designed around durable state and idempotent writes. Duration creates the second breakage pattern. Ingestion may fan out across thousands of chunks, while a risky action may need approval hours later. Temporal workflows can receive external write events through Signals, and durable timers persist across worker and service downtime, so a workflow can pause without collapsing into callback code and scheduled cleanups. Temporal also requires workflow logic to remain deterministic during replay and provides versioning methods so new executions can adopt new code while long-running executions remain on compatible paths. Those concerns are not edge cases in LLM systems; they are normal once the pipeline extends beyond a single synchronous call. Output shape is another common source of confusion. OpenAI’s Structured Outputs guide exists because unconstrained text is not a reliable contract; the feature is designed to enforce a supplied JSON Schema and avoid missing required keys or invalid enum values. But schema compliance is only the first gate. A response can be structurally valid and still be semantically wrong, stale, or unsafe to automate. Production failures happen when formatting success is mistaken for business correctness. Why Kafka solves only part of it Kafka is a strong fit at the ingestion boundary because it turns synchronous pressure into a durable stream of work. Kafka topics are partitioned, ordering is guaranteed within a partition, and each partition is consumed by exactly one consumer in a consumer group at a given time. Consumers also control offsets and can rewind to replay records. That combination is well suited to bursty LLM demand, key-based ordering, and reprocessing after a model or prompt change. Java public void submitRequest(LlmRequest request) { LlmRequestEvent event = new LlmRequestEvent( request.requestId(), request.tenantId(), request.documentId(), request.templateId() ); kafkaTemplate.send("llm.requests", request.requestId(), event); } This pattern keeps the API narrow. The service records intent by publishing an event keyed by requestId; Kafka’s default partitioning uses the key hash, so related records land on the same partition and preserve that partition’s order. Kafka’s producer is also optimized for batching, and its pull-based consumer model lets downstream services fall behind and catch up instead of being overwhelmed by broker-driven push traffic. That is useful when inference latency varies, and demand arrives in bursts. But Kafka only states that work was published and later consumed. It does not know whether retrieval already succeeded, whether a model provider timed out after actually producing output, or whether persistence ran before a crash. Offsets capture consumption position, not business completion. Kafka is excellent for transport, buffering, replay, and fan-out, but insufficient as the sole control plane for a multi-step inference process. Why Temporal Changes the Outcome Temporal addresses the state problem directly. Its model is durable execution: workflows advance through an event history stored by the Temporal service, and that history is what allows an execution to recover from a crash and continue making progress. Worker crashes, network interruptions, and infrastructure outages are handled differently from ordinary application failures because the workflow state is not reconstructed from logs after the fact; it is already part of the execution record. Java @KafkaListener(topics = "llm.requests") public void onRequest(LlmRequestEvent event, Acknowledgment ack) { InferenceWorkflow workflow = workflowClient.newWorkflowStub( InferenceWorkflow.class, WorkflowOptions.newBuilder() .setWorkflowId(event.requestId()) .setTaskQueue("llm-inference") .build() ); try { WorkflowClient.start(workflow::run, event); } catch (WorkflowExecutionAlreadyStarted ex) { log.info("Workflow already started for {}", event.requestId()); } ack.acknowledge(); } The important detail is the workflow identifier. Temporal guarantees workflow ID uniqueness within a namespace and prevents another open workflow with the same ID from starting, which turns duplicate Kafka deliveries into a safe re-entry case instead of parallel duplicate execution. The Kafka listener acknowledges the record after the workflow start is accepted, not after the entire inference path finishes. Kafka remains the transport layer; Temporal becomes the durable execution layer for the request. Inside that workflow, each external operation should sit in an activity with explicit timeout and retry policy rather than inside scattered retry loops. Temporal’s retry model is declarative; activities retry by default, and the platform documentation recommends making activities idempotent and granular because a retry re-executes the whole activity. That fits LLM systems unusually well, since retrieval, prompt rendering, model invocation, validation, and persistence rarely fail for the same reason. Java private final LlmActivities activities = Workflow.newActivityStub( LlmActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(45)) .setRetryOptions(RetryOptions.newBuilder() .setMaximumAttempts(5) .setInitialInterval(Duration.ofSeconds(2)) .build()) .build() ); public InferenceResult run(LlmRequestEvent event) { Context context = activities.loadContext(event.documentId()); Prompt prompt = activities.renderPrompt(event.templateId(), context); ModelResponse response = activities.callModel(prompt, event.requestId()); return activities.validateAndPersist(event.requestId(), response); } That separation also makes the final validation step honest. Structured output may guarantee a schema, but business rules still need to decide whether the result is usable, and persistence still needs an idempotent write keyed by requestId so a retry cannot create duplicate approvals, tickets, or rows. In practice, this is the difference between a pipeline that merely retries and a pipeline that resumes safely. Why the Handoff Works in Production The pattern that holds up is a clean handoff. Kafka should own ingress, buffering, replay, and downstream fan-out. Temporal should own the lifecycle of one logical request. When a workflow completes, it can publish a result event for indexing, notifications, analytics, or billing, and each downstream system can consume that event independently. Kafka moves facts through the platform. Temporal ensures the process that produced those facts reaches a correct terminal state. This split also improves the cases that usually appear after launch. Human approval can arrive as a Temporal Signal without keeping compute hot. A workflow can pause for days using a durable timer and continue after downtime. Workflow code can be versioned so new executions take a new path while long-running executions stay on compatible logic, which is important because replay depends on deterministic workflow behavior. The reason LLM pipelines fail in production is not that language models are impossible to operationalize. The reason is that they are often deployed as request handlers when they are actually distributed workflows with uncertain latency, replay risk, and side effects. Kafka fixes the transport problem by providing durable, replayable streams with partition ordering and load decoupling. Temporal fixes the execution problem by persisting progress, surviving worker failure, and making retries, timeouts, and human pauses part of the design instead of a patch applied after incidents. When those two responsibilities are separated cleanly, an LLM pipeline stops behaving like an experimental chain of API calls and starts behaving like a production system.

By Akhil Madineni CORE

The Failure You Have Probably Already Seen An enterprise AI agent is deployed against production data. It answers the first ten questions confidently and correctly. Then, on the eleventh question, it produces an answer that looks reasonable but is completely wrong. The team investigates. The model is fine. The prompt is fine. The tool integrations are fine. The problem is buried in the data itself. A field the agent relied on has drifted. A join it assumed existed no longer holds. A quality signal that used to be reliable has silently degraded. This is not a rare edge case. It is becoming one of the most common failure patterns in enterprise AI systems moving from prototype to production. And it points to a simple, uncomfortable truth: most enterprise data infrastructure was built for a consumer we no longer have. I have spent the past couple of years designing agentic AI systems against production data at Fortune 500 scale. What follows is the runtime governance pattern I now design around, and the failure modes it protects against. Who this article is for: This article is for data engineers, platform architects, AI engineers, and governance teams building enterprise agents that depend on production data. It focuses less on prompt design and more on the runtime data controls required to make agent answers reliable. Twenty Years of Data Built for Humans Every large enterprise data platform in production today was designed for human consumption. Analysts, business users, data scientists, and BI teams. Those consumers share a common trait: they exercise judgment. A human analyst looking at a broken dashboard notices it. A data scientist opening a table with unusual distributions asks a colleague. A finance user reviewing a report questions the number when it does not match their gut. Enterprise data governance evolved to support this consumer. Documentation lives in wikis. Quality is enforced by expected-value alerts that a human triages. Lineage is captured at the ETL job level, not the field level. Access is granted through role-based permissions and refined by manual data stewardship. All of this works when a human is at the end of the pipeline. An AI agent is not that consumer. An agent has no judgment. It processes what it is given and returns an answer. If the data is stale, the agent produces a stale answer with high confidence. If the lineage is broken, the agent cannot trace why. If a quality signal exists only as a wiki page, the agent cannot use it. The Four Gaps Most Enterprises Have Across the AI-in-production work I have seen, the same four gaps show up almost every time. Gap 1: Machine-Readable Data Contracts Most contracts exist as documentation, not as programmatic constraints. An agent cannot ask a Confluence page whether it is safe to trust a field. Data contracts need to be enforced at the platform layer, with schema, type, freshness, and quality guarantees expressed as executable rules. Gap 2: Use-Case-Aware Quality Fitness A dataset that is 95 percent complete may be fine for a marketing dashboard and completely wrong for a clinical AI model. Traditional data quality checks are use-case-agnostic. Agentic AI requires quality signals that answer a different question: is this data fit for this specific decision, right now? Gap 3: Field-Level Lineage That Updates in Real Time When a pipeline changes, human consumers get an email. Agents get a wrong answer. Lineage systems need to update as pipelines evolve and expose change signals in a form agents can consume, not just visualize. Gap 4: A Discovery Layer Agents Can Query Most catalog systems are designed for humans to browse. Agents need a machine interface to ask questions like which tables contain the concept I care about, and which of them is authoritative for this domain. Design Principles for Agentic Data Governance Closing these gaps does not require rebuilding the entire data platform. It requires making governance executable in the same path where the agent retrieves data, evaluates context, and produces an answer. Three design principles matter most. Start with the decision, not the data. For each production AI use case, define what a wrong answer looks like and work backward to the data requirements that would prevent it. This surfaces the specific quality signals, lineage nodes, and freshness constraints that matter. Make governance runnable, not readable. Every governance artifact your agents depend on should be programmatically executable at inference time. If a rule cannot be checked in code, an agent cannot use it. Documentation is useful for humans, but for agents it is invisible. Instrument for continuous evaluation. A governance framework that only fires at deployment is not enough. Models drift, data drifts, and use cases evolve. The governance layer needs to continuously evaluate agent outputs against real-world outcomes and flag drift before it becomes damage. Reference Architecture: Runtime Data Governance for AI Agents A practical implementation usually introduces a lightweight runtime governance layer between the agent and the underlying data platform. The goal is not to slow the agent down. The goal is to give the agent a reliable way to ask whether the data behind an answer is safe to use. At a minimum, this pattern includes five components: a data catalog that exposes authoritative sources, a contract registry that stores schema and business rules as executable checks, a lineage service that tracks upstream dependencies at the field and metric level, a quality service that publishes freshness and fitness signals, and an agent guardrail service that evaluates these signals before the agent responds. Runtime flow: User question → Agent → Semantic/data resolver → Governance service → Catalog, contract registry, lineage service, and quality service → Pass/Warn/Block decision → Agent response. Layer Responsibility Example Signal Catalog Identify authoritative datasets and business definitions. Certified source for booked deal value. Contract registry Validate schema, data types, null thresholds, and business rules. Discount variance must use the approved baseline method. Lineage service Track upstream source, transformation, and metric dependencies. Metric changed because a new source was added. Quality service Publish freshness, completeness, anomaly, and fitness scores. Dataset refreshed within SLA and passed threshold checks. Agent guardrail Block, warn, or allow the answer based on governance signals. Answer allowed only if lineage and contract checks pass. The agent should not directly trust a dataset simply because it can access it. Before answering, it should evaluate the data path, the contract status, the freshness window, the lineage change history, and the use-case-specific fitness score. If any critical check fails, the agent should either decline to answer or return the answer with an explicit data reliability warning. How the Runtime Governance Check Works In practice, the check is a short pre-answer step. The agent does not need to understand every governance rule directly. It needs a stable contract with a governance service that can evaluate the data path and return a decision. The user asks a business question.The agent resolves the requested metric, entity, dataset, or semantic concept.The agent calls the governance service with the resolved data assets and intended use case.The governance service checks catalog certification, contract status, lineage changes, freshness, completeness, and use-case fitness.The service returns a pass, warn, or block decision with machine-readable reasons.The agent answers, adds a caveat, escalates, or declines based on that decision. What a Machine-Readable Data Contract Actually Looks Like The abstract idea of a data contract only becomes real when you can point to one that an agent can actually consume. Here is a compact YAML example for a deal variance metric, expressing schema constraints, business rules, freshness expectations, and quality thresholds in a single artifact: YAML contract: dataset: deal.discount_variance schema: - field: discount_variance_pct type: decimal(18,2) required: true calculation: approved_discount_baseline_v2 - field: source_system type: string allowed_values: [crm_v3, revenue_hub] freshness: sla_hours: 24 breach_action: warn quality: completeness_threshold: 0.95 anomaly_score_max: 3.0 lineage: change_window_days: 30 on_upstream_change: require_review With this in place, an agent can call a single governance endpoint before responding, receive a machine-readable pass, warn, or block decision, and either answer confidently, answer with a caveat, or decline. The rule is not buried in a wiki page. It is live at inference time. Example Runtime API Pattern The runtime call does not need to be complicated. A minimal request can identify the metric, dataset, use case, and decision context. The response should be small enough for the agent to use directly in its control flow. JSON POST /governance/evaluate Request: { "metric": "deals.discount_variance_pct", "dataset": "deals.discount_variance", "use_case": "deal_desk_agent_review", "decision_context": "discount_variance_explanation" } Response: { "decision": "warn", "reasons": ["upstream_lineage_changed", "freshness_within_sla"], "agent_action": "answer_with_caveat" } In the agent workflow, this response becomes a control decision. A pass allows the agent to answer normally. A warn allows the answer but requires a reliability caveat. A block prevents the answer and routes the request to review, remediation, or a safer fallback path. Pseudocode: Turning Governance Into Agent Control Flow Python decision = governance.evaluate(metric, dataset, use_case) if decision.status == "block": return decline_with_reason(decision.reasons) if decision.status == "warn": return answer_with_caveat(query, decision.reasons) return answer(query) This is the core shift: governance is no longer a document the team reads during design review. It becomes a runtime dependency that the agent uses to decide whether to answer, qualify the answer, or stop. Runtime Checks an AI Agent Should Perform Before Answering Is this dataset or metric certified for the requested business domain?Has the schema changed since the agent workflow was last validated?Did all required fields meet completeness and validity thresholds?Is the data fresh enough for the decision being requested?Has any upstream lineage changed within a defined risk window?Does the requested answer depend on a metric with multiple calculation methods?Should the agent answer, warn, escalate, or decline based on the governance outcome? This does not require a heavyweight approval workflow for every query. In many cases, the runtime check can be a fast metadata call that returns a simple decision: pass, warn, or block. The important design principle is that governance must be available in the same execution path as the agent response, not in a separate documentation process that only humans can interpret. Failure Modes and Runtime Controls Failure mode What causes it Runtime control Stale answer Dataset missed its refresh SLA. Freshness check with warn or block behavior. Wrong metric Multiple calculation methods exist for the same business concept. Contract and semantic registry validation. Silent lineage change An upstream source or transformation changed after validation. Field-level lineage check within a defined risk window. Misused dataset The dataset is accessible but not certified for the requested domain. Catalog certification and use-case fitness check. Incomplete evidence Required fields fail completeness or validity thresholds. Quality service decision with explicit failure reasons. A Concrete Example From the Field On one enterprise AI project in a regulated environment, we deployed an agentic assistant to help analysts explore a large deal registration and booking dataset. Early testing looked solid. Several weeks into production, the agent began returning confidently wrong answers about a specific discount variance metric. The model had not changed. The prompt had not changed. What changed was an upstream ingestion job that added a new source that computed discount against a different price baseline. A human analyst would likely have questioned the number because it felt off. The agent did not. It saw a valid number in a valid field and reported it as authoritative. The fix was not in the model. We added a machine-readable contract for the approved discount baseline, a lineage signal for recent upstream changes, and a runtime check the agent could call before answering. After that, the same failure could not recur silently. The agent either answered correctly or flagged that the underlying data had changed and required review. The lesson was not that agents are unreliable. It was that agent reliability is a property of the data layer, not the model layer. Once we treated the governance layer as an active runtime dependency instead of static documentation, the entire class of silent-failure risk collapsed. Implementation Considerations Cache low-risk governance decisions to reduce latency, but recheck high-risk metrics at runtime.Separate warn rules from block rules so agents can still answer safely when risk is explainable.Version data contracts alongside pipelines, semantic models, and metric definitions.Log every agent answer with the governance decision, reasons, dataset version, and lineage snapshot used.Start with high-risk metrics and regulated workflows before expanding the pattern across the broader data estate. Why This Belongs in the Architecture, Not the Prompt Prompt engineering can reduce some surface-level errors, but it cannot solve a missing contract, stale dataset, broken lineage path, or ambiguous metric definition. Those failures sit below the model. They need to be handled in the platform architecture, where data access, metadata, quality, lineage, and policy decisions are available at runtime. For teams building enterprise AI agents, the practical takeaway is straightforward: treat runtime governance as part of the agent stack. If an agent can call a retrieval service, vector index, SQL endpoint, or workflow tool, it should also be able to call a governance service before committing to an answer. The next generation of enterprise AI reliability will not come only from better models. It will come from data platforms that can tell agents, in real time, whether an answer is safe to give. About the Author. Avinash Maddineni is a Lead Data Engineer with 15 years of enterprise data infrastructure experience across healthcare, financial services, energy, and travel. He builds agentic AI and data governance systems at Fortune 500 scale and is founder of PureStrokeAI (USPTO provisional patent filed May 2026).

Compliance-reporting teams keep spreadsheets in the loop for a practical reason: a workbook lets domain experts inspect assumptions, formulas, source rows, and intermediate values without reading a line of application code. That transparency is genuinely useful, and it's a big part of why replacing Excel outright so often fails to stick. The trouble starts once that workbook becomes part of a repeatable, audited reporting process — a regulatory filing, an IFRS report, a periodic compliance submission. At that point, a shared Excel file isn't enough on its own. What's actually needed is version control, validation, an audit trail, a review step, and a reliable way to connect the spreadsheet's logic to the systems downstream. The spreadsheet itself isn't the problem. It's a review surface domain experts genuinely need. The problem is treating it as a loose file sitting outside the application. The goal isn't to eliminate spreadsheets, but to preserve the spreadsheet experience while letting the application govern how it's used. This article walks through an architecture that keeps the workbook where domain experts can see it, but moves execution — validation, calculation, output generation, logging — into a Java application. The scenario is inspired by a real-world IFRS reporting project, and the same architecture applies to regulatory reporting, statutory filings, actuarial review, and other spreadsheet-driven compliance workflows. The pattern itself doesn't require a specific product: it works with any spreadsheet engine that can load a workbook and expose read/write access to Java, and parts of it apply even if you only use a file library like Apache POI at the edges. Three Ways Teams Usually Respond Rewrite everything in Java. Engineering gets control, tests, and CI. But the calculation logic moves away from the people who understand it. Every threshold change, every new currency, every adjusted formula now goes through a sprint. Sometimes that's correct — if the rules are stable and nobody inspects formulas, do this. For living, business-owned logic, it breeds shadow spreadsheets. Leave the desktop spreadsheet alone. Finance keeps full flexibility. The organization keeps none of the guarantees: no version control, no audit trail, no way to prove which file produced the submitted numbers. Use a file library only at the edges. Java imports the workbook, exports the results. Better — but the correction loop still happens in desktop Excel: download, fix locally, re-upload, re-validate, repeat. Every round trip is an audit gap. Now there is a fourth option: embed the workbook directly into the web application. Domain experts continue working in a familiar spreadsheet interface, while the application governs when users can edit data, when validation runs, which outputs become visible, and how every operation is logged. The rest of this article is about what that looks like in practice. The Big Idea: One Workbook, Two Roles In this pattern, the workbook plays two roles at the same time: For users, it is the interface. They inspect rows, correct values, maintain rule tables, and review generated outputs in a familiar grid.For the application, it is a runtime artifact. Java loads a known template, reads specific sheets and regions, runs validation, writes outputs, and records every run. The design decision that makes this work: Java never wanders through the workbook looking for data. It reads and writes only through agreed sheets and regions — a contract. Finance owns what's inside the regions: values, formulas, rules. Engineering owns the boundary and everything behind it: execution, permissions, persistence, export. Let's see the two stages of a typical reporting workflow through this lens. Stage 1: Let Users Fix Data Issues Without Leaving the App Reporting source data almost never arrives clean. A currency code says US instead of USD. An FX rate is missing. A service fee breaks a policy limit. The template for this stage has two sheets. Input CSV holds the source rows users can inspect and correct. ETL Rule holds the validation rules — as an ordinary spreadsheet table with columns like Field, Check, and Allowed Values. A rule row might say: currency must be one of USD, EUR. Finance can read and change these rules without asking anyone. When the user clicks Run Validation, the application takes over. To make this concrete: the examples in this article use Keikai Spreadsheet, a Java-based spreadsheet UI component, to embed the workbook in the browser and read and write it from Java — though the same three-step logic applies with any comparable engine. Conceptually, the Java service reads the data rows, reads the rule rows, and checks every row against every rule: Java List<SourceRow> rows = sheetReader.readTable(workbook, "Input CSV"); List<Rule> rules = ruleParser.parse(sheetReader.readTable(workbook, "ETL Rule")); for (SourceRow row : rows) for (Rule rule : rules) rule.check(row).ifPresent(report::add); Notice what this is not: the rules are not hard-coded in Java. Java only knows how to read the rule table and apply generic checks. The actual business knowledge — which currencies are allowed, what a valid fee looks like — stays in the workbook where its owners can see it. One detail carries most of the user experience: every validation error records which cell failed — sheet, row, and column. That lets the UI show a panel saying “policy P-1024, field currency, value US, expected USD or EUR” with a link that jumps the user straight to the offending cell. They fix it in the grid, click run again, and validation passes. Compare that to the traditional loop — download, fix in Excel, upload, pray. Here, nothing leaves the system, and every edit can be logged with user, timestamp, old value, and new value. Stage 2: Generate Outputs Under Application Control Once the data is clean, the second stage produces the actual reporting outputs: journal entries, impact tables, export-ready CSV sheets. The input is a policy sheet with assumptions (premium totals, fees, FX rates) plus a rule table that maps accounting events to journal lines. Before the run, the application shows only the input sheet — output sheets stay hidden, because they don't exist meaningfully yet. When the user triggers generation, the same Keikai-backed workbook is read and written from Java: it reads the inputs, computes the metrics, builds the journal rows, and writes them back into the workbook: Java PolicyInput policy = policyReader.read(workbook, "Policy Input"); Metrics metrics = deriveMetrics(policy); // plain Java arithmetic List<JournalRow> rows = journalBuilder.build(readJournalRules(workbook), metrics); sheetWriter.replaceTable(workbook, "Journal Entries", rows); revealSheets(workbook, "Journal Entries", "Report Impact", "Journal CSV"); The interesting part is the last line. Sheet visibility is an application decision: outputs appear only after a successful run, so a reviewer can never mistake stale output for fresh output. The reviewer then sees everything in one place — assumptions, rules, generated journals, report impact — in the same grid, and the export button produces a file the application has logged and versioned. deriveMetrics itself is deliberately simple — a handful of multiplications and subtractions. In a real system it may be far more complex, or it may even delegate back to formulas in the workbook. The architecture doesn't change: inputs go into agreed regions, outputs come from agreed regions, and Java owns the trigger. The Part Everyone Skips: The Workbook Is Now an API The moment Java code depends on a sheet named ETL Rule with a header called Allowed Values, the workbook has stopped being a document. It has become an interface — and interfaces break when they're changed casually, without review. The fix is to make the contract explicit and test it. Distinguish two kinds of change: Value changes – a new allowed currency, an adjusted threshold, a reviewed formula edit. These live inside the contract. Finance can make them without touching Java.Structural changes – renaming a sheet, deleting a header, moving an output table three columns right. These are API changes and should be reviewed like one. Then write this test: Java @Test void templateSatisfiesReportingContract() { Workbook wb = engine.load("reporting-template.xlsx"); assertSheetExists(wb, "Input CSV", "ETL Rule", "Journal Entries"); assertHeaders(wb, "ETL Rule", "Field", "Check", "Allowed Values"); } It looks almost too simple to matter, but most real-world workbook integration failures are exactly this mundane — a renamed sheet or a deleted header, discovered the night before a regulatory filing is due. Catching it in CI, before any template goes live, is what makes the difference. Finally, log runs, not just files: template version, who ran it, validation status, output row counts, a hash of the inputs. When someone asks “why does this quarter's filing look wrong?”, you answer from the run log instead of from archaeology on a shared drive. For compliance teams, these controls turn the workbook from an informal file into evidence the organization can explain. A reviewer can trace which template version produced a number, which source data was used, who ran the process, whether validation passed, and which output was exported. If a template structure changes, the contract test shows whether the workbook still satisfies the application’s required sheets and headers before it reaches production. In other words, the system does not just calculate results; it records the evidence needed to defend how those results were produced. When to Consider a Simpler Approach This approach pays off when the workbook is a genuine shared language between domain experts and developers — something both sides actually read, edit, and rely on. If the rules rarely or never need to change, and nobody inspects formulas, plain Java is simpler to test and operate. And if the workbook is really just a transfer format between systems, a straightforward import/export covers it. Takeaways The compliance-reporting spreadsheet doesn't have to be rewritten or worked around. Put it inside the application and split ownership along a clear line: The workbook owns what users must see and maintain: source rows, rule tables, assumptions, reviewable outputs.The application owns execution: validation, generation, sheet visibility, permissions, logging, export.The contract between them — named sheets, headers, regions — is documented, tested in CI, and changed only with review. Do that, and the workbook stops being an unversioned file nobody can fully account for. It becomes a governed part of the application — the place where domain experts and the system finally agree on the numbers.

By Hawk Chen CORE

Landing a data engineering role means clearing a gauntlet that no other software discipline has to face all at once: airtight SQL, production-grade Python, data modeling instincts, distributed-compute fluency (Spark, warehouses, ETL), and system design that has to survive real data volume. Generic coding prep barely scratches the surface, and "just grind LeetCode" advice falls apart the moment an interviewer asks you to model a slowly changing dimension or reason about a skewed join. So we did the work. We evaluated the resources data engineers actually use, judged on five things that matter: relevance to the DE interview loop, depth of practice, realism of the questions, feedback quality, and price. Below is the ranked list. A quick note on methodology: this ranking favors resources that target the data engineering loop specifically, not generic algorithm grinding. That bias is intentional, and it is why the order may surprise you. 1. DataDriven.io Most "interview prep" platforms were built for generic SWE roles and bolt on a SQL section as an afterthought. This one was built from the ground up for the data engineering loop. The catchphrase you will hear repeated in DE communities is that DataDriven.io is LeetCode for data engineers, and it fits: instead of inverting binary trees, you are writing window functions against realistic schemas, designing star schemas, debugging an ETL transform, and reasoning about partitioning, all in an in-browser SQL and Python sandbox that runs your query against real data and tells you exactly where it broke. It is also the rare place where the whole product is built for the job rather than adjacent to it, which is why datadriven.io is great for data engineer interview prep specifically: SQL practice that ramps to multi-CTE analytics, a deep set of Python practice problems, plus data modeling, dimensional modeling, PySpark, and system-design tracks, with execution-based feedback and a difficulty curve that reaches the staff-level questions that actually separate offers from rejections. Verdict: The most targeted, realistic data engineering interview practice available today. Earns the top spot. 2. "Cracking the Coding Interview" (the book, by Gayle Laakmann McDowell) A deserved classic, and intentionally a book rather than a website. CTCI is still the best single artifact for understanding how technical interviews are actually structured: how the conversation flows, how to think out loud so the interviewer can follow your reasoning, how to recover when you get stuck, and how to handle the behavioral and negotiation segments that strong candidates routinely fumble. Most people lose offers not because they could not solve the problem but because they could not show their work, and this book is the canonical fix for that. Where it falls short for our purposes is scope. It will not teach you windowed SQL, slowly changing dimensions, or how to design a lakehouse, and its algorithm focus skews toward generalist software roles rather than the data engineering loop. The data structures and big-O chapters are still worth a pass because algorithm screens do show up, but treat them as a refresher, not your main event. Read CTCI once early in your prep to fix your interview mechanics, internalize the communication patterns, then spend the rest of your time on hands-on, domain-specific platforms. Verdict: Essential reading for interview mechanics; not a substitute for domain practice. 3. "Designing Data-Intensive Applications" (the book, by Martin Kleppmann) If CTCI teaches you how to interview, "DDIA" teaches you what a data engineer is actually supposed to know. Replication, partitioning, consistency models, batch versus stream processing, storage engine internals, the failure modes of distributed systems: this is the conceptual backbone of nearly every data engineering system design round. When an interviewer asks why you would choose a log-structured merge tree over a B-tree, or how you would keep two datastores in sync without losing events, the answers live in these pages. It is dense, and it is emphatically not an interview drill book. You will not find practice questions, and you cannot cram it the night before. What it gives you instead is judgment: the candidate who has internalized DDIA answers "how would you design this pipeline" with the calm of someone who has already thought through the tradeoffs, names the failure cases before being prompted, and explains why a choice holds up under real data volume. Read it slowly over weeks, ideally early in your prep, and pair it with a hands-on platform so the concepts attach to actual queries and schemas rather than floating as theory. Verdict: The definitive conceptual reference. Read it slowly, alongside real practice. 4. LeetCode The default destination, and it earns its spot for one practical reason: the Database problem set is sizable, the algorithm catalog is enormous, and the platform's brand means a large share of companies still pull their initial coding screen straight from it. If your target company is known to run a generic algorithm round before the data-specific rounds, you need exposure here, and the sheer volume of problems plus community discussion means you will rarely be surprised by a pattern you have never seen. The catch for data engineers is that LeetCode was built for the algorithm interview, not the DE loop. Its SQL section is genuinely solid but secondary; the questions are puzzle-shaped rather than drawn from real schemas, and you will not find data modeling, ETL design, dimensional modeling, or Spark anywhere on the platform. There is also a real failure mode here: candidates over-invest in LeetCode because it is comfortable and gamified, then walk into a DE loop under-practiced on the things that actually decide it. Use it deliberately to clear the algorithm gate and to keep your raw coding sharp, then move the bulk of your hours to resources that target data engineering directly. Verdict: Necessary for the algorithm screen; thin for the data-engineering-specific rounds. 5. HackerRank HackerRank is where a surprising number of companies host their take-home and timed online assessments, so practicing in its environment carries a payoff most resources cannot offer: you get comfortable with the exact editor, the exact test-case runner, and the exact time-pressure UI you may actually be scored in. For an assessment you cannot retake, that familiarity is worth real points, because fighting an unfamiliar interface while the clock runs is a self-inflicted way to lose. Its SQL and problem-solving tracks are beginner-friendly, well-structured, and free to work through. The ceiling, though, is lower than you want for a senior DE loop. The problems lean academic and self-contained rather than job-realistic, the SQL rarely reaches the messy multi-table analytics that real interviews probe, and there is nothing on modeling, pipelines, or system design. The smart way to use HackerRank is as format rehearsal: run a few timed sets so the assessment environment feels routine, then build your actual depth somewhere that mirrors the work. Do not let a green checkmark on an easy problem set convince you that you are loop-ready. Verdict: Great for getting comfortable with the testing environment; limited depth. 6. SQLZoo A long-running, completely free interactive SQL tutorial that runs entirely in the browser with no signup, no setup, and no paywall. It walks you from SELECT basics through joins, grouping, subqueries, and window functions, with short hands-on exercises after each concept so you are writing real queries from the first lesson rather than just reading about them. For anyone whose SQL has gone rusty, or who learned it informally and has gaps they cannot quite name, it is the most painless way to rebuild muscle memory before stepping up to interview-grade problems. It is a teaching tool, not an interview platform, and you should treat it as exactly that. The problems stay introductory, the datasets are small and tidy, and there is nothing on data modeling, ETL, pipelines, or system design — the parts of the loop that actually separate data engineers from analysts. Its value is as a fast diagnostic and warm-up: work through the sections that feel shaky, confirm your fundamentals are solid, then graduate to harder, execution-based practice against realistic schemas. Linger here too long, and you will plateau well below where a real interview will push you. Verdict: A friendly free SQL primer; foundational rather than interview-level. 7. "Python for Data Analysis" (by Wes McKinney) Written by the creator of pandas, this is the reference for the kind of data-wrangling Python that shows up constantly in DE take-homes and pairing rounds: reshaping, grouping and aggregating, merging on imperfect keys, handling missing values, parsing dates, and cleaning the kind of messy tabular data that never looks like a tidy LeetCode input. Many data engineering interviews quietly assume this fluency, then hand you a notebook and a dirty CSV and watch how you move; if your Python is sharp on algorithms but clumsy on real data manipulation, this book is exactly the gap-closer. It is a library-and-technique book, not interview prep, and it will not touch SQL, data modeling, distributed compute, or system design. There are also no interview questions to grind, which is fine, because its job is to make the tools second nature so that during a timed exercise you are reasoning about the problem instead of fumbling for the right pandas idiom. Read the chapters on data loading, cleaning, and group operations, keep it nearby as a reference, then go apply the techniques in hands-on practice against problems that actually resemble the job. Verdict: The definitive practical Python reference for data work; not a drill book. 8. "Fundamentals of Data Engineering" (the book, by Joe Reis & Matt Housley) Another deliberate book pick, and the best single survey of the modern data engineering lifecycle: generation, ingestion, storage, transformation, and serving, plus the cross-cutting concerns like orchestration, data quality, and governance that interviewers increasingly probe. Where DDIA goes deep on systems internals, this book goes broad on how the pieces fit together into a working data platform, which is precisely the framing you want for the "walk me through how you'd build X" and "what would you consider before choosing this approach" portions of a loop. It is a framework-and-vocabulary book, not a practice book, and that is both its strength and its limit. It will give you the mental model and the shared language to discuss tradeoffs like a practitioner, which makes you sound, accurately, like someone who understands the field. But it contains no exercises, so reading it alone will not build the hands-on skill an interviewer also tests. Use it to organize everything you know into a coherent lifecycle, fill the conceptual gaps, then go write the queries and design the schemas somewhere that gives you real feedback. Verdict: The best lifecycle overview in print; conceptual, not hands-on. 9. Mode SQL Tutorial A free, well-regarded interactive SQL tutorial built by an analytics company, which shows in its framing: it teaches SQL the way analysts and engineers actually use it, oriented around answering real questions from data rather than solving abstract puzzles. It runs in the browser, takes you from the basics through intermediate analytics queries including aggregation and the early window-function territory, and the explanations are unusually clear about why a query is shaped the way it is. For someone shoring up SQL foundations before diving into harder problems, it is one of the cleanest no-cost on-ramps available. Like SQLZoo, it is a tutorial rather than an interview-prep platform, so it stops well short of the difficulty a real DE loop will throw at you, and it covers none of the modeling, pipeline, or system-design ground. It is best read as a companion to a hands-on platform: use Mode to internalize the analytical mindset and clean up your SQL fundamentals, then take that foundation into execution-based practice where the problems are harder, the schemas messier, and the feedback tells you exactly where your query went wrong. Verdict: A clean free SQL on-ramp; foundational rather than interview-level. 10. Pramp/Interviewing.io (mock interviews) Rounding out the list: peer and expert mock interviews. All the solo practice in the world cannot reproduce the specific pressure of explaining your reasoning out loud to a real human while a clock runs and someone is judging you, and that pressure is exactly where otherwise-prepared candidates fall apart. A handful of mock loops surface the weaknesses you cannot see in yourself: the long silences, the jumping to code before clarifying the question, the inability to narrate a tradeoff. Pramp pairs you with peers for free, while Interviewing.io connects you with experienced interviewers, often anonymously, for higher-fidelity feedback. The honest limitation is supply and specificity. Data-engineering-focused interviewers are scarcer than generalist software ones, so depending on availability, you may land in an algorithm or general system-design mock that only partially mirrors a true DE loop. That is still worth doing, because the communication skills, the structure, the clarifying questions, the calm narration, transfer directly regardless of the exact problem. Schedule one or two once your technical prep is underway, treat the feedback as data, and fix the delivery habits well before the interview that counts. Verdict: Best for rehearsing delivery and nerves; DE-specific matches can be hit-or-miss. How to Actually Use This List You do not need all ten. A focused plan beats a scattered one: Build the foundation. Skim CTCI for interview mechanics and start DDIA for concepts.Do the reps where it counts. Spend the bulk of your time on hands-on, DE-shaped practice that maps directly onto what you will be asked (see #1).Patch specific gaps. Use LeetCode for the algorithm screen, SQLZoo or the Mode tutorial to shore up SQL, and a mock interview or two to rehearse out loud. The candidates who get offers are not the ones who consumed the most content. They are the ones who practiced the actual job. Pick the resources that put you closest to it, start today, and write more queries than you read. Good luck with your loop.

AWS Glue makes it easy to get a PySpark pipeline running quickly. It is significantly harder to build one that stays maintainable as logic grows, performs reliably at scale, and does not quietly accumulate operational debt over time. Most Glue pipelines start simple and become difficult to manage gradually — formulas get hardcoded, modules grow without boundaries, output files proliferate, and before long a single job is doing too many things in ways that are hard to test, hard to debug, and expensive to change. This article presents a set of design principles drawn from production Glue ETL pipelines processing billions of rows. Each principle is independent — you do not need to adopt all of them to benefit from any one. But together they form a coherent approach to building Glue pipelines that are modular, observable, cost-efficient, and built to last. Principle 1: Externalize Logic Into Config, Not Code The single most impactful structural decision in a Glue pipeline is where business logic lives. When formulas, dataset references, column selections, and filter conditions are hardcoded in PySpark, every change requires modifying job code, redeploying, and re-validating the full pipeline. A one-line formula change carries the same deployment risk as a structural refactor. Over time, this creates a strong disincentive to make changes, and the pipeline calcifies. The better pattern is to treat the Spark job as a generic executor and externalize all business-specific declarations into configuration. Formulas are declared as config entries with operands, rounding rules, and output names. Dataset loading behavior — which table, which columns, which filters, whether to cache — is declared per source rather than scripted per job. Schema shapes for complex types are declared explicitly rather than inlined. JSON { "source_table": "headcount_actuals", "database": "finance_db", "select_columns": ["site", "badge_type", "headcount", "fiscal_week"], "filters": [{"column": "is_active", "value": "Y"}], "rename": {"hc_count": "headcount"}, "cache": true } When a new dataset is needed, a new config entry is added — no Spark code changes. When a formula changes, the config entry is updated — no job redeployment required. The job itself becomes stable and generic; only config changes as business requirements evolve. This principle pays increasing dividends over time. Pipelines with externalized logic are faster to modify, safer to deploy, and easier to hand off because the business rules are readable independently of the execution engine. Principle 2: Design Modules With Explicit Boundaries A Glue job that does everything in one place is easy to write and hard to maintain. As pipelines grow, the instinct to add more logic to an existing job accelerates technical debt faster than almost any other decision. The more durable pattern is to decompose computation into modules with explicit input and output contracts. Each module receives one or more DataFrames, applies a focused set of transformations, and produces a named output DataFrame. Modules communicate exclusively through in-memory DataFrame references — there is no disk I/O between stages, no shared mutable state, and no implicit dependency on execution order beyond what the data flow itself requires. Utilities follow the same boundary principle, organized into two layers. Generic pipeline utilities handle cross-cutting concerns — file writing, dataset loading, filtering, deduplication, pivot operations — and are shared across all modules. Module-specific utilities implement transformation logic scoped to a single module and are never invoked outside it. This structure means adding a new module requires only writing its scoped utilities and wiring it into the pipeline. The generic layer is never touched. Existing modules are never at risk from new module development. The downstream benefit is testability. Each module with clean boundaries can be validated independently using mocked PySpark DataFrames with no Glue environment required. Engineers can run pytest locally against individual modules, iterate quickly, and deploy only after local validation passes. Principle 3: Choose Your Job Topology Deliberately A common default in complex pipelines is to split computation across multiple Glue jobs, using S3 as the handoff layer between stages. This is sometimes the right choice — but it should be a deliberate decision, not an instinct. Multi-job topologies make sense when stages have genuinely different compute profiles, when intermediate outputs need to be reused independently by other consumers, or when a stage failure should not force a full recompute from the beginning. In these cases, job separation gives you independent retry boundaries, independent DPU sizing, and the ability to schedule stages on different cadences. Single-job topologies — where the full pipeline runs within one Spark session — make sense when all computation is tightly coupled, modules share the same input datasets, and intermediate outputs have no standalone value. Running everything in one session eliminates cold start overhead for intermediate stages, avoids the cost of serializing data to S3 and deserializing it back between jobs, and keeps the execution model simple to reason about: one trigger, one job, one result. The question to ask is whether the stages truly need to be independent. If intermediate S3 persistence adds coordination complexity without adding value — no independent consumers, no differential retry requirements, no meaningful DPU difference between stages — then collapsing to a single job is usually faster, simpler, and cheaper. If stages have real independence requirements, splitting them is the right call and the operational overhead is justified. Neither topology is inherently superior. The mistake is defaulting to one without evaluating the trade-offs for the specific pipeline at hand. Principle 4: Overlap Writes With Computation When Latency Matters Overlapping writes with computation is a well-established technique in high-performance computing, deep learning training, and heavy database operations. The core idea is to hide the slow latency of I/O operations by running them in the background while the CPU or GPU continues processing data. Rather than waiting for a write to complete before starting the next computation, both proceed simultaneously — I/O latency is absorbed into computation time rather than added on top of it. In Glue ETL pipelines, the same principle applies directly. In a pipeline where multiple output DataFrames are produced, the naive write strategy — complete all computation, then write all outputs sequentially — has two compounding problems. First, it creates a peak memory spike: all computed results are held in memory simultaneously while writes proceed one by one. Second, it serializes work that does not need to be serial: every millisecond spent waiting for S3 acknowledgment is a millisecond the Spark executors are idle. This is worth addressing only when latency is a meaningful constraint. For low-frequency batch jobs running overnight with no user-facing SLA, sequential writes are perfectly adequate. But for pipelines where users or downstream systems are waiting on results — or where job duration directly affects infrastructure cost — overlapping writes with computation delivers measurable wall-clock reduction. The two-phase write strategy implements this directly. Outputs from early modules are written to S3 in background threads immediately after those modules complete, running in parallel with later computation stages. By the time all computation finishes, a significant portion of the output data has already landed in S3. Remaining outputs are then flushed concurrently in a second phase. The implementation leans on Python's concurrent.futures.ThreadPoolExecutor to manage background write threads while the main Spark session continues computation on the driver. A generic write orchestration utility can wrap this pattern so individual modules never need to manage thread lifecycle directly — they simply declare their output and the utility handles scheduling, thread management, and error propagation. Python from concurrent.futures import ThreadPoolExecutor, as_completed def write_phase_a(write_tasks): with ThreadPoolExecutor(max_workers=len(write_tasks)) as executor: futures = {executor.submit(task["fn"], task["df"], task["path"]): task["name"] for task in write_tasks} for future in as_completed(futures): name = futures[future] future.result() logger.info(f"[Phase A] Write complete: {name}") The practical effect is that peak memory pressure is distributed over the job's lifetime rather than concentrated at the end, and total wall-clock time is reduced by the overlap between I/O and CPU-bound computation. For pipelines with many output datasets and a latency SLA to meet, the savings compound significantly. Principle 5: Right-Size Output Files With a Reusable Writer Utility Right-sizing output files is the practice of tuning file sizes to balance disk I/O performance, network transfer speeds, and downstream processing efficiency. Too many small files and downstream readers spend more time on metadata operations and S3 API calls than on actual data reads. Too few large files and parallelism suffers — readers cannot split work efficiently across threads or nodes. The target is consolidated, evenly sized files that match the read patterns of downstream consumers. Spark's default output behavior writes one file per partition, and partition counts are typically tuned for computation throughput rather than output shape. A job optimized for shuffle performance might produce hundreds of partitions, each containing a few megabytes of output data — perfectly reasonable for Spark internals, but harmful for any reader that comes after. This small file problem compounds over time as output partitions accumulate in S3 and the Glue Catalog metadata grows with them. The fix is a reusable writer utility that decouples output file sizing from Spark's internal partition count. Rather than accepting the default, the utility estimates the DataFrame's actual size, calculates the appropriate number of output files for a target file size — typically 128MB to 256MB per file — and coalesces partitions before writing. Python def write_optimized(df, output_path, partition_cols, target_file_size_mb=128): estimated_size_mb = df.rdd.map(lambda row: len(str(row))).sum() / (1024 * 1024) optimal_partitions = max(1, int(estimated_size_mb / target_file_size_mb)) df.coalesce(optimal_partitions) \ .write \ .partitionBy(*partition_cols) \ .parquet(output_path, mode="overwrite") Making this a shared generic utility rather than inline logic in each module has two practical benefits. First, it enforces consistent file sizing behavior across all outputs in the pipeline — no module accidentally writes thousands of tiny files because an engineer forgot to coalesce. Second, it centralizes the tuning knob: when the target file size needs to change — because downstream query patterns shift or a new consumer has different read characteristics — it changes in one place and applies everywhere. Right-sized output files improve Athena scan performance, reduce per-query S3 API costs, keep Glue Catalog partition metadata manageable, and make the output data easier to consume for any downstream system reading from S3. This is a low-effort, high-payoff improvement that applies to virtually every Glue pipeline writing to S3. Principle 6: Use Complex Types to Defer Denormalization SQL-based pipelines are constrained to flat, fully denormalized row structures at every intermediate stage because SQL has no native complex type support. This forces denormalization to happen early, inflating data volume at every subsequent join and aggregation. PySpark has native support for structs, maps, and arrays. Using these types at intermediate stages allows related values to be grouped logically without inflating row counts. A row that would require five denormalized rows in SQL can be represented as a single row with a struct or array column in Spark. Denormalization is then deferred to the final output layer only — applied once, at write time, for consumers that require flat structures. Everything upstream of the final write benefits from reduced volume, fewer shuffles, and faster joins. This principle is particularly impactful in pipelines with multi-level aggregations or wide schemas where dozens of metrics attach to the same dimensional key. Keeping those metrics grouped in a struct until the final output stage reduces the effective row count and join complexity throughout the pipeline. Principle 7: Build Observability Into Every Stage Glue jobs that fail silently or surface errors as opaque stack traces at the end of a long execution are expensive to debug. The investment in step-level observability pays back quickly the first time something goes wrong in production. The minimum viable observability pattern is row count logging at every materialization point. After each module completes and after each write, log the output row count with a descriptive label. This gives a running picture of data volume through the pipeline and makes it immediately obvious when a transformation has dropped rows unexpectedly or produced more rows than expected. Python def log_step(df, step_name): count = df.count() logger.info(f"[{step_name}] Row count: {count:,}") return df Pair this with a try/except/finally pattern at the job level that ensures spark.catalog.clearCache() is always called on exit — whether the job succeeds or fails — to release cached DataFrames and avoid memory leaks across retries. Python try: run_pipeline() except Exception as e: logger.error(f"Pipeline failed: {e}") raise finally: spark.catalog.clearCache() CloudWatch captures all logs automatically. When a job fails, the row count trail shows exactly where in the pipeline the problem occurred, making triage faster and reducing the time between failure and fix. Principle 8: Isolate Executions for Concurrency Pipelines that share compute resources across simultaneous executions create contention that is difficult to predict and expensive to manage. The common response — queue-based serialization — adds operational complexity without solving the underlying resource constraint. AWS Glue's execution model eliminates this problem structurally. Each job execution gets its own isolated DPU allocation. There is no shared compute pool. Ten simultaneous executions consume ten independent DPU allocations and do not interfere with each other in any way. Designing for this means treating each execution as fully independent: no shared state, no cross-execution coordination, no assumption about what other executions are running. Combined with idempotent writes — using overwrite mode so a retry produces the same result as the original execution — the pipeline becomes safe to run concurrently at any scale without additional coordination logic. The cost model reinforces this. Glue bills per DPU-second of actual compute consumed. An execution that takes eight minutes on 240 DPUs costs the same whether it runs alone or alongside a hundred other executions. There is no premium for concurrency and no shared pool to provision for peak load. Putting It Together These eight principles are independent but complementary. A pipeline that applies all of them is modular enough to develop in parallel, observable enough to debug quickly, cost-efficient enough to run at scale, and stable enough to maintain over time without accumulating structural debt. The quickest wins for most existing pipelines are Principles 1, 5, and 7 — externalizing logic into config, right-sizing output files with a shared utility, and adding row count logging at every stage. Each can be applied incrementally without restructuring the full pipeline. The remaining principles become more valuable as pipeline complexity grows and concurrency requirements increase. The underlying thesis is simple: a well-designed Glue pipeline should be easy to change, easy to test, easy to debug, and cheap to run. None of those properties require exotic infrastructure. They require deliberate design decisions applied consistently from the start.

By Janani Annur Thiruvengadam CORE

Why Delta Lake? Apache Parquet on cloud storage was a great first step for data lakes — but it left engineers dealing with a painful set of problems in production: No ACID transactions — concurrent reads/writes could corrupt data silentlySchema drift — nothing stopped upstream systems from changing column typesNo deletes or updates — GDPR compliance meant rewriting entire partitionsPainful failure recovery — half-written data after a job crash became your problem Delta Lake solves all of this by sitting on top of Parquet and adding a transaction log (_delta_log/) that records every operation atomically. On Databricks, Delta is the default table format, deeply integrated with Apache Spark, Auto Optimize, and the Photon execution engine. The Medallion Architecture The medallion (or multi-hop) architecture organizes data into three progressive refinement layers. Each layer has a clear contract with the layers around it. Each layer has a distinct responsibility: LayerAliasPurposeRetentionBronzeRawLand data as-is, preserve source fidelityYears (audit trail)SilverCleansedDeduplicate, validate, type-cast, conform schemasMonthsGoldAggregatedBusiness-level KPIs, domain-specific aggregatesMonths–Years The key design principle is: never skip a layer. Debugging a production incident is infinitely easier when you can replay from raw Bronze data. Delta Lake Internals: The Transaction Log Before writing a single line of code, it's worth understanding how Delta Lake achieves ACID semantics. Every write operation (INSERT, UPDATE, DELETE, MERGE) produces a new JSON entry in _delta_log/. This log-based approach gives you time travel for free — querying VERSION AS OF 2 simply replays only the log entries up to that point. Checkpoints every 10 commits keep read performance snappy even with thousands of versions. Setting Up Your Databricks Environment All code here runs on Databricks Runtime 13.x+ (which ships Delta Lake 2.x). For local dev, use the delta-spark package. # Databricks notebook — Runtime 13.3 LTS or higher # Delta Lake is pre-installed; no pip install needed from pyspark.sql import SparkSession from pyspark.sql import functions as F from pyspark.sql.types import ( StructType, StructField, StringType, LongType, DoubleType, TimestampType, BooleanType ) from delta.tables import DeltaTable # On Databricks, SparkSession is pre-created as `spark` # For local testing: # spark = (SparkSession.builder # .appName("medallion-pipeline") # .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") # .config("spark.sql.catalog.spark_catalog", "spark.sql.delta.catalog.DeltaCatalog") # .getOrCreate()) # Unity Catalog paths (recommended for production) CATALOG = "prod" BRONZE_DB = f"{CATALOG}.bronze" SILVER_DB = f"{CATALOG}.silver" GOLD_DB = f"{CATALOG}.gold" # DBFS / external storage path (for raw file landing) RAW_LANDING = "abfss://[email protected]/events/" Building the Bronze Layer Bronze is your append-only raw ingestion layer. The goal is landing data with zero transformation — preserve everything, even malformed records. Schema is inferred or declared loosely. # ── Bronze Ingestion ────────────────────────────────────────────────────────── # Using Auto Loader (cloudFiles) — the Databricks-native incremental ingest # mechanism. It tracks which files have been processed via a checkpoint, # so re-running never double-injects data. raw_event_schema = StructType([ StructField("event_id", StringType(), True), StructField("user_id", StringType(), True), StructField("event_type", StringType(), True), StructField("event_ts", StringType(), True), # keep as string in bronze StructField("properties", StringType(), True), # raw JSON blob StructField("session_id", StringType(), True), StructField("platform", StringType(), True), ]) bronze_stream = ( spark.readStream .format("cloudFiles") .option("cloudFiles.format", "json") .option("cloudFiles.schemaLocation", "/checkpoints/bronze_events_schema") .option("cloudFiles.inferColumnTypes", "false") # keep raw types .schema(raw_event_schema) .load(RAW_LANDING) # Enrich with ingestion metadata — critical for debugging .withColumn("_ingest_ts", F.current_timestamp()) .withColumn("_source_file", F.input_file_name()) .withColumn("_ingest_date", F.to_date(F.current_timestamp())) ) ( bronze_stream.writeStream .format("delta") .outputMode("append") .option("checkpointLocation", "/checkpoints/bronze_events") .option("mergeSchema", "true") # allow new columns from upstream .partitionBy("_ingest_date") # partition for incremental silver reads .trigger(availableNow=True) # run-once trigger (for scheduled jobs) .tableCheckpoint(f"{BRONZE_DB}.events_raw") .toTable(f"{BRONZE_DB}.events_raw") ) Pro tip: Always add _ingest_ts and _source_file metadata columns in Bronze. When an upstream system sends corrupt data at 3 AM, these columns tell you exactly which file batch caused it. Transforming to Silver Silver is where the real engineering work happens: deduplication, type casting, schema validation, and applying business rules. We also handle CDC (Change Data Capture) upserts here using Delta's MERGE. # ── Silver Transformation ───────────────────────────────────────────────────── def transform_bronze_to_silver(bronze_df): """ Apply cleansing and conforming rules to raw Bronze events. Returns a Silver-ready DataFrame with enforced schema. """ return ( bronze_df # 1. Parse timestamps properly .withColumn("event_ts", F.to_timestamp("event_ts", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")) # 2. Parse the raw JSON properties blob into a struct .withColumn("props", F.from_json( F.col("properties"), schema="page_url STRING, referrer STRING, duration_ms LONG, revenue DOUBLE" )) .drop("properties") # 3. Normalize platform values .withColumn("platform", F.lower(F.trim(F.col("platform")))) .withColumn("platform", F.when( F.col("platform").isin("ios", "android"), F.col("platform") ).when( F.col("platform").isin("web", "browser", "desktop"), F.lit("web") ).otherwise(F.lit("unknown"))) # 4. Filter out test/internal traffic .filter(~F.col("user_id").startswith("test_")) .filter(F.col("event_id").isNotNull()) # 5. Deduplicate within the micro-batch (window by event_id) .dropDuplicates(["event_id"]) # 6. Add Silver metadata .withColumn("_silver_ts", F.current_timestamp()) .withColumn("event_date", F.to_date("event_ts")) # partition key ) # Upsert into Silver using MERGE (handles late-arriving / duplicate events) def upsert_to_silver(micro_batch_df, batch_id): micro_batch_df = transform_bronze_to_silver(micro_batch_df) silver_table = DeltaTable.forName(spark, f"{SILVER_DB}.events_clean") ( silver_table.alias("target") .merge( micro_batch_df.alias("source"), "target.event_id = source.event_id" # dedup key ) .whenMatchedUpdateAll() # update if record arrived late with corrections .whenNotMatchedInsertAll() # insert new records .execute() ) # Stream from Bronze → Silver using foreachBatch ( spark.readStream .format("delta") .option("readChangeFeed", "true") # CDF — only process new Bronze rows .table(f"{BRONZE_DB}.events_raw") .writeStream .foreachBatch(upsert_to_silver) .option("checkpointLocation", "/checkpoints/silver_events") .trigger(availableNow=True) .start() ) Aggregating to Gold Gold tables are business-ready aggregates consumed directly by BI tools, dashboards, and ML feature pipelines. They are typically batch-refreshed on a schedule. # ── Gold Aggregation ────────────────────────────────────────────────────────── daily_revenue = ( spark.table(f"{SILVER_DB}.events_clean") .filter(F.col("event_type") == "purchase") .filter(F.col("event_date") >= F.date_sub(F.current_date(), 90)) # rolling 90d .groupBy("event_date", "platform") .agg( F.sum("props.revenue").alias("total_revenue"), F.countDistinct("user_id").alias("unique_buyers"), F.count("event_id").alias("transaction_count"), F.avg("props.duration_ms").alias("avg_session_duration_ms"), ) .withColumn("revenue_per_buyer", F.round(F.col("total_revenue") / F.col("unique_buyers"), 2)) .withColumn("_gold_ts", F.current_timestamp()) ) # Overwrite with replaceWhere — only touch the last 90 days, not the full table ( daily_revenue.write .format("delta") .mode("overwrite") .option("replaceWhere", "event_date >= date_sub(current_date(), 90)") .saveAsTable(f"{GOLD_DB}.daily_revenue") ) Z-Ordering and Data Skipping Z-ordering is Databricks' multi-dimensional clustering technique. It co-locates related data within the same set of Parquet files, so Spark can skip irrelevant files entirely during queries — without the overhead of strict partitioning. -- Run OPTIMIZE + ZORDER after significant writes -- This rewrites data files to cluster on the most commonly filtered columns OPTIMIZE prod.silver.events_clean ZORDER BY (user_id, event_date, event_type); -- Check how many files were skipped in your last query -- (run immediately after a SELECT with filters) SELECT operation, operationMetrics['numFilesAdded'] AS files_added, operationMetrics['numFilesRemoved'] AS files_removed, operationMetrics['numRemovedBytes'] AS bytes_removed FROM ( DESCRIBE HISTORY prod.silver.events_clean ) WHERE operation = 'OPTIMIZE' ORDER BY timestamp DESC LIMIT 5; Rule of thumb: Z-order on your top 3–4 most-filtered columns. Beyond that, the clustering benefit diminishes and OPTIMIZE runtimes grow significantly. Never Z-order on partition columns — they're already physically separated. Optimized Spark Writes Poorly tuned Spark writes are the #1 cause of small-file problems in Delta Lake. Here's a production-hardened write configuration: # ── Write Configuration Reference ──────────────────────────────────────────── SILVER_WRITE_CONFIG = { # Coalesce output files to ~128MB each (avoids small-file explosion) "spark.sql.shuffle.partitions": "200", # tune to cluster size "spark.databricks.delta.optimizeWrite.enabled": "true", # auto bin-packing "spark.databricks.delta.autoCompact.enabled": "true", # background compaction # Target file size for Auto Optimize "spark.databricks.delta.optimizeWrite.binSize": "134217728", # 128 MB in bytes # Enable deletion vectors (Databricks 12.2+) — soft-deletes without file rewrites "spark.databricks.delta.enableDeletionVectors": "true", } # Apply at the session level for the pipeline job for k, v in SILVER_WRITE_CONFIG.items(): spark.conf.set(k, v) # For partitioned tables: control output file count per partition ( silver_df .repartition(F.col("event_date")) # one task group per date partition .write .format("delta") .mode("overwrite") .option("dataChange", "true") .option("overwriteSchema", "false") # never silently change schema in prod .partitionBy("event_date") .saveAsTable(f"{SILVER_DB}.events_clean") ) Pipeline Comparison Table Here's how different ingestion patterns stack up for common production scenarios on Databricks: PatternLatencyThroughputDedup SupportBest ForAuto Loader + AppendNear-real-timeVery High❌ NoEvent logs, immutable streamsAuto Loader + MERGENear-real-timeHigh✅ YesCDC, late-arriving eventsBatch COPY INTOMinutesHigh❌ NoScheduled file ingestionStructured Streaming + foreachBatchSecondsMedium✅ YesComplex stateful pipelinesDelta Live Tables (DLT)ConfigurableHigh✅ Yes (expectations)Declarative, managed pipelinesMERGE only (batch)MinutesLow–Medium✅ YesSmall-to-medium upsert volumes DLT note: Delta Live Tables is Databricks' managed pipeline framework that handles the orchestration, monitoring, and retry logic described above declaratively. For teams starting fresh, DLT is worth evaluating before building the plumbing manually. Key Takeaways Medallion architecture separates concerns cleanly: Bronze for fidelity, Silver for correctness, Gold for consumption.Delta's transaction log is the foundation of all ACID guarantees — understanding it helps you debug merge conflicts, time travel, and VACUUM safely.Auto Loader is the right default for cloud file ingestion on Databricks — it handles exactly-once semantics and schema evolution automatically.MERGE with foreachBatch is the idiomatic pattern for deduplication and CDC in Spark Structured Streaming.Z-ORDER + Auto Optimize should be standard practice for Silver and Gold tables that receive frequent queries with selective filters.Deletion Vectors (Databricks 12.2+) make point deletes significantly cheaper — enable them for tables with GDPR or compliance requirements. References Delta Lake Documentation — Delta Lake Transaction Log — The official deep dive into how the _delta_log works internally.Databricks — Medallion ArchitectureDatabricks — Auto Loader (cloudFiles)Databricks — Delta Lake OPTIMIZE and Z-OrderingDatabricks — Auto Optimize (Optimized Writes + Auto Compaction)Databricks — Deletion VectorsDatabricks — Delta Live Tables OverviewStructured Streaming + foreachBatch — Apache Spark Docs"The Delta Lake Paper" — VLDB 2020 (Armbrust et al.)Databricks Blog — Diving Into Delta Lake: Unpacking the Transaction Log

By Jubin Soni, FBCS CORE