DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

The Latest Big Data Topics

article thumbnail
Designing Replay-Safe CDC Pipelines With Kafka, Debezium, and Recovery Contracts
How to design CDC pipelines with Kafka, Debezium, idempotent writes, deterministic projections, replay workflows, reconciliation checks, and recovery evidence.
September 1, 2026
by Ishan Shah
· 3,428 Views · 2 Likes
article thumbnail
Real-Time Supply Chain Event Streaming With Kafka and Neo4j
A Kafka producer publishes shipment events, a Python consumer writes them into Neo4j, and a live Plotly dashboard shows network health updating as events arrive.
August 18, 2026
by Akmal Chaudhri DZone Core CORE
· 2,420 Views · 1 Like
article thumbnail
Orchestrating Small Language Models Without Losing Events or Context
Temporal and Kafka orchestrate small language models reliably through durable workflows, ordered events, idempotency, retries, replay, and context preservation.
August 13, 2026
by Akhil Madineni DZone Core CORE
· 2,278 Views · 4 Likes
article thumbnail
A Practical Pipeline for Identifying Sensitive Columns Before Test Data Masking
In this article, I will be introducing a pipeline designed to identify sensitive data columns before masking steps and improve the efficiency of the data masking process.
August 10, 2026
by Siyuan Feng
· 1,816 Views · 1 Like
article thumbnail
Supply Chain Resilience Analysis With Apache Spark and Neo4j
We model a supply chain in Neo4j using Apache Spark to load data, NetworkX to identify critical nodes, and Cypher to find alternative routes after a disruption.
August 10, 2026
by Akmal Chaudhri DZone Core CORE
· 1,850 Views · 2 Likes
article thumbnail
How We Cut PyFlink Pipeline p99 Latency from 3-5 Seconds to ~500ms
We eliminated per-record Python-side Protobuf parsing and JVM-to-Python crossings by letting Flink's native Protobuf format decode records directly into typed columns.
August 7, 2026
by Arjun Shah
· 3,906 Views · 2 Likes
article thumbnail
TensorFlow vs PyTorch: The Real Difference Isn’t Accuracy
A direct CNN benchmark on CIFAR-10 shows TensorFlow and PyTorch achieve identical accuracy (~68%). Choose TensorFlow for production and PyTorch for flexibility.
August 5, 2026
by Rakshath Naik
· 1,853 Views · 1 Like
article thumbnail
Why LLM Pipelines Fail in Production and How Temporal and Kafka Fix Them
LLM pipelines fail from retries, failures, and long-running workflows; Kafka provides reliable event streaming, while Temporal ensures durable, fault-tolerant execution.
August 5, 2026
by Akhil Madineni DZone Core CORE
· 3,101 Views · 3 Likes
article thumbnail
Why Enterprise AI Agents Fail: A Runtime Data Governance Pattern for Reliable Answers
Why enterprise AI agents fail on production data, and a runtime governance pattern using data contracts, lineage signals, and guardrails to prevent it.
August 3, 2026
by Avinash Maddineni
· 3,215 Views · 4 Likes
article thumbnail
Compliance Reporting Without Losing the Spreadsheet or the Control
Keep the spreadsheet UI for domain experts, but move validation, execution, logging, and export into a governed Java application.
July 14, 2026
by Hawk Chen DZone Core CORE
· 4,239 Views · 4 Likes
article thumbnail
AWS Glue ETL Design Principles for Production PySpark Pipelines
Learn eight AWS Glue ETL design principles for building production PySpark pipelines that are maintainable, scalable, observable, and cost-efficient.
July 14, 2026
by Janani Annur Thiruvengadam DZone Core CORE
· 3,971 Views · 3 Likes
article thumbnail
Top 10 Best Places to Prepare for Your Next Data Engineer Interview
Candidates must demonstrate strong SQL, Python, data modeling, ETL, Spark, data warehousing, and system design expertise while solving real-world data challenges.
July 10, 2026
by Rahul Han
· 3,186 Views · 2 Likes
article thumbnail
Building Production-Grade Delta Lake Pipelines With Apache Spark on Databricks
This article walks through building a modern Databricks pipeline using the Medallion Architecture, explains Delta Lake's transaction log and ACID guarantees.
July 8, 2026
by Jubin Soni, FBCS DZone Core CORE
· 4,730 Views · 3 Likes
article thumbnail
Azure Databricks for Scalable MLOps and Feature Engineering With Apache Spark, Delta Lake, and MLflow
A practical guide to feature engineering at scale with Azure Databricks, covering distributed data processing with Spark and reliable storage with Delta Lake.
July 6, 2026
by Jubin Soni, FBCS DZone Core CORE
· 6,425 Views · 2 Likes
article thumbnail
From Polling to PubSub: Building an Asynchronous OPC UA Stack in Python
The architectural design and engineering required to build a native, asynchronous OPC UA Pub/Sub (IEC 62541-14) stack in Python for the open-source opcua-asyncio library.
July 3, 2026
by Harshith Narasimhan Srivatsa
· 2,803 Views · 2 Likes
article thumbnail
Real-Time AI Feature Engineering With Spark Structured Streaming and Databricks Feature Store
How Spark Structured Streaming and the Databricks Feature Store work together to build point-in-time-correct features from Kafka events to streaming transformations.
July 2, 2026
by Jubin Soni, FBCS DZone Core CORE
· 7,195 Views · 1 Like
article thumbnail
Dead Letter Queue Patterns in Apache Flink: Handling Poison Messages Without Stopping Your Stream
A poison message can trap a Flink job in a restart loop. Use side outputs, retries, tiered DLQs, durable sinks, and replay jobs to keep the stream running.
July 2, 2026
by Rohit Muthyala
· 4,798 Views · 1 Like
article thumbnail
Apache Spark Query Optimization on Databricks: Catalyst, AQE, and Photon Engine
Spark query performance on Databricks is driven by a multi-layer optimization stack: Catalyst transforms SQL into optimized execution plans.
July 2, 2026
by Jubin Soni, FBCS DZone Core CORE
· 7,260 Views · 1 Like
article thumbnail
Fine-Tuning LLMs at Scale With Databricks MLflow and Spark
Learn how Databricks, Apache Spark, MLflow, and Hugging Face Transformers work together to create an end-to-end fine-tuning platform.
June 30, 2026
by Jubin Soni, FBCS DZone Core CORE
· 6,439 Views · 1 Like
article thumbnail
Data Pipeline Observability: Why Your AI Model Fails in Production
Your machine learning model had 95% accuracy in testing, but crashes in production. The problem isn't the model, it's your data pipeline.
June 26, 2026
by Abhilash Rao Mesala
· 2,022 Views · 2 Likes
  • Previous
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • ...
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×