DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

The Latest Big Data Topics

article thumbnail
Implementing Governance on Databricks Using Unity Catalog
We learn how to treat data as a product through governance in Unity Catalog, ensuring the right people, metadata about the datasets.
October 1, 2025
by Junaith Haja
· 2,777 Views · 3 Likes
article thumbnail
Master Advanced Error-Handling to Make PySpark Pipelines Production-Ready
PySpark jobs often fail because of bad data, network issues, or logic errors. Sometimes, after hours of processing. Learn how to make your Spark pipelines more reliable.
September 30, 2025
by Ram Ghadiyaram DZone Core CORE
· 4,793 Views · 6 Likes
article thumbnail
Complex Data Tasks Are Now One-Liners With AI in Databricks SQL
In this guide, learn how to simplify data tasks with AI in Databricks SQL — summarize, translate, analyze sentiment, and mask PII with one-liner queries.
September 26, 2025
by Junaith Haja
· 3,533 Views · 2 Likes
article thumbnail
AWS Glue Crawlers: Common Pitfalls, Schema Challenges, and Best Practices
Learn key challenges and best practices for using AWS Glue crawlers, from handling CSV schema issues to schema evolution, partitions, and ETL jobs.
September 25, 2025
by Saradha Nagarajan
· 3,939 Views
article thumbnail
LLMs at the Edge: Decentralized Power and Control
Deploying LLMs at the edge decentralizes intelligence, enhances privacy, reduces latency, increases autonomy, and empowers local control.
September 23, 2025
by Bhanuprakash Madupati
· 22,938 Views · 4 Likes
article thumbnail
Azure IOT Cloud-to-Device Communication Methods
Learning and choosing the correct cloud-to-device communication method to send a message to the device using the Azure IoT Hub to build an effective IoT system.
September 22, 2025
by Anup Rao
· 2,897 Views
article thumbnail
The Real-time Data Transfer Magic of Doris Kafka Connector's "Data Package": Part 1
One Man Stands Guard, and Ten Thousand Cannot Pass! Learn all about real-time data import, transformation, and error handling with Doris Kafka Connector.
September 12, 2025
by Michael Hayden
· 2,889 Views · 1 Like
article thumbnail
Azure VM Instance Types and Their Roles in Different Distributed Software Systems
Azure provides various VM instance types optimized for compute, memory, storage, or GPU needs, such as Databricks, Snowflake, AKS, Synapse, and Azure Functions.
September 11, 2025
by Srinivasarao Rayankula
· 19,535 Views · 3 Likes
article thumbnail
AI on the Fly: Real-Time Data Streaming From Apache Kafka to Live Dashboards
Real-time data streaming plays a key role for AI models as it allows them to handle and respond to data as it comes in, instead of just using old fixed datasets.
September 11, 2025
by Gautam Goswami DZone Core CORE
· 6,975 Views
article thumbnail
From HTTP to Kafka: A Custom Source Connector
Learn how to implement a custom Kafka Connect HTTP source connector to integrate with HTTP endpoints, covering connector configuration, deployment and usage.
September 10, 2025
by Ion Pascari
· 4,661 Views · 5 Likes
article thumbnail
API Design First: AsyncAPI in .Net
AsyncAPI isn't as widely adopted as OpenAPI Spec, however, it's getting significant attention in the world of everything async and distributed.
September 8, 2025
by Shashi Kumar
· 2,234 Views · 2 Likes
article thumbnail
The Role of Data Governance in Data Strategy: Part 4
Explore the critical role of data retention in governance: reduce costs, mitigate legal and cybersecurity risks, and ensure compliance with clear policies.
September 5, 2025
by Satish Gaddipati
· 2,173 Views · 2 Likes
article thumbnail
Observability for the Invisible: Tracing Message Drops in Kafka Pipelines
Kafka lag lies. Use Fluent Bit, OpenTelemetry, DLQs, and trace IDs to expose missing messages and harden observability in event-driven pipelines.
September 3, 2025
by Prakash Wagle
· 2,993 Views · 2 Likes
article thumbnail
Simple Efficient Spring/Kafka Datastreams
Datastreams from Souce via Kafka to Sink done simple, easy to code, efficient to run. Deployed in Kubernetes and written in Java and Kotlin.
September 3, 2025
by Sven Loesekann
· 6,520 Views · 7 Likes
article thumbnail
Understanding Apache Spark Join Types
Three join types in Spark data frame SQL operations are crucial for the performance of big data Apache Spark applications.
September 3, 2025
by Ram Ghadiyaram DZone Core CORE
· 3,955 Views · 5 Likes
article thumbnail
File Systems <> Database: Full Circle
The start of the computer storage era was a file-based system, which evolved into databases; However, data advancement made file systems relevant again.
September 3, 2025
by BHUSHAN FADNIS
· 1,649 Views
article thumbnail
Keep Your Search Cluster Fit: Essential Health Checks to Keep Elasticsearch Healthy
A search cluster in top notch state requires frequent monitoring for health stats. Let's look at some health checks to always keep your ES cluster fit.
August 29, 2025
by Govind Singh Rawat
· 2,483 Views · 2 Likes
article thumbnail
Designing Scalable Ingestion and Access Layers for Policy and Enforcement Data
Build a scalable, low-latency architecture for ingesting and accessing policy and enforcement data using Apache Spark and in-memory data grids.
August 28, 2025
by Pankaj Taneja
· 1,392 Views · 1 Like
article thumbnail
How Healthy Is Your Data in the Age of AI? An In-Depth Checklist to Assess Data Accuracy, Governance, and AI Readiness
This guide provides a complete checklist to assess, monitor, and improve data quality for AI success, ensuring accuracy, compliance, and long-term reliability.
August 28, 2025
by Sukanya Konatam
· 2,546 Views · 2 Likes
article thumbnail
Implementing Scalable IoT Architectures on Azure
Microsoft’s Azure IoT platform has emerged as a leading choice, powering innovative solutions across industries — from manufacturing floors to smart buildings.
August 27, 2025
by Bhimraj Ghadge
· 3,175 Views · 4 Likes
  • Previous
  • ...
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • ...
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×