Big Data Resources

Delta Lake 4.0 and Delta Kernel: What's New in the Future of Data Lakehouses

Delta Lake 4.0 pushes the lakehouse forward with flexible schemas, stronger transactions, remote, multi‑engine access, self‑optimizing performance, and AI‑ready storage.

November 3, 2025

by Sairamakrishna BuchiReddy Karri

· 4,231 Views · 2 Likes

An Open-Source ChatGPT App Generator

Create fully functional ChatGPT apps using AI — no coding needed. Build and deploy interactive UI widgets directly inside ChatGPT in just minutes.

October 31, 2025

by Thomas Hansen

CORE

· 2,249 Views · 4 Likes

When Coalesce Is Slower Than Repartition: A Spark Performance Paradox

In this article, learn why repartition() can outperform coalesce() in Apache Spark — and how Catalyst optimizer pushdown can throttle your job’s parallelism.

October 30, 2025

by Janani Annur Thiruvengadam

CORE

· 3,424 Views · 3 Likes

Debugging a Spark Driver Out of Memory (OOM) Issue With Large JSON Data Processing

This article draws on real-world debugging experience and aims to provide insights into Spark's memory management challenges.

October 29, 2025

by Raju Ansari

· 2,701 Views · 2 Likes

Unlocking Scalable Data Lakes: Building With Apache Iceberg, AWS Glue, and S3

Apache Iceberg + AWS Glue + S3 bring ACID, schema evolution, and time travel to data lakes—fixing schema drift, small files, and cost sprawl at enterprise scale.

October 28, 2025

by Vivek Venkatesan

· 4,171 Views · 1 Like

Set Up Spring Data Elasticsearch With Basic Authentication

Guide to configure SSL communication with Elasticsearch via Spring Data Elasticsearch. Additionally, the communication is secured with BASIC authentication.

October 27, 2025

by Arnošt Havelka

CORE

· 2,477 Views

Enterprise-Grade Document Intelligence: Cloud Big Data AI With YOLOv9 and Spark on AWS

Automate document analysis with YOLOv9, Apache Spark, and AWS. Boost speed, accuracy, and fraud detection across finance, healthcare, insurance, and more.

October 27, 2025

by Ram Ghadiyaram

CORE

· 2,802 Views · 4 Likes

The Dark Side of Apache Iceberg’s Data Time Travel Feature

This article talks about the hidden aspects of the Apache Iceberg Time Travel Query feature. It also highlights how to address those hidden negative aspects.

October 22, 2025

by Pravin Dwiwedi

· 3,479 Views · 1 Like

Building Scalable CRM Systems: Architecture Patterns and Data Modeling Strategies

A hands-on guide to building scalable CRM systems with the right architecture, data models, and performance and security strategies.

October 22, 2025

by Chitrapradha Ganesan

· 5,470 Views · 2 Likes

A Fresh Look at Optimizing Apache Spark Programs

Optimize Spark jobs by tuning configurations, writing efficient code (Data Frames, broadcast joins), using optimized storage, and monitoring the Spark UI and logs.

October 14, 2025

by Nataraj Mocherla

· 3,754 Views · 2 Likes

How Developers Use Synthetic Data to Stress-Test Models in Noisy Markets

Synthetic data lets quants stress-test equity strategies beyond noisy markets, preserving volatility, and building resilience before risking real capital.

October 14, 2025

by Jay Mehta

· 1,714 Views · 1 Like

Operationalizing Responsible AI: Turning Ethics Into Engineering

This article will provide a direction on how to build a reliable AI system in production by incorporating bias mitigation strategies.

October 13, 2025

by Jofia Jose Prakash

· 2,511 Views · 2 Likes

Apache Iceberg REST Catalog: The Key to Vendor-Agnostic Data Interoperability

In this article, I have demonstrated how Iceberg Data can be accessed through the Iceberg REST Catalog from Data Mesh with a simple Python application.

October 13, 2025

by Pravin Dwiwedi

· 2,996 Views · 1 Like

Introduction to Spring Data Elasticsearch 5.5

Getting started with the latest version of Spring Data Elasticsearch 5.5 and Elasticsearch 8.18 as a NoSQL database for our data storage.

October 10, 2025

by Arnošt Havelka

CORE

· 3,504 Views · 2 Likes

8 Challenges in Multimodal Training Data Creation

Creating high-quality multimodal training data is essential yet complex, involving challenges in synchronization, scalability, context capture, and tooling.

October 8, 2025

by Chirag Shivalker

· 3,487 Views

7 AWS Services Every Data Engineer Should Master

In 2025, S3, Glue, Lambda, Athena, Redshift, EMR, and Kinesis form the core AWS toolkit for building fast, reliable, and scalable data pipelines.

October 6, 2025

by Sai Mounika Yedlapalli

· 4,625 Views · 3 Likes

From Big Data to Agents: My Decade Building Systems

How a simple scraper, a few dashboards, and a lot of curiosity turned into agentic systems that actually ship value. A builder’s path.

October 3, 2025

by Nacho Corcuera

· 2,996 Views · 2 Likes

Building a Scalable and Reliable Marketing Data Stack on GCP

A resilient marketing data stack on GCP leverages BigQuery, Pub/Sub, and Dataflow to deliver real-time insights, handle schema drift, and scale analytics.

October 2, 2025

by Shafeeq Ur Rahaman

· 1,737 Views · 2 Likes

Salesforce Data Cloud: Setting Up and Using the Ingestion API

In this guide, learn to use Salesforce Data Cloud Ingestion API for real-time and bulk data ingestion to deliver accurate, personalized customer experiences.

October 2, 2025

by Ramesh Bellamkonda

· 4,764 Views · 2 Likes

Implementing Governance on Databricks Using Unity Catalog

We learn how to treat data as a product through governance in Unity Catalog, ensuring the right people, metadata about the datasets.

October 1, 2025

by Junaith Haja

· 3,189 Views · 3 Likes

The Latest Big Data Topics