Big Data Resources

Optimizing Trino Performance With Materialized Views in a Data Lake

In this article, learn how Trino materialized views boosted our Iceberg-based data lake, improving real-time query speed, reducing load, and cutting costs.

November 27, 2025

by Mikhail Povolotskii

· 1,464 Views

Revamping Real-Time Data Ingestion for Scalable Media Intelligence

AI-augmented microservices streaming architecture (Spring Boot, Kafka, Elasticsearch) to handle 8.64 million real-time media articles per day.

November 25, 2025

by Sindhukumar Rajakumaran

· 1,304 Views

Advanced Usage of Decodable in Swift: Handling Dynamic Keys

Use DynamicKey to safely decode JSON with unpredictable keys — it avoids fragile if let chains and makes your decoding logic flexible and maintainable.

November 20, 2025

by Ada Kirsch

· 1,425 Views · 1 Like

Iceberg Compaction and Fine-Grained Access Control: Performance Challenges and Solutions

Implementing fine-grained access control on Apache Iceberg can create major performance challenges. Learn how Glue, Redshift, and Athena handle FGAC at scale.

November 19, 2025

by Janani Annur Thiruvengadam

CORE

· 3,210 Views · 2 Likes

Meta Data: How Data about Your Data is Optimal for AI

Metadata enhances AI performance by providing crucial context for models. Learn key benefits, implementation strategies, and real-world examples for smarter AI systems.

November 19, 2025

by Kevin Vu

· 1,377 Views

Databricks vs Snowflake: Complete Architecture Mapping for Enterprise AI and Big Data

This guide maps core data, big data, and AI/ML concepts between Databricks and Snowflake, with examples, diagrams, and a framework for choosing or combining the two.

November 13, 2025

by Ram Ghadiyaram

CORE

· 7,156 Views · 7 Likes

Event-Driven Architecture Patterns: Real-World Lessons From IoT Development

Real-world lessons on building event-driven systems for IoT and microservices — and model optimization with practical code examples.

November 10, 2025

by Bhavna Hirani

· 7,820 Views · 6 Likes

Hudi vs. Delta vs. Iceberg: How to Choose the Right Lakehouse Table Format

Hudi excels at real-time upserts, Delta handles ACID workloads, and Iceberg supports large-scale analytics with flexible schemas.

November 10, 2025

by harshraj bhoite

· 3,593 Views · 1 Like

Regression Analysis for Time Series Data: Models and Applications

Ready to use regression analysis for time series data? Explore how this method works in practice to effectively predict future outcomes and drive growth.

November 10, 2025

by Oleksandr Stefanovskyi

· 1,570 Views · 1 Like

Master Production-Ready Big Data, Apache Spark Jobs in Databricks and Beyond: An Expert Guide

Refine Apache Spark performance in Databricks with strategies. Includes expert insights, PySpark examples, and diagrams for efficient data processing.

November 6, 2025

by Ram Ghadiyaram

CORE

· 1,872 Views · 1 Like

5 Critical Databricks Performance Hacks That Most Engineers Miss (100x Faster Queries)

In this article, learn to boost Databricks' performance with six proven optimization strategies for UDFs, AQE, Delta Lake, broadcasts, and Photon acceleration.

November 3, 2025

by Ram Ghadiyaram

CORE

· 3,363 Views · 4 Likes

Delta Lake 4.0 and Delta Kernel: What's New in the Future of Data Lakehouses

Delta Lake 4.0 pushes the lakehouse forward with flexible schemas, stronger transactions, remote, multi‑engine access, self‑optimizing performance, and AI‑ready storage.

November 3, 2025

by Sairamakrishna BuchiReddy Karri

· 3,404 Views · 2 Likes

An Open-Source ChatGPT App Generator

Create fully functional ChatGPT apps using AI — no coding needed. Build and deploy interactive UI widgets directly inside ChatGPT in just minutes.

October 31, 2025

by Thomas Hansen

CORE

· 1,995 Views · 4 Likes

When Coalesce Is Slower Than Repartition: A Spark Performance Paradox

In this article, learn why repartition() can outperform coalesce() in Apache Spark — and how Catalyst optimizer pushdown can throttle your job’s parallelism.

October 30, 2025

by Janani Annur Thiruvengadam

CORE

· 2,727 Views · 3 Likes

Debugging a Spark Driver Out of Memory (OOM) Issue With Large JSON Data Processing

This article draws on real-world debugging experience and aims to provide insights into Spark's memory management challenges.

October 29, 2025

by Raju Ansari

· 2,347 Views · 2 Likes

Unlocking Scalable Data Lakes: Building With Apache Iceberg, AWS Glue, and S3

Apache Iceberg + AWS Glue + S3 bring ACID, schema evolution, and time travel to data lakes—fixing schema drift, small files, and cost sprawl at enterprise scale.

October 28, 2025

by Vivek Venkatesan

· 3,401 Views · 1 Like

Set Up Spring Data Elasticsearch With Basic Authentication

Guide to configure SSL communication with Elasticsearch via Spring Data Elasticsearch. Additionally, the communication is secured with BASIC authentication.

October 27, 2025

by Arnošt Havelka

CORE

· 2,107 Views

Enterprise-Grade Document Intelligence: Cloud Big Data AI With YOLOv9 and Spark on AWS

Automate document analysis with YOLOv9, Apache Spark, and AWS. Boost speed, accuracy, and fraud detection across finance, healthcare, insurance, and more.

October 27, 2025

by Ram Ghadiyaram

CORE

· 2,511 Views · 4 Likes

The Dark Side of Apache Iceberg’s Data Time Travel Feature

This article talks about the hidden aspects of the Apache Iceberg Time Travel Query feature. It also highlights how to address those hidden negative aspects.

October 22, 2025

by Pravin Dwiwedi

· 2,823 Views · 1 Like

Building Scalable CRM Systems: Architecture Patterns and Data Modeling Strategies

A hands-on guide to building scalable CRM systems with the right architecture, data models, and performance and security strategies.

October 22, 2025

by Chitrapradha Ganesan

· 4,014 Views · 2 Likes

The Latest Big Data Topics