Big Data Resources

Ensuring Data Quality With Great Expectations and Databricks

Ensure data quality in pipelines with Great Expectations. Learn to integrate with Databricks, validate data, and automate checks for reliable datasets.

March 26, 2025

by Sairamakrishna BuchiReddy Karri

· 5,611 Views · 2 Likes

A Beginner's Guide to Spark UI: Concepts and How to Use It

Understanding the fundamentals of Spark UI is essential for optimization and debugging. Although the UI is old, the information it contains is gold.

March 26, 2025

by Saransh Arora

· 3,563 Views · 3 Likes

Lakehouse: Manus? MCP? Let’s Talk About Lakehouse and AI

This is the second article in the “Lakehouse: What’s the Big Deal?” series, where I will periodically discuss Lakehouse. Your comments and discussions are welcome.

March 26, 2025

by Mingyu Chen

· 4,092 Views · 2 Likes

How to Integrate Event-Driven Ansible With Kafka

Integrate Ansible with Kafka for real-time automation: trigger playbooks via Kafka events, enhance incident response, optimize workflows, and scale seamlessly.

March 26, 2025

by Binoj Melath Nalinakshan Nair

CORE

· 5,769 Views · 12 Likes

Lakehouse: Starting With Apache Doris + S3 Tables

This is the first article in the “Lakehouse: What’s the Big Deal?” series, where I will periodically discuss Lakehouse. Your comments and discussions are welcome.

March 25, 2025

by Mingyu Chen

· 4,014 Views · 3 Likes

Using KRaft Kafka for Development and Kubernetes Deployment

Simplify Kafka with KRaft—ditch ZooKeeper, streamline configs for Docker and Kubernetes, and integrate easily with Spring Boot for development and deployment.

March 25, 2025

by Sven Loesekann

· 6,297 Views · 6 Likes

Recommender Systems Best Practices: Collaborative Filtering

Recommender systems predict preferences using feedback, tackling sparsity and cold starts with collaborative filtering, matrix factorization, and hybrid models.

March 24, 2025

by Salman Khan

CORE

· 9,274 Views · 2 Likes

When Doris Meets Iceberg: A Data Engineer's Redemption

Apache Doris and Iceberg are redefining the way data lakes work. It's not just a simple 1+1=2; it brings a qualitative leap!

March 21, 2025

by Zen Hua

· 3,748 Views · 2 Likes

Data Lake vs. Data Warehouse vs. Data Lakehouse

The pros and cons of the legacy data warehouse, the more recent data lake, and contemporary data lakehouse architectures.

Updated March 21, 2025

by Noa Shavit

· 7,988 Views · 6 Likes

Patch Management in the Age of IoT: Challenges and Solutions

IoT patch management tackles risks via automation, lightweight patches, and centralized tools, ensuring security despite device variety and resource limits.

March 20, 2025

by andrew vereen

· 4,984 Views · 1 Like

Comparing DuckDB, Snowflake, and Databricks

In this article, we do an in-depth comparison of DuckDB, Snowflake, and Databricks to help you find the best data processing platform for your organization.

March 19, 2025

by Noa Shavit

· 4,762 Views · 1 Like

Building a Distributed Multi-Language Data Science System

The starter consists of hexagonal microservices (MERN monorepo, Spring Boot Camel, Flask), Gateway, Eureka, that communicate via REST, GraphQL, gRPC, and AMQP.

March 18, 2025

by Alexander Eleseev

CORE

· 4,172 Views · 3 Likes

Bridging Cloud and On-Premises Log Processing

This article emphasizes on the feasibility of ingesting logs from public cloud platforms into on-premises services by highlighting key points.

March 18, 2025

by Avinash Ibbandi

· 3,698 Views · 3 Likes

Best Practices for Data Warehouses in Microsoft Fabric

Leverage Microsoft Fabric for unified data warehousing; follow best practices for schema, ingestion, transformation, security, optimization, and continuous monitoring.

March 17, 2025

by Aravind Nuthalapati

CORE

· 25,229 Views · 4 Likes

ETL With Large Language Models: AI-Powered Data Processing

LLMs transform ETL with schema-less extraction, adaptive transformations, and multi-modal support, enabling scalable, efficient, and accessible data workflows.

March 10, 2025

by Suri (thammuio)

CORE

· 31,846 Views

IoT Communication Protocols for Efficient Device Integration

In this blog, we will explore the most widely used IoT protocols and the importance of selecting the right protocol for efficient device integration.

March 5, 2025

by Richard Kaplan

· 6,511 Views · 1 Like

Harnessing Real-Time Insights With Streaming SQL on Kafka

Streaming SQL enables real-time data processing and analytics on the fly, seamlessly querying Kafka topics for actionable insights without complex coding.

March 5, 2025

by Rama Krishna Panguluri

· 6,531 Views

AI Agents for Data Warehousing

AI agents are revolutionizing data warehousing by enhancing efficiency, accuracy, and automation across various aspects of data management today.

March 4, 2025

by Ajay Tanikonda

· 4,676 Views · 1 Like

Materialized Views in Data Stream Processing With RisingWave

Materialized views enhance data streaming by improving incremental computation, enabling efficient retrieval and calculation of aggregated or pre-processed data.

March 3, 2025

by Gautam Goswami

CORE

· 3,705 Views · 2 Likes

Modern Data Processing Libraries: Beyond Pandas

In this article, we explore the alternatives to pandas for data processing and data analysis. We'll compare and contrast based on performance.

March 3, 2025

by Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

· 6,680 Views · 6 Likes

The Latest Big Data Topics