Big Data Resources

Emerging Data Architectures: The Future of Data Management

This article explores the latest advancements in data architecture, focusing on frameworks and newer paradigms such as LakeDB and zero ETL architectures.

April 15, 2025

by Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

· 7,853 Views · 10 Likes

A Deep Dive into Apache Doris Indexes

Apache Doris offers various index types, each designed for specific needs. Let’s delve into their features and discover what drives their exceptional performance.

April 14, 2025

by Darren Xu

· 5,911 Views

How Doris + Hudi Turned the Impossible Into the Everyday

Want to know how this "data giant ship" navigates the waves? Follow this article to uncover the amazing story of Doris and Hudi, the "dream team" of data.

April 14, 2025

by Zen Hua

· 5,794 Views · 4 Likes

AWS S3 Strategies for Scalable and Secure Data Lake Storage

Optimize your Amazon S3 data lake with strategic bucket configurations, data layers, encryption, and lifecycle policies for security, efficiency, and cost savings.

April 8, 2025

by Digvijay Waghela

· 6,912 Views

Optimizing Data Storage With Hybrid Partitioned Tables in Oracle 19c

Learn how to implement Hybrid Partitioned Tables (HPT) to streamline data management, boost query performance, and cut storage costs.

April 8, 2025

by arvind toorpu

CORE

· 4,041 Views · 3 Likes

Building a Cost-Effective ELK Stack for Centralized Logging

This tutorial covers the setup of a budget-friendly, secure, and scalable ELK logging platform using Infrastructure as Code (IaC) with Terraform and Ansible.

April 4, 2025

by Pradeep Kumar Goud Gaddamidi

· 9,364 Views · 5 Likes

Integrating Apache Doris and Hudi for Data Querying and Migration

Learn how to integrate Apache Doris with Apache Hudi for efficient federated querying, real-time analytics, and seamless data migration in big data environments.

April 3, 2025

by li yy

· 5,390 Views · 1 Like

Bridging OT and IT: IIoT Middleware for Edge and Cloud With Kafka and Flink

Explore how IIoT middleware and data streaming tools like Kafka and Flink bridge OT and IT, enabling real-time integration and smarter industrial operations.

April 2, 2025

by Kai Wähner

CORE

· 4,506 Views · 4 Likes

Building Scalable Data Lake Using AWS

Build a secure, scalable AWS data lake with S3, Glue, and Lake Formation, and learn key components, best practices, and analytics for optimized insights.

April 1, 2025

by Digvijay Waghela

· 4,553 Views · 3 Likes

Doris vs Elasticsearch: A Comparison and Practical Cost Case Study

Apache Doris excels in analytics, SQL support, and cost efficiency, while Elasticsearch leads in text search but has higher storage costs and complexity.

March 31, 2025

by Darren Xu

· 3,600 Views · 2 Likes

Accurate Quantitative Analysis With ChatGPT and Azure AI Hub

I asked ChatGPT the question, ‘9.9 or 9.11, which is bigger?’ ChatGPT alone answered incorrectly, but with the help of Python, it provided the correct answer: 9.9.

March 27, 2025

by Sanjeev Kumar

· 3,612 Views · 1 Like

Self-Healing Data Pipelines: The Next Big Thing in Data Engineering?

Build self-healing data pipelines on GCP using AI-powered tools like GPT-4 and Python to detect and fix errors automatically.

March 27, 2025

by Naresh Erukulla

· 9,469 Views

A Comprehensive Guide to Protect Data, Models, and Users in the GenAI Era

This article provides practical checklists to help enterprises securely adopt GenAI.

March 27, 2025

by Boris Zaikin

CORE

· 5,191 Views · 3 Likes

Ensuring Data Quality With Great Expectations and Databricks

Ensure data quality in pipelines with Great Expectations. Learn to integrate with Databricks, validate data, and automate checks for reliable datasets.

March 26, 2025

by Sairamakrishna BuchiReddy Karri

· 5,220 Views · 2 Likes

A Beginner's Guide to Spark UI: Concepts and How to Use It

Understanding the fundamentals of Spark UI is essential for optimization and debugging. Although the UI is old, the information it contains is gold.

March 26, 2025

by Saransh Arora

· 3,297 Views · 3 Likes

Lakehouse: Manus? MCP? Let’s Talk About Lakehouse and AI

This is the second article in the “Lakehouse: What’s the Big Deal?” series, where I will periodically discuss Lakehouse. Your comments and discussions are welcome.

March 26, 2025

by Mingyu Chen

· 3,922 Views · 2 Likes

How to Integrate Event-Driven Ansible With Kafka

Integrate Ansible with Kafka for real-time automation: trigger playbooks via Kafka events, enhance incident response, optimize workflows, and scale seamlessly.

March 26, 2025

by Binoj Melath Nalinakshan Nair

CORE

· 5,511 Views · 12 Likes

Lakehouse: Starting With Apache Doris + S3 Tables

This is the first article in the “Lakehouse: What’s the Big Deal?” series, where I will periodically discuss Lakehouse. Your comments and discussions are welcome.

March 25, 2025

by Mingyu Chen

· 3,808 Views · 3 Likes

Using KRaft Kafka for Development and Kubernetes Deployment

Simplify Kafka with KRaft—ditch ZooKeeper, streamline configs for Docker and Kubernetes, and integrate easily with Spring Boot for development and deployment.

March 25, 2025

by Sven Loesekann

· 5,879 Views · 6 Likes

Recommender Systems Best Practices: Collaborative Filtering

Recommender systems predict preferences using feedback, tackling sparsity and cold starts with collaborative filtering, matrix factorization, and hybrid models.

March 24, 2025

by Salman Khan

CORE

· 8,868 Views · 2 Likes

The Latest Big Data Topics