Big Data Resources

Control Your Services With OTEL, Jaeger, and Prometheus

In this article, we will combine OTEL, Jaeger, and Prometheus for faster, centralized observability and quick troubleshooting in distributed systems.

February 7, 2025

by Ilia Ivankin

· 5,118 Views · 3 Likes

Best Practices for Scaling Kafka-Based Workloads

Kafka is a famous technology with a lot of great features and capabilities. This article explains Kafka producer and consumer configurations best practices.

February 6, 2025

by Narendra Lakshmana gowda

· 5,890 Views · 1 Like

The Role of DQ Checks in Data Pipelines

Why are DQ checks critical for every data pipeline, and what are some of the different types of DQ alerts you can set up to enhance the reliability of your pipeline?

February 5, 2025

by Ajay Krishnan Prabhakaran

· 5,455 Views

Mastering the Transition: From Amazon EMR to EMR on EKS

Optimize your data processing by seamlessly transitioning from Amazon EMR to EMR on EKS. Discover best practices and tips for a smooth migration.

February 5, 2025

by Satrajit Basu

CORE

· 6,994 Views · 4 Likes

AI Regulation in the U.S.: Navigating Post-EO 14110

The Trump administration revoked EO 14110, shifting the U.S. toward a market-driven AI strategy to spur innovation and investment.

February 5, 2025

by Frederic Jacquet

CORE

· 4,525 Views · 5 Likes

Teradata Performance and Skew Prevention Tips

Use unique identifiers such as Teradata Primary Index to evenly distribute data across AMPs to improve database performance by 70%.

February 4, 2025

by Sudheer Kumar Lagisetty

· 4,282 Views · 10 Likes

Data Governance Essentials: Policies and Procedures (Part 6)

Learn how data quality, policies, and procedures strengthen data governance by ensuring accuracy, compliance, and security for better decision-making.

February 4, 2025

by Sukanya Konatam

· 5,473 Views · 4 Likes

All You Need to Know About Apache Spark

Apache Spark is a fast, open-source cluster computing framework for big data, supporting ML, SQL, and streaming. It’s scalable, efficient, and widely used.

February 3, 2025

by Abhishek Trehan

· 4,185 Views · 2 Likes

How Apache Flink and Apache Paimon Influence Data Streaming

Apache Flink is a crucial component of Apache Paimon since it offers the real-time processing power that enhances Paimon's strong consistency and storage features.

January 28, 2025

by Gautam Goswami

CORE

· 5,804 Views · 4 Likes

CubeFS: High-Performance Storage for Cloud-Native Apps

CubeFS is a CNCF-graduated, cloud-native distributed file system delivering scalable performance, fault tolerance, and smooth Kubernetes integration.

January 24, 2025

by Sai Sandeep Ogety

CORE

· 7,787 Views · 8 Likes

Understanding HyperLogLog for Estimating Cardinality

Learn about the probabilistic algorithm designed to estimate the cardinality of a dataset with both high accuracy and low memory usage.

January 23, 2025

by Bhala Ranganathan

CORE

· 5,409 Views · 2 Likes

OPC-UA and MQTT: A Guide to Protocols, Python Implementations

Explore two essential IoT protocols: OPC-UA for secure and structured industrial device communication and MQTT, a lightweight, real-time protocol for telemetry.

January 21, 2025

by Nikhil Makhija

· 5,655 Views · 2 Likes

Seamless Transition from Elasticsearch to OpenSearch

A step-by-step guide for migrating from Elasticsearch to OpenSearch, ensuring compatibility, performance, and cost-efficiency.

January 20, 2025

by Nilesh Jain

· 6,819 Views · 3 Likes

Building a Reactive Event-Driven App With Dead Letter Queue

Learn how to build fault-tolerant, reactive event-driven applications using Spring WebFlux, Apache Kafka, and Dead Letter Queue to handle data loss efficiently.

January 20, 2025

by Sulakshana Singh

· 4,835 Views · 4 Likes

Dark Data: Recovering the Lost Opportunities

Dark data is the vast amounts of unstructured information collected by organizations that often go unused. It includes emails, customer interactions, sensor data, etc.

January 17, 2025

by Vijay Singh Khatri

CORE

· 3,562 Views

Efficient Long-Term Trend Analysis in Presto Using Datelists

To make long-term trend analysis easier, we can leverage datelists, where we store each metric value corresponding to a date in an array in a sequential manner.

January 15, 2025

by Ajay Krishnan Prabhakaran

· 4,147 Views

Kafka vs NATS: A Comparison for Message Processing

Kafka and NATS are both popular tools for message processing. This article provides a comparison between Kafka and NATS.

January 15, 2025

by Josson Paul Kalapparambath

· 7,545 Views · 2 Likes

Data-First IDP: Driving AI Innovation in Developer Platforms

A Data-First IDP integrates governance, traceability, and quality into workflows, transforming how data is managed, enabling scalable, AI-ready ecosystems.

January 15, 2025

by Paul Gale

· 6,517 Views · 2 Likes

Custom SCD2 Implementation Using PySpark

Learn to implement Slowly Changing Dimension Type 2 (SCD2) in a data warehouse for tracking historical data, ensuring data integrity, and enabling scalability.

January 13, 2025

by Abhishek Trehan

· 4,417 Views · 1 Like

Process Mining Key Elements

This article introduces process mining, explaining its key elements and practical applications for discovering and analyzing workflows using event data.

January 13, 2025

by Puneet Malhotra

· 4,701 Views · 5 Likes

The Latest Big Data Topics