Big Data Resources

Mastering the Transition: From Amazon EMR to EMR on EKS

Optimize your data processing by seamlessly transitioning from Amazon EMR to EMR on EKS. Discover best practices and tips for a smooth migration.

February 5, 2025

by Satrajit Basu

CORE

· 7,269 Views · 4 Likes

AI Regulation in the U.S.: Navigating Post-EO 14110

The Trump administration revoked EO 14110, shifting the U.S. toward a market-driven AI strategy to spur innovation and investment.

February 5, 2025

by Frederic Jacquet

CORE

· 4,613 Views · 5 Likes

Teradata Performance and Skew Prevention Tips

Use unique identifiers such as Teradata Primary Index to evenly distribute data across AMPs to improve database performance by 70%.

February 4, 2025

by Sudheer Kumar Lagisetty

· 4,427 Views · 10 Likes

Data Governance Essentials: Policies and Procedures (Part 6)

Learn how data quality, policies, and procedures strengthen data governance by ensuring accuracy, compliance, and security for better decision-making.

February 4, 2025

by Sukanya Konatam

· 5,754 Views · 4 Likes

All You Need to Know About Apache Spark

Apache Spark is a fast, open-source cluster computing framework for big data, supporting ML, SQL, and streaming. It’s scalable, efficient, and widely used.

February 3, 2025

by Abhishek Trehan

· 4,308 Views · 2 Likes

How Apache Flink and Apache Paimon Influence Data Streaming

Apache Flink is a crucial component of Apache Paimon since it offers the real-time processing power that enhances Paimon's strong consistency and storage features.

January 28, 2025

by Gautam Goswami

CORE

· 5,899 Views · 4 Likes

CubeFS: High-Performance Storage for Cloud-Native Apps

CubeFS is a CNCF-graduated, cloud-native distributed file system delivering scalable performance, fault tolerance, and smooth Kubernetes integration.

January 24, 2025

by Sai Sandeep Ogety

CORE

· 8,017 Views · 8 Likes

Understanding HyperLogLog for Estimating Cardinality

Learn about the probabilistic algorithm designed to estimate the cardinality of a dataset with both high accuracy and low memory usage.

January 23, 2025

by Bhala Ranganathan

CORE

· 5,595 Views · 2 Likes

OPC-UA and MQTT: A Guide to Protocols, Python Implementations

Explore two essential IoT protocols: OPC-UA for secure and structured industrial device communication and MQTT, a lightweight, real-time protocol for telemetry.

January 21, 2025

by Nikhil Makhija

· 5,969 Views · 2 Likes

Seamless Transition from Elasticsearch to OpenSearch

A step-by-step guide for migrating from Elasticsearch to OpenSearch, ensuring compatibility, performance, and cost-efficiency.

January 20, 2025

by Nilesh Jain

· 7,053 Views · 3 Likes

Building a Reactive Event-Driven App With Dead Letter Queue

Learn how to build fault-tolerant, reactive event-driven applications using Spring WebFlux, Apache Kafka, and Dead Letter Queue to handle data loss efficiently.

January 20, 2025

by Sulakshana Singh

· 5,027 Views · 4 Likes

Dark Data: Recovering the Lost Opportunities

Dark data is the vast amounts of unstructured information collected by organizations that often go unused. It includes emails, customer interactions, sensor data, etc.

January 17, 2025

by Vijay Singh Khatri

CORE

· 3,652 Views

Efficient Long-Term Trend Analysis in Presto Using Datelists

To make long-term trend analysis easier, we can leverage datelists, where we store each metric value corresponding to a date in an array in a sequential manner.

January 15, 2025

by Ajay Krishnan Prabhakaran

· 4,270 Views

Kafka vs NATS: A Comparison for Message Processing

Kafka and NATS are both popular tools for message processing. This article provides a comparison between Kafka and NATS.

January 15, 2025

by Josson Paul Kalapparambath

· 7,932 Views · 2 Likes

Data-First IDP: Driving AI Innovation in Developer Platforms

A Data-First IDP integrates governance, traceability, and quality into workflows, transforming how data is managed, enabling scalable, AI-ready ecosystems.

January 15, 2025

by Paul Gale

· 6,619 Views · 2 Likes

Custom SCD2 Implementation Using PySpark

Learn to implement Slowly Changing Dimension Type 2 (SCD2) in a data warehouse for tracking historical data, ensuring data integrity, and enabling scalability.

January 13, 2025

by Abhishek Trehan

· 4,631 Views · 1 Like

Process Mining Key Elements

This article introduces process mining, explaining its key elements and practical applications for discovering and analyzing workflows using event data.

January 13, 2025

by Puneet Malhotra

· 4,846 Views · 5 Likes

Revolutionizing Catalog Management for Data Lakehouse With Polaris Catalog

Achieve cross-query-engine interoperability and zero-data-copy architecture with Polaris Catalog, enhance data management, and streamline workflows.

January 10, 2025

by Dipankar Saha

· 4,241 Views · 1 Like

Quantum Machine Learning for Large-Scale Data-Intensive Applications

This article examines how QML can harness the principles of quantum mechanics to achieve significant computational advantages over classical approaches.

January 9, 2025

by Manasi Sharma

· 7,937 Views · 1 Like

AI Governance: Building Ethical and Transparent Systems for the Future

This article takes a deep dive into AI governance, including insights surrounding its challenges, frameworks, standards, and more.

January 9, 2025

by Sukanya Konatam

· 8,516 Views · 3 Likes

The Latest Big Data Topics