Big Data Resources

Modern ETL Architecture: dbt on Snowflake With Airflow

Build a scalable ETL pipeline with dbt, Snowflake, and Airflow, and address data engineering challenges with modular architecture, CI/CD, and best practices.

February 27, 2025

by Digvijay Waghela

· 6,342 Views · 2 Likes

Top Methods to Improve ETL Performance Using SSIS

Improve ETL performance in SSIS with parallel extraction, optimized transformations, and proper configuration of concurrency, batch sizes, and data types.

February 27, 2025

by DZone Editorial

· 5,735 Views · 1 Like

Cloud-Driven Analytics Solution Strategy in Healthcare

Detailed insights into compute resource management, cluster optimization, storage efficiency, and cost governance in cloud-based environments.

February 27, 2025

by Abrar Ahmed Syed

· 4,889 Views · 5 Likes

How to Scale Elasticsearch to Solve Your Scalability Issues

Scaling Elasticsearch requires balancing sharding, query performance, and memory tuning for optimal efficiency in high-traffic, real-time applications.

February 26, 2025

by Vivek Kumar

· 8,258 Views · 3 Likes

Spark Job Optimization

Spark jobs can be optimized to maximize resource utilization in a cluster, improving performance and reducing costs for large-scale data processing.

February 25, 2025

by Chandra Shekar r Chekuri

· 3,363 Views · 1 Like

The Future of Data Lakehouses: Apache Iceberg Explained

This blog post is the first in a three-part series exploring Apache Iceberg and its role in modern data architectures and the emergence of data lakehouses.

February 25, 2025

by Fawaz Ghali, PhD

CORE

· 4,167 Views · 5 Likes

The Hidden Cost of Dirty Data in AI Development

Dirty data weakens AI, increases costs, introduces bias, and causes compliance risks. Strong data governance ensures reliable AI outcomes.

February 25, 2025

by Ilya Dudkin

CORE

· 4,111 Views · 3 Likes

Deduplication of Videos Using Fingerprints, CLIP Embeddings

Video deduplication optimizes storage by removing duplicates using techniques like segmentation, embeddings, and clustering to manage massive datasets efficiently.

February 21, 2025

by Praneeth Reddy Vatti

· 7,012 Views · 5 Likes

Scaling Image Deduplication: Finding Needles in a Haystack

Learn to efficiently deduplicate 100M+ images using distributed architectures, embeddings, FAISS for ANN search, and clustering to ensure accurate results.

February 20, 2025

by Praneeth Reddy Vatti

· 6,039 Views · 3 Likes

Data Pattern Automation With AI and Machine Learning

Pattern recognition and AI improve data workflows, automate insights, and drive efficiency in business processes across industries.

February 19, 2025

by Sandip Gami

· 4,405 Views · 2 Likes

ETL Generation Using GenAI

Learn about how GenAI automates ETL pipelines, generates code, adapts to schema changes, and improves data processes with speed, efficiency, and precision.

February 14, 2025

by Ramesh Daddala

· 6,504 Views · 2 Likes

Loading XML into MongoDB

Learn how to export XML data to MongoDB using SmartXML ETL tools, simplifying the process and ensuring efficient data handling and storage.

February 12, 2025

by Luca Sanders

· 6,871 Views · 1 Like

The Right ETL Architecture for Multi-Source Data Integration

Dedicated ETL pipelines are easy to set up but hard to scale, while common pipelines offer efficiency at the cost of complexity. Know which one to choose.

February 12, 2025

by Murat Balkan

CORE

· 6,194 Views · 1 Like

SQL as the Backbone of Big Data and AI Powerhouses

SQL powers Big Data and AI with tools like BigQuery, remaining a cornerstone of data-driven innovation through its simplicity and adaptability.

February 11, 2025

by Medha Gupta

· 3,604 Views · 1 Like

Relational DB Migration to S3 Data Lake Via AWS DMS, Part I

This article discusses the challenges faced during relational database migration to AWS using DMS, including source data, logging, and network bandwidth issues.

February 7, 2025

by Vijay Bhosale

· 5,375 Views · 3 Likes

Control Your Services With OTEL, Jaeger, and Prometheus

In this article, we will combine OTEL, Jaeger, and Prometheus for faster, centralized observability and quick troubleshooting in distributed systems.

February 7, 2025

by Ilia Ivankin

· 5,272 Views · 3 Likes

Best Practices for Scaling Kafka-Based Workloads

Kafka is a famous technology with a lot of great features and capabilities. This article explains Kafka producer and consumer configurations best practices.

February 6, 2025

by Narendra Lakshmana gowda

· 6,014 Views · 1 Like

The Role of DQ Checks in Data Pipelines

Why are DQ checks critical for every data pipeline, and what are some of the different types of DQ alerts you can set up to enhance the reliability of your pipeline?

February 5, 2025

by Ajay Krishnan Prabhakaran

· 5,532 Views