Big Data Resources

Incremental Jobs and Data Quality Are On a Collision Course

Big data isn’t dead; it’s just going incremental. But bad things happen when uncontrolled changes collide with incremental jobs. Reacting to changes is a losing strategy.

January 1, 2025

by Jack Vanlightly

· 5,825 Views · 1 Like

Streamlining Database Management: Running PostgreSQL in Docker Containers

You will learn how to run PostgreSQL in a Docker container with a portable, isolated, and resource-efficient database setup that is easy to manage and scalable.

December 31, 2024

by VARUNREDDY DEVIREDDY

· 6,841 Views · 3 Likes

Iceberg Catalogs: A Guide for Data Engineers

Learn about Apache Iceberg catalogs, their types, configurations, and the best-fit solutions for managing metadata and large datasets in different environments.

December 31, 2024

by Naresh Dulam

· 6,698 Views · 2 Likes

Bridging the Gap: Unlocking the Power of HDFS-Based Data Lakes With Streaming Databases

Big data technologies' quick development has highlighted the necessity of a smooth transition between real-time data analytics and batch processing systems.

December 30, 2024

by Gautam Goswami

CORE

· 3,009 Views · 1 Like

The Future of Data Lies in Transformer Models vs. Big Data Transformations

Last year witnessed the explosive rise of large models, generating global enthusiasm and making AI seem like a solution to all problems.

December 25, 2024

by William Guo

· 5,229 Views · 1 Like

Creating Scalable, Compliant Cloud Data Pipelines in SaaS through AI Integration

Cloud-based AI data pipelines provide SaaS companies with scalable, cost-effective solutions for real-time insights and efficient data management.

December 24, 2024

by Venkata Gummadi

· 23,795 Views · 3 Likes

Key Considerations for On-Premise to Cloud Data Warehouse Migration

Migrating your on-premise data warehouse to the cloud requires thorough planning. This article will help in the decision-making process by addressing critical factors.

December 18, 2024

by Maria Anurag Reddy Basani

· 6,162 Views

Delta Live Tables in Databricks: A Guide to Smarter, Faster Data Pipelines

Delta Live Tables (DLT) in Databricks streamlines data pipelines, enhancing data quality, simplifying pipeline management, and enabling real-time data processing.

December 18, 2024

by Kiran Polimetla

· 8,282 Views · 2 Likes

How to Optimize Edge Devices for AI Processing

Desiging for edge AI is crucial in today's world with on-device processing and AI applications increasing in popularity.

December 17, 2024

by Emily Newton

· 5,069 Views · 1 Like

Implementing OneLake With Medallion Architecture in Microsoft Fabric

This guide to Microsoft OneLake covers structured layers and security, enabling high-quality data optimization for analytics.

December 17, 2024

by Aravind Nuthalapati

CORE

· 25,292 Views · 3 Likes

Databricks 101: An Introductory Guide on Navigating and Optimizing This Data Powerhouse

Learn about the pivotal role of Databricks Workflows and the nuanced world of compute resources.

December 12, 2024

by Noa Shavit

· 5,514 Views · 1 Like

Data Processing With Python: Choosing Between MPI and Spark

Know the differences between MPI and Spark for big data processing and find out which framework suits your needs for parallel and distributed computing.

December 11, 2024

by Anil Kumar Moka

· 18,036 Views

Leveraging Golang for Modern ETL Pipelines

Golang enhances ETL pipelines with real-time processing, efficient concurrency, low latency, and minimal resource usage for handling large data.

December 9, 2024

by Vivek Kumar

· 16,322 Views

Snowflake vs. Databricks: How to Choose the Right Data Platform

Snowflake is ideal for data warehousing and SQL analytics, while Databricks excels at data engineering, machine learning, and real-time analytics.

December 6, 2024

by Rambabu Bandam

· 6,591 Views · 4 Likes

Deploying Kafka With Kubernetes: A Complete Guide

Learn how to effectively deploy and manage Kafka on Kubernetes with our comprehensive guide. Discover best practices, tips, and tools to optimize your streaming applications.

Updated December 5, 2024

by Alvin Lee

CORE

· 61,298 Views · 8 Likes

How to Design Event Streams, Part 2

This article covers the degree of normalization when using events created from a relational source, and how to go about denormalizing them.

December 2, 2024

by Adam Bellemare

· 4,576 Views · 2 Likes

How Open-Source BI Tools Are Transforming DevOps Pipelines

Learn how open-source BI tools transform and improve DevOps pipelines by enhancing data visibility, automation, and collaboration for streamlined workflows.

November 29, 2024

by Ayodele Johnson

· 3,139 Views · 2 Likes

Protecting Your Data Pipeline: Avoid Apache Kafka Outages With Topic and Configuration Backups

Applications that are unable to publish messages to a Kafka topic or be consumed by downstream applications are considered to be experiencing an outage.

November 29, 2024

by Gautam Goswami

CORE

· 3,419 Views · 1 Like

Elasticsearch Query and Indexing Architecture

This article breaks down Elasticsearch's core architecture by explaining how search queries and indexing requests flow through the system.

November 26, 2024

by Udbhav Prasad

· 5,992 Views · 6 Likes

Dust Actors and Large Language Models: An Application

Use Dust Java Actors to create a pipeline that automatically finds, reads, and extracts specific info from news articles based on your topic of interest.

November 25, 2024

by Alan Littleford

· 4,864 Views

The Latest Big Data Topics