Big Data Resources

Databricks 101: An Introductory Guide on Navigating and Optimizing This Data Powerhouse

Learn about the pivotal role of Databricks Workflows and the nuanced world of compute resources.

December 12, 2024

by Noa Shavit

· 5,557 Views · 1 Like

Data Processing With Python: Choosing Between MPI and Spark

Know the differences between MPI and Spark for big data processing and find out which framework suits your needs for parallel and distributed computing.

December 11, 2024

by Anil Kumar Moka

· 18,126 Views

Leveraging Golang for Modern ETL Pipelines

Golang enhances ETL pipelines with real-time processing, efficient concurrency, low latency, and minimal resource usage for handling large data.

December 9, 2024

by Vivek Kumar

· 16,447 Views

Snowflake vs. Databricks: How to Choose the Right Data Platform

Snowflake is ideal for data warehousing and SQL analytics, while Databricks excels at data engineering, machine learning, and real-time analytics.

December 6, 2024

by Rambabu Bandam

· 6,654 Views · 4 Likes

Deploying Kafka With Kubernetes: A Complete Guide

Learn how to effectively deploy and manage Kafka on Kubernetes with our comprehensive guide. Discover best practices, tips, and tools to optimize your streaming applications.

Updated December 5, 2024

by Alvin Lee

CORE

· 61,445 Views · 8 Likes

How to Design Event Streams, Part 2

This article covers the degree of normalization when using events created from a relational source, and how to go about denormalizing them.

December 2, 2024

by Adam Bellemare

· 4,639 Views · 2 Likes

How Open-Source BI Tools Are Transforming DevOps Pipelines

Learn how open-source BI tools transform and improve DevOps pipelines by enhancing data visibility, automation, and collaboration for streamlined workflows.

November 29, 2024

by Ayodele Johnson

· 3,200 Views · 2 Likes

Protecting Your Data Pipeline: Avoid Apache Kafka Outages With Topic and Configuration Backups

Applications that are unable to publish messages to a Kafka topic or be consumed by downstream applications are considered to be experiencing an outage.

November 29, 2024

by Gautam Goswami

CORE

· 3,494 Views · 1 Like

Elasticsearch Query and Indexing Architecture

This article breaks down Elasticsearch's core architecture by explaining how search queries and indexing requests flow through the system.

November 26, 2024

by Udbhav Prasad

· 6,077 Views · 6 Likes

Dust Actors and Large Language Models: An Application

Use Dust Java Actors to create a pipeline that automatically finds, reads, and extracts specific info from news articles based on your topic of interest.

November 25, 2024

by Alan Littleford

· 4,923 Views

The Benefits of Using Cloud for Big Data Processing

Let's discuss the multiple advantages of using cloud computing for big data processing, from scalability to cost-effectiveness and enhanced collaboration.

November 25, 2024

by Job Ready Program

· 3,313 Views · 2 Likes

Deployment Strategies for Apache Kafka Cluster Types

Multiple Kafka clusters enable hybrid integration, aggregation, migration, and disaster recovery across edge, data center, and multi-cloud environments.

November 19, 2024

by Kai Wähner

CORE

· 2,377 Views · 2 Likes

Data Architectures in the AI Era: Key Strategies and Insights

Data architecture is evolving rapidly due to the rise of GenAI, requiring companies to move away from data silos toward integrated data fabrics and data meshes.

November 14, 2024

by Suri (thammuio)

CORE

· 24,491 Views · 5 Likes

The Science Behind Durability: Write-Ahead Logging Explained

For any persistence store system, guaranteeing durability of data being managed is of prime importance. Read on to know how write ahead logging ensures durability.

November 14, 2024

by Ammar Husain

CORE

· 2,105 Views · 2 Likes

Apache Iceberg: The Open Table Format for Lakehouses and Data Streaming

This article explores the table format wars of Apache Iceberg, Hudi, Delta Lake and XTable; and the product strategy of Snowflake, Databricks, Confluent, AWS, and Google.

November 12, 2024

by Kai Wähner

CORE

· 5,177 Views · 2 Likes

iRODS: An Open-Source Approach to Data Management in Large-Scale Research Environments

Discover iRODS, the open-source data management platform revolutionizing how enterprises handle large-scale datasets with policy-based automation and federation.

November 12, 2024

by Tom Smith

CORE

· 2,097 Views · 2 Likes

A Guide to Building Data Intelligence Systems: Strategic Practices to Building Robust, Ethical, and AI-Driven Data Structures

The foundation of data intelligence systems centers around transparency, governance, and the ethical and responsible exploitation of cutting-edge technologies, particularly GenAI.

November 8, 2024

by Frederic Jacquet

CORE

· 3,172 Views · 2 Likes

The Data (Pipeline) Movement: A Guide to Real-Time Data Streaming and Future Proofing Through AI Automation and Vector Databases

Dive into the essential strategies for leveraging real-time data streaming, AI automation, and vector databases to drive actionable insights.

November 7, 2024

by Tuhin Chattopadhyay

CORE

· 3,706 Views · 3 Likes

Building Scalable AI-Driven Microservices With Kubernetes and Kafka

AI microservices, Kubernetes, and Kafka enable scalable, resilient intelligent applications through modular architecture and efficient resource management.

November 6, 2024

by Dileep Kumar Pandiya

· 6,374 Views · 4 Likes

Leveraging Apache Flink Dashboard for Real-Time Data Processing in AWS Apache Flink Managed Service

Find out how to utilize the Apache Flink Dashboard for monitoring, optimizing, and managing real-time data processing applications within AWS-managed services.

November 6, 2024

by Sneha Murganoor

· 27,904 Views · 7 Likes

The Latest Big Data Topics