Big Data Resources

What Is Lambda Architecture? Ultimate Guide to Getting Started

Discover the basics of lambda architecture in our comprehensive guide. Explore its components, workflow, applications, and best practices.

Updated October 1, 2024

by Michael Bogan

CORE

· 16,930 Views · 12 Likes

5 Best Practices for Data Warehousing

Getting started in data warehousing is a major undertaking, so it's important to consider a few best practices before beginning.

Updated October 1, 2024

by Zac Amos

· 8,373 Views · 2 Likes

Stream Processing in the Serverless World

Learn how real-time data processing enables businesses to gain actionable insights without the overhead of managing infrastructure.

September 30, 2024

by Rajesh Kumar Pandey

· 5,178 Views · 4 Likes

Assigning Pods to Nodes Using Affinity Rules

Affinity and anti-affinity are Kubernetes features to help you manage the schedule of your applications. Explore two scenarios where to use these features.

September 27, 2024

by Rafael Natali

· 5,514 Views · 4 Likes

Is Spring AI Strong Enough for AI?

Explore Spring's capabilities within the AI domain, its potential integration with AI libraries, and its ability to effectively manage AI workflows.

September 27, 2024

by Reza Ganji

CORE

· 16,800 Views · 15 Likes

Integrating Data Engineering Into Artificial Intelligence

Importance of tight coupling between data engineering and artificial intelligence for data engineers and data scientists and best practices.

September 26, 2024

by Akshay Agarwal

· 3,831 Views · 3 Likes

Using Flink, Kafka, and NiFi for Real-Time Airport Arrivals and Departures

Learn to build a streaming application using the best of NiFi, Kafka, and Flink for event-driven apps. OpenSky networks rest feeds provide all the data.

September 26, 2024

by Timothy Spann

· 4,614 Views · 5 Likes

Automating Data Pipelines With Snowflake: Leveraging DBT and Airflow Orchestration Frameworks for ETL/ELT Processes

This article explores the automation of data pipelines using Snowflake, dbt, and Airflow, detailing best practices for efficient data processing and orchestration.

September 25, 2024

by Harshavardhan Yedla

· 5,643 Views · 2 Likes

Feature Engineering Transforming Predictive Models

Delve into the transformative power of feature engineering in applied machine learning, and learn how carefully crafted features can elevate your models.

September 25, 2024

by Sundeep Goud Katta

· 4,224 Views · 3 Likes

What Is a Data Pipeline?

In this post, we examine data pipelines, explore their benefits, compare them to other data processes, and discuss various implementation methods.

Updated September 24, 2024

by Garrett Alley

· 53,100 Views · 18 Likes

Navigating the Regulatory Maze: Simplifying Data Compliance

Explore how IT professionals can address evolving regulatory compliance challenges in data management, reducing organizational risks and costs.

September 20, 2024

by Tom Smith

CORE

· 4,508 Views · 1 Like

Kafka Message Testing

Utilizing tools like RecordCaptor, as well as adhering to isolation principles and clear separation of test stages, ensures high accuracy and efficiency.

September 18, 2024

by Anton Belyaev

· 4,737 Views · 10 Likes

Setting Up Secure Data Lakes for Starlight Financial: A Guide to AWS Implementation

This guide delves into securing financial data lakes with AWS services, focusing on best practices for data protection and compliance.

September 13, 2024

by Harsh Daiya

CORE

· 5,771 Views · 1 Like

Optimizing Data Management for AI Success: Industry Insights and Best Practices

Explore key strategies for effective data management in AI projects, including real-time access, federated queries, and data literacy for developers and engineers.

September 11, 2024

by Tom Smith

CORE

· 5,751 Views · 1 Like

The Significance of Complex Event Processing (CEP) With RisingWave for Delivering Accurate Business Decisions

Learn more about CEP, how it addresses a key challenge in real-time processing by detecting patterns in data streams, and compare FlinkCEP and RisingWave.

September 11, 2024

by Gautam Goswami

CORE

· 5,708 Views · 1 Like

Real-Time GenAI With RAG Using Apache Kafka and Flink to Prevent Hallucinations

Learn about context-specific real-time Generative AI (GenAI) with Retrieval Augmentation Generation (RAG) using Kafka and Flink to prevent hallucinations.

September 11, 2024

by Kai Wähner

CORE

· 4,783 Views · 2 Likes

Exploring Real-Time Data Ingestion Into Snowflake Using CockroachDB, Redpanda, and Kafka Connect

Explore Kafka Connect as a solution to stream changefeeds into Snowflake for greater control over how messages are delivered to Snowflake.

September 10, 2024

by Artem Ervits

CORE

· 5,950 Views · 1 Like

Accelerate Your Journey to a Modern Data Platform Using Coalesce

This article will identify key challenges organizations face today in managing data platforms, and explore how advanced ETL tools can address these challenges.

September 10, 2024

by Asia Banu Shaik

· 5,909 Views · 5 Likes

Data Storage Formats for Big Data Analytics: Performance and Cost Implications of Parquet, Avro, and ORC

This article compares the performance and cost efficiency of three storage formats Parquet, Avro, and ORC on Google Cloud Platform.

September 9, 2024

by Rahul Sarabu

· 8,596 Views · 4 Likes

Principles of Modern Data Infrastructure

Explore principles of modern data infrastructure such as scalability, high availability, speed, security, maintainability, efficiency, and developer experience.

September 6, 2024

by Joe Zhou

· 5,606 Views · 1 Like

The Latest Big Data Topics