Big Data Resources

Take a deep dive into the architectural concepts of data pipelines along with a hands-on tutorial for implementation, demonstrating the concepts in action.

August 21, 2024

by Sohil Shah

· 7,193 Views · 2 Likes

Preventing and Fixing Bad Data in Event Streams: Part 2

The techniques we use for dealing with bad data in event streams differ from those in the batch world. Here, learn more about overcoming bad data in streaming.

August 21, 2024

by Adam Bellemare

· 6,576 Views · 3 Likes

Mastering PUE for Unmatched Data Center Performance

PUE can make or break data center energy efficiency. Improving this figure will help your facility lower costs and become more sustainable.

August 20, 2024

by Emily Newton

· 4,657 Views · 1 Like

Understanding PolyBase and External Stages: Making Informed Decisions for Data Querying

Explore the differences and strengths of PolyBase and Snowflake external tables to optimize data querying strategies and achieve efficient data integration.

August 19, 2024

by Harshavardhan Yedla

· 5,104 Views · 2 Likes

Data Pipeline vs. ETL Pipeline

Understand the key differences, use cases, and best practices for real-time data processing and batch data integration.

August 15, 2024

by Pier-Jean MALANDRINO

CORE

· 7,583 Views · 4 Likes

Essential Guidelines for Building Optimized ETL Data Pipelines in the Cloud With Azure Data Factory

Discover how to optimize Azure Data Factory pipelines by selecting the right activities and managing resources for reliable data processing.

August 14, 2024

by Aravind Nuthalapati

CORE

· 37,615 Views · 5 Likes

DynamoDB: How to Move Out

Cover specific characteristics related to DynamoDB migrations and strategies employed to integrate with and migrate data seamlessly to other databases.

August 12, 2024

by Felipe Mendes

· 6,509 Views · 4 Likes

Data Fusion: Enhancing Interoperability, Privacy, and Security

Big Data alone isn't enough; quality and integration are key. Discover how data fusion enhances efficiency and decision-making in AI systems.

August 12, 2024

by Saigurudatta Pamulaparthyvenkata

· 3,794 Views · 2 Likes

Apache Kafka + Flink + Snowflake: Cost-Efficient Analytics and Data Governance

Explore shift-left architecture with Snowflake, Apache Kafka, Flink, and Iceberg for cost-efficient analytics and data governance in the cloud.

August 10, 2024

by Kai Wähner

CORE

· 6,303 Views · 1 Like

Connecting ChatGPT to Code Review Made Easy

GPT can generate code quite well, can it review it just as smoothly? Read this article to find out more.

August 9, 2024

by Dmytro Diachenko

· 5,836 Views · 1 Like

Batch vs. Real-Time Processing: Understanding the Differences

Learn the differences between batch and real-time data processing, and explore the decision-making factors for choosing the right approach to optimize data pipelines.

August 8, 2024

by Stelios Manioudakis

CORE

· 17,009 Views · 3 Likes

Apache Flink 101: A Guide for Developers

Learn more about Apache Flink, a powerful stream processing tool, for building streaming data pipelines, real-time analytics, and event-driven applications.

August 8, 2024

by David Anderson

· 9,573 Views · 2 Likes

How to Create a CRUD Application in Less Than 15 Minutes

This step-by-step guide explores how to create a front end for your database with Java only, without any JavaScript knowledge needed.

August 8, 2024

by Amir Shelli

· 18,590 Views · 8 Likes

How to Scale RAG and Build More Accurate LLMs

Retrieval augmented generation (RAG) needs the right data architecture to scale efficiently. Learn how data streaming helps data and application teams innovate.

August 8, 2024

by Andrew Sellers

· 3,688 Views · 2 Likes

Enhancing Agile Product Development With AI and LLMs

AI and LLMs streamline user story creation, optimize backlog, and predict trends, improving agile product development speed, relevance, and user engagement.

August 7, 2024

by Varun Milind Kulkarni

· 7,502 Views · 4 Likes

Handling Schema Versioning and Updates in Event Streaming Platforms Without Schema Registries

Learn how to handle schema versioning and updates in Kafka and other event streaming platforms without using schema registries through custom deserializers.

August 6, 2024

by Sanket Panhale

· 24,022 Views · 1 Like

Building an IoT-based Waste Management System: A Software Architect's Guide

Explore how IoT waste management optimizes collection, reduces costs, and enhances sustainability using smart bins, mobile apps, and analytics.

August 5, 2024

by Arun Pandey

CORE

· 6,213 Views · 2 Likes

Streaming Data Joins: A Deep Dive Into Real-Time Data Enrichment

Unravel the complexities of streaming data joins in this guide covering key concepts, design, and best practices for optimal real-time data enrichment.

August 5, 2024

by Preet Shah

· 5,300 Views · 4 Likes

Upgrading Spark Pipelines Code: A Comprehensive Guide

Discuss the strategic importance of Spark code upgrades and explore an introduction to a powerful toolkit designed to streamline this process: Scalafix.

August 5, 2024

by Suri (thammuio)

CORE

· 25,992 Views · 1 Like

Industrial IoT: Exploring Microcontrollers for Robust Applications

Let's explore the features, applications, and top choices for microcontrollers in industrial IoT, achieving reliability, efficiency, and security in harsh environments.

August 2, 2024

by Carsten Rhod Gregersen

· 5,467 Views · 1 Like

The Latest Big Data Topics