Big Data Resources

Batch vs. Real-Time Processing: Understanding the Differences

Learn the differences between batch and real-time data processing, and explore the decision-making factors for choosing the right approach to optimize data pipelines.

August 8, 2024

by Stelios Manioudakis, PhD

CORE

· 13,056 Views · 2 Likes

Apache Flink 101: A Guide for Developers

Learn more about Apache Flink, a powerful stream processing tool, for building streaming data pipelines, real-time analytics, and event-driven applications.

August 8, 2024

by David Anderson

· 6,797 Views · 2 Likes

How to Create a CRUD Application in Less Than 15 Minutes

This step-by-step guide explores how to create a front end for your database with Java only, without any JavaScript knowledge needed.

August 8, 2024

by Amir Shelli

· 16,398 Views · 7 Likes

How to Scale RAG and Build More Accurate LLMs

Retrieval augmented generation (RAG) needs the right data architecture to scale efficiently. Learn how data streaming helps data and application teams innovate.

August 8, 2024

by Andrew Sellers

· 2,921 Views · 2 Likes

Enhancing Agile Product Development With AI and LLMs

AI and LLMs streamline user story creation, optimize backlog, and predict trends, improving agile product development speed, relevance, and user engagement.

August 7, 2024

by Varun Milind Kulkarni

· 4,851 Views · 4 Likes

Handling Schema Versioning and Updates in Event Streaming Platforms Without Schema Registries

Learn how to handle schema versioning and updates in Kafka and other event streaming platforms without using schema registries through custom deserializers.

August 6, 2024

by Sanket Panhale

· 16,011 Views · 1 Like

Building an IoT-based Waste Management System: A Software Architect's Guide

Explore how IoT waste management optimizes collection, reduces costs, and enhances sustainability using smart bins, mobile apps, and analytics.

August 5, 2024

by Arun Pandey

CORE

· 4,513 Views · 2 Likes

Streaming Data Joins: A Deep Dive Into Real-Time Data Enrichment

Unravel the complexities of streaming data joins in this guide covering key concepts, design, and best practices for optimal real-time data enrichment.

August 5, 2024

by Preet Shah

· 3,745 Views · 4 Likes

Upgrading Spark Pipelines Code: A Comprehensive Guide

Discuss the strategic importance of Spark code upgrades and explore an introduction to a powerful toolkit designed to streamline this process: Scalafix.

August 5, 2024

by Suri Nuthalapati

CORE

· 24,760 Views · 1 Like

Industrial IoT: Exploring Microcontrollers for Robust Applications

Let's explore the features, applications, and top choices for microcontrollers in industrial IoT, achieving reliability, efficiency, and security in harsh environments.

August 2, 2024

by Carsten Rhod Gregersen

· 4,063 Views · 1 Like

Criticality in Data Stream Processing and a Few Effective Approaches

Opposite to traditional batch data processing technique, here processing works on the data as it is produced in real-time. Learn more in this post.

August 2, 2024

by Gautam Goswami

CORE

· 4,951 Views · 1 Like

How to Run Apache Spark on Kubernetes in Less Than 5 Minutes

Learn how to quickly set up and run Apache Spark on Kubernetes using Ilum for easier deployment, management, and scaling.

July 30, 2024

by Adam Lichtenstein

· 8,744 Views · 9 Likes

Real-Time Anomaly Detection Using Large Language Models

Real-time anomaly detection using LLMs enhances accuracy for finance, healthcare, and cybersecurity through contextual analysis and pattern recognition.

July 30, 2024

by Harsh Daiya

CORE

· 4,946 Views · 2 Likes

Apache Iceberg Table Management

Apache Iceberg Table Management is essential to ensure smooth table operations by expiring/deleting unused data, metadata files.

July 30, 2024

by Rohit Kumar

· 1,972 Views · 1 Like

Real-Time Streaming Architectures: A Technical Deep Dive Into Kafka, Flink, and Pinot

While individual components like Kafka, Flink, and Pinot are very powerful, managing them at scale across cloud and on-premises deployments can be operationally complex.

July 28, 2024

by Abhishek Gupta

CORE

· 6,074 Views · 3 Likes

Snowflake Integration Patterns: Apache Kafka vs. Zero ETL and Reverse ETL

Integration patterns for cloud-native Snowflake data warehouse: Batch data, Zero ETL, Reverse ETL, Real-time data ingestion with Apache Kafka.

July 24, 2024

by Kai Wähner

CORE

· 5,220 Views · 2 Likes

Step-By-Step Guide To Setting up and Training GANs for Image Generation

Learn to set up and train Generative Adversarial Networks (GANs) using TensorFlow and PyTorch for image generation and manipulation.

July 23, 2024

by Neha Dhaliwal

· 3,193 Views · 2 Likes

Emerging Trends in Data Warehousing: What’s Next?

Explore the latest advances and future trends in data warehousing technologies, highlighting the innovations that are shaping the next generation of data warehouses.

July 23, 2024

by Metin SARIKAYA

· 42,869 Views · 1 Like

Building Your Own IoT Project: A Step-By-Step Guide

This guide will walk you through the essential steps to build your first IoT project, ensuring you have a solid foundation to explore this fascinating field.

July 22, 2024

by Sachin Reddy

· 4,572 Views · 3 Likes

When Not to Choose Google Apache Kafka for BigQuery

When not to choose Google Apache Kafka for BigQuery — a review, comparison with Confluent Cloud, and data streaming landscape outlook.

July 16, 2024

by Kai Wähner

CORE

· 4,393 Views · 1 Like

The Latest Big Data Topics