Big Data Resources

Simple Efficient Spring/Kafka Datastreams

Datastreams from Souce via Kafka to Sink done simple, easy to code, efficient to run. Deployed in Kubernetes and written in Java and Kotlin.

September 3, 2025

by Sven Loesekann

· 6,658 Views · 7 Likes

Understanding Apache Spark Join Types

Three join types in Spark data frame SQL operations are crucial for the performance of big data Apache Spark applications.

September 3, 2025

by Ram Ghadiyaram

CORE

· 4,439 Views · 5 Likes

File Systems <> Database: Full Circle

The start of the computer storage era was a file-based system, which evolved into databases; However, data advancement made file systems relevant again.

September 3, 2025

by BHUSHAN FADNIS

· 1,847 Views

Keep Your Search Cluster Fit: Essential Health Checks to Keep Elasticsearch Healthy

A search cluster in top notch state requires frequent monitoring for health stats. Let's look at some health checks to always keep your ES cluster fit.

August 29, 2025

by Govind Singh Rawat

· 2,739 Views · 2 Likes

Designing Scalable Ingestion and Access Layers for Policy and Enforcement Data

Build a scalable, low-latency architecture for ingesting and accessing policy and enforcement data using Apache Spark and in-memory data grids.

August 28, 2025

by Pankaj Taneja

· 1,566 Views · 1 Like

How Healthy Is Your Data in the Age of AI? An In-Depth Checklist to Assess Data Accuracy, Governance, and AI Readiness

This guide provides a complete checklist to assess, monitor, and improve data quality for AI success, ensuring accuracy, compliance, and long-term reliability.

August 28, 2025

by Sukanya Konatam

· 2,787 Views · 2 Likes

Implementing Scalable IoT Architectures on Azure

Microsoft’s Azure IoT platform has emerged as a leading choice, powering innovative solutions across industries — from manufacturing floors to smart buildings.

August 27, 2025

by Bhimraj Ghadge

· 3,469 Views · 4 Likes

Blockchain, AI, and Edge Computing: Redefining Modern App Development

Blockchain, AI, and edge computing are enhancing modern app development by enhancing security, performance, and automation.

August 27, 2025

by Balasubramani Murugesan

· 3,099 Views · 1 Like

Digital Twins Reborn: How AI Is Finally Fulfilling the Promise of IoT

AI is revolutionizing digital twin technology, finally overcoming IoT's integration challenges and delivering real business value across industries.

August 27, 2025

by Tom Smith

CORE

· 2,601 Views · 11 Likes

Zero-Latency Data Analytics for Modern PostgreSQL Applications

Learn in this article how to set up Amazon RDS for PostgreSQL zero-ETL integration with Amazon Redshift for near-real-time analytics using the AWS CLI.

August 26, 2025

by Harpreet Kaur Chawla

· 2,927 Views · 2 Likes

Building a 3D WebXR Game with WASI Cycles: Integrating WasmEdge, Wasmtime, and Wasmer to Invoke MongoDB, Kafka, and Oracle

Develop Fullstack WASM and WASI using all the most popular frameworks, databases, frontends, etc., from top to bottom, including source code

August 25, 2025

by Paul Parkinson

· 2,695 Views · 4 Likes

From History to the Future of AI Communication—IPC to MCP and A2A

MCP and A2A are stablishing standards while existing middleware markets are promising to provide the rails. The comparison and the outlook.

August 22, 2025

by Shashi Kumar

· 2,497 Views · 1 Like

Real-Time Model Inference With Apache Kafka and Flink for Predictive AI and GenAI

Learn how data streaming with Kafka and Flink enhances AI/ML model inference, enabling low-latency, scalable predictions in real-time business use cases.

August 22, 2025

by Kai Wähner

CORE

· 1,869 Views · 2 Likes

Data Lake, Warehouse, or Lakehouse? Rethinking the Future of Data Architecture

AI is driving data lakes, warehouses, and lakehouses to converge into open, real-time platforms powered by Iceberg, Trino, and DuckDB.

August 22, 2025

by Miguel Garcia

CORE

· 3,004 Views · 1 Like

Data Storage: The Foundation for Scalable Analytics

This article provides a blueprint to build a scalable data storage foundation using a three-step framework of 5Q, BSG, and HWC with practical application.

August 22, 2025

by Junaith Haja

· 1,519 Views · 3 Likes

Greenplum vs Apache Doris: Features, Performance, and Advantages Compared

Compare Greenplum vs. Apache Doris for MPP-based analytics. Learn which database suits real-time, high-concurrency workloads and evolving data architectures.

August 21, 2025

by li yy

· 2,751 Views · 2 Likes

Operationalizing the OWASP AI Testing Guide: Building Secure AI Foundations Through NHI Governance

Align your AI pipelines with OWASP AI Testing principles using identity-based insights to monitor, enforce, and audit secrets and token usage best practices.

August 20, 2025

by Dwayne McDaniel

· 1,502 Views · 3 Likes

What We Learned Migrating to a Pub/Sub Architecture: Real-World Case Studies from High-Traffic Systems

Kafka-powered migration of an e-commerce platform—focused on scalability, fault tolerance, and clean event-driven design.

August 19, 2025

by Ravi Teja Thutari

CORE

· 8,629 Views · 4 Likes

Prompt-Based ETL: Automating SQL Generation for Data Movement With LLMs

Prompt-based ETL uses LLMs to convert plain English into validated, schema-aware SQL, automating data transformation and enabling self-serve analytics.

August 18, 2025

by Saisuman Singamsetty

· 6,982 Views · 4 Likes

Real-Time Analytics Using Zero-ETL for MySQL

Learn how to set up a zero-ETL integration from Amazon RDS for MySQL to Amazon Redshift using AWS CLI for real-time analytics without complex pipelines.

August 18, 2025

by Harpreet Kaur Chawla

· 1,995 Views · 2 Likes

The Latest Big Data Topics