Data Resources

RAG at Scale: The Data Engineering Challenges

The data engineering challenges that go beyond the basic concept of RAG when running RAG systems at scale in production.

January 16, 2026

by Guru Hegde

· 2,629 Views · 1 Like

Parallel S3 Writes for Massive Sparse DataFrames: How to Maintain Row Order Without Blowing Memory

Learn how to write massive sparse Pandas DataFrames to S3 without OOM errors by using Spark to parallelize index-based chunks while preserving row order.

January 16, 2026

by pooja chhabra

· 1,562 Views · 2 Likes

RAG on Android Done Right: Local Vector Cache Plus Cloud Retrieval Architecture

Most mobile RAG fails on latency and flaky networks. A local vector cache + cloud retrieval architecture keeps responses fast, fresh, and grounded.

January 16, 2026

by Mohan Sankaran

· 1,799 Views · 5 Likes

From Aspects to Advisors: Design Modular Cross-Cutting Features with Spring AI

Understand how Spring AI Advisors work and see how Aspect Oriented Programming concepts can be applied when interacting with LLMs.

January 15, 2026

by Horatiu Dan

CORE

· 2,261 Views · 5 Likes

Designing Chatbots for Multiple Use Cases: Intent Routing and Orchestration

Effective multi-use-case chatbots depend on strong intent routing, context-aware orchestration, tailored LLM parameters, and evaluation.

January 14, 2026

by Vinayak Prasad

· 4,549 Views · 1 Like

Integrating Retrieval-Augmented Generation (RAG) With Agentic AI: Harnessing Elasticsearch Vector Databases for Enterprise AI Systems

A practical overview of using retrieval-augmented generation and agentic AI with Elasticsearch to build reliable, enterprise-ready LLM systems.

January 14, 2026

by Devdas Gupta

· 2,700 Views · 2 Likes

Your Next Customer Is a Bot

E-commerce brands lose billions annually to cart abandonment, with over 70% of checkouts left unfinished. A new technology, Agentic AI, is here to solve this.

January 13, 2026

by Akash Lomas

· 1,498 Views · 1 Like

Optimizing Financial Data Pipelines: Accelerating OneStream-to-Snowflake Exports by 85%

A case study on optimizing financial data pipelines by refactoring legacy SQL batch inserts into a "Stage and Copy" architecture.

January 13, 2026

by Sridhar Mannava

· 1,002 Views · 3 Likes

MCP Servers Are Everywhere, but Most Are Collecting Dust: Key Lessons We Learned to Avoid That

Teams rushing to build MCP servers are discovering that enthusiasm doesn’t translate into usefulness. This article unpacks why and how to make your MCP server valuable.

January 13, 2026

by Thomas Johnson

CORE

· 2,739 Views · 4 Likes

Apache Spark 4.0: What’s New for Data Engineers and ML Developers

Spark 4.0 brings Spark Connect, enhanced SQL (PIPE, VARIANT), richer Python APIs, and advanced streaming — modernizing Spark for faster, more flexible 2025 workloads.

January 12, 2026

by harshraj bhoite

· 2,206 Views

Serverless Spark Isn't Always the Answer: A Case Study

Processing 500M+ records with 100 concurrent users under a 5-minute SLA demands smart architecture. We evaluate seven compute models and why hybrid approaches often win.

January 12, 2026

by Janani Annur Thiruvengadam

CORE

· 1,650 Views · 1 Like

Mastering Fluent Bit: Developer Guide to Telemetry Pipeline Routing (Part 12)

This intro to mastering Fluent Bit covers telemetry pipeline routing mechanisms, tag-based, conditional, and label-based, with hands-on examples for developers.

January 9, 2026

by Eric D. Schabell

CORE

· 1,627 Views · 2 Likes

Essential Techniques for Production Vector Search Systems Part 2 - Binary Quantization

Proven techniques for production vector search including when to use each one, how to combine them effectively, trade offs to understand before deployment.

January 9, 2026

by Pavan Vemuri

CORE

· 1,998 Views · 3 Likes

Multi-Region Apache Kafka using Synchronous Replication for Disaster Recovery With Zero Data Loss (RPO=0)

Kafka isn’t one-size-fits-all. Choose between self-managed, serverless, or BYOC deployments. New RPO=0 options now enable zero data loss for real-time applications.

January 9, 2026

by Kai Wähner

CORE

· 1,656 Views · 2 Likes

Essential Techniques for Production Vector Search Systems Part 1 - Hybrid Search

Proven techniques for production vector search including when to use each one, how to combine them effectively, and trade offs to understand before deployment.

January 8, 2026

by Pavan Vemuri

CORE

· 2,132 Views · 2 Likes

Secure Log Tokenization Using Aho–Corasick and Spring

This article shows how to use the Aho–Corasick algorithm and deterministic tokenization in Spring Boot to intercept logs in real time, remove sensitive values.

January 8, 2026

by Balakumaran Sugumar

· 2,165 Views · 3 Likes

Solving the Cold Start Problem in Edge AI: A Guide to Data-Saving Learning

Update edge AI models efficiently using Mix Up and contribution sampling to overcome domain shift with minimal data, ensuring continuous evolution without forgetting.

January 6, 2026

by Dippu Kumar Singh

· 3,628 Views

Metadata, Not Data Volume, Is the Real Bottleneck in Modern Data Lakes

In Apache Iceberg data lakes, growing snapshots and manifests often make metadata resolution — not data scanning — the primary performance bottleneck.

January 6, 2026

by Vivek Venkatesan

· 3,415 Views

Why Data Engineers Need to Think Like Product Managers

Data engineers who think like product managers build more valuable, trusted, and user-centric data systems; they focus on outcomes, ownership, and UX, not just pipelines.

January 6, 2026

by harshraj bhoite

· 2,344 Views

Understanding Parquet Scans: How Readers Skip Work and Stay Fast

Parquet accelerates scans by skipping data through metadata driven pushdowns. This article explains how the main mechanisms work in practice.

January 6, 2026

by Hitarth Trivedi

· 2,547 Views · 1 Like

The Latest Data Topics