Data Engineering Resources

AI as a Co-Creator, Not Just an Assistant: The Rise of Collaborative Intelligence in Software Development

AI is evolving from a passive coding assistant into an active engineering collaborator. Modern intelligent agents can now design architectures, generate tests, and even deploy full applications—moving far beyond autocomplete and syntax checks.

January 13, 2026

by Pankaj Gupta

· 1,153 Views

Supercharge AI Workflows on Azure: Remote MCP Tool Triggers + Your First TypeScript MCP Server

Remote MCP in Azure Functions exposes serverless tools for AI assistants, enabling scalable, cloud-native workflows with Azure services and bindings.

January 13, 2026

by Swapnil Nagar

· 2,105 Views · 1 Like

UX Research in Agile Product Development: Making AI Workflows Work for People

UX research in agile product development helps teams build AI workflows grounded in real user needs, reducing guesswork and improving ROI.

January 12, 2026

by Priyanka Kuvalekar

· 1,884 Views

Apache Spark 4.0: What’s New for Data Engineers and ML Developers

Spark 4.0 brings Spark Connect, enhanced SQL (PIPE, VARIANT), richer Python APIs, and advanced streaming — modernizing Spark for faster, more flexible 2025 workloads.

January 12, 2026

by harshraj bhoite

· 2,104 Views

The Night We Split the Brain: A Telling of Control & Data Planes for Cloud Microservices

Split control and data planes with versioned snapshots and four contracts — routing, policy, limits, release — for safer rollouts and reliably boring systems.

January 12, 2026

by Bharath Kumar Reddy Janumpally

· 8,317 Views · 2 Likes

Leveraging AI-Based Authentication Factors in Modern Identity and Access Management Solutions

Discover how AI is revolutionizing identity management by making authentication smarter, faster, more adaptive, and highly secure against evolving cyber threats.

January 12, 2026

by Atish Kumar Dash

· 1,415 Views · 1 Like

Why PostgreSQL Vacuum Matters More Than You Think

Learn how dead tuples slow queries, when to use autovacuum or manual Vacuum, and how tuning improves large or high traffic tables.

January 12, 2026

by arvind toorpu

CORE

· 2,481 Views · 3 Likes

Pragmatic Paths to On-Device AI on Android with ML Kit

Use ML Kit to add on-device AI (Text/Barcode/OCR, Object & Pose, Translation) with simple Kotlin APIs-fast, offline, private.

January 12, 2026

by Mohan Sankaran

· 1,880 Views · 6 Likes

Serverless Spark Isn't Always the Answer: A Case Study

Processing 500M+ records with 100 concurrent users under a 5-minute SLA demands smart architecture. We evaluate seven compute models and why hybrid approaches often win.

January 12, 2026

by Janani Annur Thiruvengadam

CORE

· 1,562 Views · 1 Like

Why Encryption Alone Is Not Enough in Cloud Security

Practical Lessons from Real-World Scenarios That Clearly Demonstrate Why Relying on Encryption Alone Cannot Fully Protect Cloud Workloads

January 9, 2026

by Atish Kumar Dash

· 5,719 Views · 1 Like

The Rise of Diskless Kafka: Rethinking Brokers, Storage, and the Kafka Protocol

Diskless Kafka stores all event data in object storage without using brokers for scalable and cost-efficient data streaming architectures.

January 9, 2026

by Kai Wähner

CORE

· 1,787 Views · 2 Likes

Mastering Fluent Bit: Developer Guide to Telemetry Pipeline Routing (Part 12)

This intro to mastering Fluent Bit covers telemetry pipeline routing mechanisms, tag-based, conditional, and label-based, with hands-on examples for developers.

January 9, 2026

by Eric D. Schabell

CORE

· 1,567 Views · 2 Likes

How to Build and Deploy an AI Agent on Kubernetes With AWS Bedrock, FastAPI and Helm

Learn how to build, containerize, and deploy a lightweight, cloud-native AI agent on Amazon EKS using FastAPI, AWS Bedrock, Docker, and Helm.

January 9, 2026

by Chandrasekhar Rao Katru

· 3,547 Views · 6 Likes

Essential Techniques for Production Vector Search Systems Part 2 - Binary Quantization

Proven techniques for production vector search including when to use each one, how to combine them effectively, trade offs to understand before deployment.

January 9, 2026

by Pavan Vemuri

CORE

· 1,934 Views · 3 Likes

Multi-Region Apache Kafka using Synchronous Replication for Disaster Recovery With Zero Data Loss (RPO=0)

Kafka isn’t one-size-fits-all. Choose between self-managed, serverless, or BYOC deployments. New RPO=0 options now enable zero data loss for real-time applications.

January 9, 2026

by Kai Wähner

CORE

· 1,619 Views · 2 Likes

From Code to Runtime: How AI Is Bridging the SAST–DAST Gap

A practical exploration of how AI connects static code vulnerabilities with their potential runtime exploitability in modern cloud-native applications

January 9, 2026

by Atish Kumar Dash

· 1,698 Views · 1 Like

How AI Is Rewriting DevOps: Practical Patterns for Faster, Safer Releases

AI is reshaping DevOps by accelerating development, testing, and deployment while enhancing reliability through smart automation.

January 9, 2026

by Bhanu Sekhar Guttikonda

CORE

· 2,041 Views · 2 Likes

When Services Think for Themselves: Traditional Orchestration vs. Agentic AI Microservices

Why traditional orchestration falls short and how agentic AI enables predictive, autonomous microservices for modern cloud-native architectures.

January 8, 2026

by Atish Kumar Dash

· 2,450 Views · 1 Like

Essential Techniques for Production Vector Search Systems Part 1 - Hybrid Search

Proven techniques for production vector search including when to use each one, how to combine them effectively, and trade offs to understand before deployment.

January 8, 2026

by Pavan Vemuri

CORE

· 2,017 Views · 2 Likes

Telemetry-Driven AI Architecture: Closing the Loop from UX to Models

Most Android AI features fail after launch because they don’t learn from real users — this architecture logs predictions and user outcomes.

January 8, 2026

by Mohan Sankaran

· 2,867 Views · 5 Likes

The Latest Data Engineering Topics