Data Engineering Resources

The Role of Data Governance in Data Strategy: Part 4

Explore the critical role of data retention in governance: reduce costs, mitigate legal and cybersecurity risks, and ensure compliance with clear policies.

September 5, 2025

by Satish Gaddipati

· 2,250 Views · 2 Likes

Change Data Capture for Apache Phoenix Stream

Apache Phoenix, a relational low latency, high throughput database backed by Apache HBase has implemented Change Data Capture. Learn the details of the CDC Streaming.

September 5, 2025

by Viraj Jasani

· 2,627 Views · 4 Likes

Build a RAG Application With LangChain and Local LLMs Powered by Ollama

Learn how to run local LLMs with Ollama and Azure Cosmos DB for RAG, ensuring data privacy, offline use, and full control of your AI workflow.

September 5, 2025

by Abhishek Gupta

CORE

· 3,279 Views · 1 Like

Toward Explainable AI (Part 7): Bridging Theory and Practice—SHAP: Bringing Clarity to Financial Decision-Making

Explainable AI bridges the gap between complex models and real-world accountability, helping teams build trust, ensure compliance, and make smarter decisions.

September 5, 2025

by Frederic Jacquet

CORE

· 3,280 Views · 2 Likes

How to Use AI to Enhance Scrum Ceremonies

This article cites some trends on Scrum and AI usage and provides details on how AI tools can help automate Scrum ceremonies without impacting human values.

September 4, 2025

by Arpit Garg

· 3,683 Views · 2 Likes

Protecting PII in LLM Applications: A Complete Guide to Data Anonymization

One of the biggest concerns companies face when implementing Large Language Models (LLMs) is data privacy and protection.

September 4, 2025

by Swati Tyagi

· 12,611 Views · 1 Like

Building AI Agents? 5 Critical Questions to Ask Before You Automate

Five essential questions to ask before deploying AI agents, focused on data, context, trust, governance, and real-world readiness.

September 4, 2025

by Pradeep Kumar Sharma

· 2,193 Views

Take AI Out of Your Silos—Why Team Experience Trumps Developer Tools

Stop optimizing individual dev tools with AI. Team workflows need AI that carries context end-to-end, not another siloed copilot that makes you its secretary.

September 4, 2025

by Michael Smith

· 1,738 Views · 1 Like

Active Learning and Human-in-the-Loop for NLP Annotation and Model Improvement

A practical guide to using active learning and human-in-the-loop workflows for efficient NLP annotation, model training, and continuous improvement.

September 4, 2025

by Surya Teja Appini

· 2,540 Views · 5 Likes

Developing a Nationwide Real-Time Telemetry Analytics Platform Using Google Cloud Platform and Apache Airflow

Designed a real-time telemetry analytics platform using GCP and Airflow to process 10TB+ daily data, reduce support escalations, and improve operational visibility.

September 3, 2025

by Sruthi Erra Hareram

· 2,911 Views · 2 Likes

DSLs vs. Libraries: Evaluating Language Design in the GenAI Era

Compare general-purpose and domain-specific languages, their AI-driven evolution, and how they optimize data pipelines and trading workflows efficiently.

September 3, 2025

by Raghav Talwar

· 2,896 Views · 1 Like

Observability for the Invisible: Tracing Message Drops in Kafka Pipelines

Kafka lag lies. Use Fluent Bit, OpenTelemetry, DLQs, and trace IDs to expose missing messages and harden observability in event-driven pipelines.

September 3, 2025

by Prakash Wagle

· 3,117 Views · 2 Likes

Simple Efficient Spring/Kafka Datastreams

Datastreams from Souce via Kafka to Sink done simple, easy to code, efficient to run. Deployed in Kubernetes and written in Java and Kotlin.

September 3, 2025

by Sven Loesekann

· 6,551 Views · 7 Likes

Understanding Zero-Copy

This article will delve into the concept of zero-copy and provides a practical Python code example demonstrating its performance benefits

September 3, 2025

by Bharath Modhipalli

· 3,367 Views

Understanding Apache Spark Join Types

Three join types in Spark data frame SQL operations are crucial for the performance of big data Apache Spark applications.

September 3, 2025

by Ram Ghadiyaram

CORE

· 4,194 Views · 5 Likes

File Systems <> Database: Full Circle

The start of the computer storage era was a file-based system, which evolved into databases; However, data advancement made file systems relevant again.

September 3, 2025

by BHUSHAN FADNIS

· 1,711 Views

Toward Explainable AI (Part 6): Bridging Theory and Practice—What LIME Shows – and What It Leaves Out

Explainable AI bridges the gap between complex models and real-world accountability, helping teams build trust, ensure compliance, and make smarter decisions.

September 3, 2025

by Frederic Jacquet

CORE

· 2,056 Views

Technical Deep Dive: Scaling GenAI-Enhanced SBOM Analysis from Trivy Fix to Enterprise DevSecOps

Discover how Trivy PR #9224 enables complete SBOM dependency mapping, powering GenAI-driven DevSecOps automation and millions in enterprise cost savings.

September 2, 2025

by Ravi Sastry Kadali

· 2,649 Views · 1 Like

Modernizing Oracle Workloads With Real-Time Analytics

AWS now supports zero-ETL integration from Amazon RDS for Oracle to Redshift, enabling near-real-time analytics and machine learning without complex pipelines.

September 2, 2025

by Harpreet Kaur Chawla

· 1,716 Views · 1 Like

Prototype for a Java Database Application With REST and Security

Prototype for a Java database application with REST and security using Spring Boot and containers for testing, using Keycloak for security and PostgreSQL for persistence.

September 2, 2025

by George Pod

· 4,208 Views · 5 Likes

The Latest Data Engineering Topics