Data Engineering Resources

Why Mocking Sucks

Mocking can be risky as it doesn't replicate real-world conditions, making real development versions of services preferable.

March 28, 2025

by Andy Pai

· 4,657 Views · 2 Likes

Using Terraform Moved Block to Refactor Resources

Terraform moved block, introduced in version 1.1.0, provides a straightforward way to refactor resources by explicitly mapping old resource addresses to new ones.

March 28, 2025

by Mariusz Michalowski

· 4,607 Views · 1 Like

Building a Scalable ML Pipeline and API in AWS

This blog explains the architecture and design for building a scalable end-to-end ML pipeline using AWS for automated model execution, real-time data processing, and API.

March 28, 2025

by Dhanush Thirugnana

· 5,542 Views · 3 Likes

Accurate Quantitative Analysis With ChatGPT and Azure AI Hub

I asked ChatGPT the question, ‘9.9 or 9.11, which is bigger?’ ChatGPT alone answered incorrectly, but with the help of Python, it provided the correct answer: 9.9.

March 27, 2025

by Sanjeev Kumar

· 3,840 Views · 1 Like

Moving PeopleSoft ERP Data Between Databases With Data Mover Scripts

Learn how to efficiently move application data between different PeopleSoft databases using Data Mover scripts, with detailed examples and step-by-step instructions.

March 27, 2025

by Tirumala Rao Chimpiri

· 4,420 Views · 1 Like

Self-Healing Data Pipelines: The Next Big Thing in Data Engineering?

Build self-healing data pipelines on GCP using AI-powered tools like GPT-4 and Python to detect and fix errors automatically.

March 27, 2025

by Naresh Erukulla

· 9,955 Views

XAI: Making ML Models Transparent for Smarter Hiring Decisions

Explainable AI (XAI) lifts the veil on machine learning in recruiting, showing why candidates get scored or rejected — like skills mismatches or low experience.

March 27, 2025

by Pankaj Khurana

· 15,733 Views · 1 Like

Automatic Versioning in Mobile Apps

This article presents an automatic versioning and migration framework for mobile app databases using schema hashes and seamless upgrades.

March 27, 2025

by Liewen Huang, Ph.D.

· 2,922 Views · 2 Likes

Creating an Application With a Personality

Learn how to create an AI assistant with LangChain4j in a Spring Boot application, which personalizes the application itself and has access to its actuators.

March 27, 2025

by Tamas Kiss

· 3,911 Views · 1 Like

Optimized Metrics Generation With Metadata-Driven Dynamic SQL

Learn about a metadata-driven approach to generate metrics with dynamic SQL, enhancing scalability, flexibility, and efficiency in auditing and reporting.

March 27, 2025

by Tapan Parekh

· 2,873 Views · 1 Like

Breaking the Context Barrier of LLMs: InfiniRetri vs RAG

InfiniRetri retrieves information internally using LLM attention, while RAG retrieves it externally. The future may lie in hybrid approaches combining both strengths.

March 27, 2025

by Graziano Casto

CORE

· 2,523 Views · 1 Like

The Future of DevOps

Learn about Infrastructure as Code (IaC) predictions for 2025, from AI-driven drift management to cost optimization, platform engineering, and multi-framework trends.

March 27, 2025

by Omry Hay

· 4,044 Views · 3 Likes

How to Port CV/ML Models to NPU for Faster Face Recognition

This article explains the process of porting face recognition models to Rockchip NPU for higher performance in access control systems (ACS).

March 27, 2025

by Sergey Alabugin

· 4,039 Views · 1 Like

A Comprehensive Guide to Protect Data, Models, and Users in the GenAI Era

This article provides practical checklists to help enterprises securely adopt GenAI.

March 27, 2025

by Boris Zaikin

CORE

· 5,396 Views · 3 Likes

Ensuring Data Quality With Great Expectations and Databricks

Ensure data quality in pipelines with Great Expectations. Learn to integrate with Databricks, validate data, and automate checks for reliable datasets.

March 26, 2025

by Sairamakrishna BuchiReddy Karri

· 5,629 Views · 2 Likes

Avoid Cross-Shard Data Movement in Distributed Databases

Learn four effective strategies to optimize distributed joins, reduce network overhead, and improve query performance in sharded databases.

March 26, 2025

by Baskar Sikkayan

· 6,849 Views · 5 Likes

Jakarta NoSQL 1.0: A Way To Bring Java and NoSQL Together

Jakarta NoSQL 1.0 standardizes NoSQL integration in Java with familiar annotations, a fluent API, and support for multiple NoSQL types—boosting productivity, portability.

March 26, 2025

by Otavio Santana

CORE

· 9,446 Views · 6 Likes

A Beginner's Guide to Spark UI: Concepts and How to Use It

Understanding the fundamentals of Spark UI is essential for optimization and debugging. Although the UI is old, the information it contains is gold.

March 26, 2025

by Saransh Arora

· 3,576 Views · 3 Likes

Lakehouse: Manus? MCP? Let’s Talk About Lakehouse and AI

This is the second article in the “Lakehouse: What’s the Big Deal?” series, where I will periodically discuss Lakehouse. Your comments and discussions are welcome.

March 26, 2025

by Mingyu Chen

· 4,103 Views · 2 Likes

How to Integrate Event-Driven Ansible With Kafka

Integrate Ansible with Kafka for real-time automation: trigger playbooks via Kafka events, enhance incident response, optimize workflows, and scale seamlessly.

March 26, 2025

by Binoj Melath Nalinakshan Nair

CORE

· 5,776 Views · 12 Likes

The Latest Data Engineering Topics