Data Engineering Resources

Optimized Metrics Generation With Metadata-Driven Dynamic SQL

Learn about a metadata-driven approach to generate metrics with dynamic SQL, enhancing scalability, flexibility, and efficiency in auditing and reporting.

March 27, 2025

by Tapan Parekh

· 2,245 Views · 1 Like

Breaking the Context Barrier of LLMs: InfiniRetri vs RAG

InfiniRetri retrieves information internally using LLM attention, while RAG retrieves it externally. The future may lie in hybrid approaches combining both strengths.

March 27, 2025

by Graziano Casto

CORE

· 1,826 Views · 1 Like

The Future of DevOps

Learn about Infrastructure as Code (IaC) predictions for 2025, from AI-driven drift management to cost optimization, platform engineering, and multi-framework trends.

March 27, 2025

by Omry Hay

· 3,206 Views · 3 Likes

How to Port CV/ML Models to NPU for Faster Face Recognition

This article explains the process of porting face recognition models to Rockchip NPU for higher performance in access control systems (ACS).

March 27, 2025

by Sergey Alabugin

· 2,544 Views · 1 Like

A Comprehensive Guide to Protect Data, Models, and Users in the GenAI Era

This article provides practical checklists to help enterprises securely adopt GenAI.

March 27, 2025

by Boris Zaikin

CORE

· 4,053 Views

Ensuring Data Quality With Great Expectations and Databricks

Ensure data quality in pipelines with Great Expectations. Learn to integrate with Databricks, validate data, and automate checks for reliable datasets.

March 26, 2025

by Sairamakrishna BuchiReddy Karri

· 2,934 Views · 1 Like

Avoid Cross-Shard Data Movement in Distributed Databases

Learn four effective strategies to optimize distributed joins, reduce network overhead, and improve query performance in sharded databases.

March 26, 2025

by Baskar Sikkayan

· 4,447 Views · 4 Likes

Jakarta NoSQL 1.0: A Way To Bring Java and NoSQL Together

Jakarta NoSQL 1.0 standardizes NoSQL integration in Java with familiar annotations, a fluent API, and support for multiple NoSQL types—boosting productivity, portability.

March 26, 2025

by Otavio Santana

CORE

· 7,966 Views · 5 Likes

A Beginner's Guide to Spark UI: Concepts and How to Use It

Understanding the fundamentals of Spark UI is essential for optimization and debugging. Although the UI is old, the information it contains is gold.

March 26, 2025

by Saransh Arora

· 2,277 Views · 3 Likes

Lakehouse: Manus? MCP? Let’s Talk About Lakehouse and AI

This is the second article in the “Lakehouse: What’s the Big Deal?” series, where I will periodically discuss Lakehouse. Your comments and discussions are welcome.

March 26, 2025

by Mingyu Chen

· 2,591 Views · 2 Likes

How to Integrate Event-Driven Ansible With Kafka

Integrate Ansible with Kafka for real-time automation: trigger playbooks via Kafka events, enhance incident response, optimize workflows, and scale seamlessly.

March 26, 2025

by Binoj Melath Nalinakshan Nair

CORE

· 4,309 Views · 12 Likes

Building AI-Driven Intelligent Applications: A Hands-On Development Guide for Integrating GenAI Into Your Applications

This article explains how to build a GenAI-powered chatbot and provides a step-by-step guide to build an intelligent application.

March 26, 2025

by Naga Santhosh Reddy Vootukuri

CORE

· 4,219 Views · 1 Like

GenAI: From Prompt to Production

GenAI-Logic is a prompt-driven approach to describing a model and logic using natural language. Create a time tracking system and post to production in less than 2 hours.

March 25, 2025

by Tyler Band

· 5,168 Views · 7 Likes

Java Reborn: Conquering Cloud-Native and Next-Gen Concurrency

Brace yourself — Java conquers cloud-native microservices, unleashes Loom’s fast concurrency, and soars with Valhalla and Panama — future-ready and fierce!

March 25, 2025

by Alok Tibrewala

· 7,969 Views · 10 Likes

Recursive Feature Elimination in Practice

Learn about Recursive Feature Elimination (RFE) to reduce feature count, boost accuracy, prevent overfitting, and build efficient machine learning models.

March 25, 2025

by Srinivas Gurrala

· 1,680 Views

Lakehouse: Starting With Apache Doris + S3 Tables

This is the first article in the “Lakehouse: What’s the Big Deal?” series, where I will periodically discuss Lakehouse. Your comments and discussions are welcome.

March 25, 2025

by Mingyu Chen

· 2,653 Views · 2 Likes

Agentic AI and Generative AI: Revolutionizing Decision Making and Automation

This article examines both agentic and generative AI systems, their relative benefits and drawbacks, their implementation challenges with ethical risks, and more.

March 25, 2025

by Nitish Upadhyay

· 2,972 Views

Make ML Models Work: A Real-World Take on Size and Imbalance

Overcoming NLP hurdles like large models and imbalanced data, the project used TF-IDF, class weighting, and simpler models for accuracy and efficiency.

March 25, 2025

by Gowsiya Syednoor Shek

· 2,133 Views

Mastering Async Context Manager Mocking in Python Tests

Simplify testing complex asynchronous Python code by mastering async context manager mocking with Pytest, ensuring reliable and maintainable test coverage.

March 25, 2025

by Aditya Karnam Gururaj Rao

· 2,491 Views · 2 Likes

Using KRaft Kafka for Development and Kubernetes Deployment

Simplify Kafka with KRaft—ditch ZooKeeper, streamline configs for Docker and Kubernetes, and integrate easily with Spring Boot for development and deployment.

March 25, 2025

by Sven Loesekann

· 4,274 Views · 5 Likes

The Latest Data Engineering Topics