Data Engineering Resources

Analysis of the Data Processing Framework of Pandas and Snowpark Pandas API

This is a process analysis of migrating existing Pandas workflows to an almost lift-and-shift approach using the Snowpark Pandas API to meet ever-growing data needs.

July 15, 2025

by Prasath Chetty Pandurangan

· 3,005 Views · 1 Like

How to Build a Real API Gateway With Spring Cloud Gateway and Eureka

Build a real-world API gateway using Spring Cloud Gateway and Eureka Server. Perfect for beginners looking to understand microservices architecture with working code.

July 15, 2025

by Vivek Rajyaguru

· 7,220 Views · 2 Likes

API Standards Are Data Standards

API standards are often created separately from an org's data standards. However, they are much more similar than many realize and, therefore, should be created together.

July 15, 2025

by Scott Sosna

CORE

· 4,379 Views · 2 Likes

Designing Configuration-Driven Apache Spark SQL ETL Jobs with Delta Lake CDC

Simplify complex ETL pipelines and enable scalable, maintainable data processing with Spark SQL and Delta Lake Change Data Capture.

July 14, 2025

by Janaki Ganapathi

· 2,359 Views

Testing Distributed Microservices Using XState

Learn how to use XState to model microservice workflows. Simplify testing, boost coverage, and debug visually using declarative state machines.

July 14, 2025

by Akash Verma

· 2,749 Views · 1 Like

Turn SQL into Conversation: Natural Language Database Queries With MCP

Exemplifies how a dedicated MCP server that is able to access databases can enable LLMs to inspect them and offer their users useful pieces of information.

July 14, 2025

by Horatiu Dan

CORE

· 3,611 Views · 8 Likes

Building Resilient Go Apps: Mocking and Testing Database Error Responses

Learn to simulate Azure Cosmos DB errors in Go to test retry logic and error handling using custom transports and policies — no real service failures needed.

July 14, 2025

by Abhishek Gupta

CORE

· 2,459 Views · 3 Likes

Understanding Time Series Databases

Time series databases (TSDBs) are specialized database systems optimized for storing, retrieving, and analyzing chronological data.

July 14, 2025

by Nikhil Makhija

· 3,308 Views · 3 Likes

Vibe Coding: Conversational Software Development — Part 2: In Practice

AI is transforming software development in Part 2 of my 'Vibe Coding' series. Explore conversational coding, code generation in practice, and developer productivity.

July 14, 2025

by Gaurav Gaur

CORE

· 2,975 Views · 4 Likes

The 7 Biggest Cloud Misconfigurations That Hackers Love (and How to Fix Them)

Everyone's an admin until they shouldn't be. Ninety-nine percent of cloud identities have excessive permissions, making lateral movement trivial for attackers.

July 14, 2025

by David Iyanu Jonathan

· 3,704 Views · 5 Likes

Cloud Hardware Diagnostics for AI Workloads

AI workloads need reliable hardware. Cloud providers are developing intelligent diagnostics to predict, detect, and resolve GPU and server failures efficiently.

July 14, 2025

by Sam Prakash Bheri

· 2,580 Views

AI-Powered Ransomware and Malware Detection in Cloud Environments

AI is reshaping malware and ransomware detection in cloud environments. It reviews core detection models, highlights technical challenges, and discusses future directions

July 14, 2025

by Eugene Aryeetey

· 2,004 Views

My Dive into Local LLMs, Part 2: Taming Personal Finance with Homegrown AI (and Why Privacy Matters)

Turn your local LLM (Llama 3 + Ollama on Ubuntu) into a private finance analyzer that categorizes spending and generates insights—no cloud, full privacy.

July 14, 2025

by Praveen Chinnusamy

· 17,765 Views · 5 Likes

How to Reduce Technical Debt With Artificial Intelligence (AI)

Technical debt hampers growth and innovation. Smart AI integration in SaaS helps reduce it by improving code quality, scalability, and proactive maintenance.

July 11, 2025

by Karan Rawal

· 3,492 Views

MongoDB Change Streams and Go

Change streams allow you to subscribe to real-time updates in your MongoDB collections and databases. Learn how to work with change streams and Go.

July 11, 2025

by Ado Kukic

· 2,811 Views · 1 Like

Beyond the Glass Slab: How AI Voice Assistants are Morphing Into Our Real-Life JARVIS

Jarvis-like AI agents are becoming real. As someone building them at Alexa, I see Agentic AI handling tasks, making decisions, and more, reshaping how we live and work.

July 11, 2025

by Praveen Chinnusamy

· 10,951 Views · 3 Likes

When MySQL, PostgreSQL, and Oracle Argue: Doris JDBC Catalog Acts as the Peacemaker

What is this magical Doris JDBC Catalog? It's like a universal key in the data world, opening the doors to multiple databases with one key.

July 11, 2025

by Zen Hua

· 2,751 Views · 1 Like

Scaling Multi-Tenant Go Apps: Choosing the Right Database Partitioning Approach

Multi-tenant applications face a fundamental challenge: how to efficiently store and query data for tenants of vastly different sizes?

July 11, 2025

by Abhishek Gupta

CORE

· 1,687 Views · 2 Likes

Indexed Views in SQL Server: A Production DBA's Complete Guide

Learn indexed views in SQL Server from a seasoned DBA's perspective with practical examples, performance testing, and best practices for database optimization.

July 11, 2025

by arvind toorpu

CORE

· 4,972 Views · 4 Likes

Testing the MongoDB MCP Server Using SingleStore Kai

In this article, learn how to set up and test the MongoDB MCP Server with SingleStore Kai using MCPHost, notebooks, and real-time queries powered by Ollama LLM.

July 11, 2025

by Akmal Chaudhri

CORE

· 1,588 Views · 1 Like

The Latest Data Engineering Topics