Data Engineering Resources

Fraud Detection in Mobility Services With Apache Kafka and Flink

Mobility services like Uber and Grab use data streaming with Kafka and Flink for fraud detection by applying AI and machine learning in real-time.

July 17, 2025

by Kai Wähner

CORE

· 4,829 Views · 1 Like

Streamline Your ELT Workflow in Snowflake With Dynamic Tables and Medallion Design

Dynamic Tables in Snowflake bring declarative, incremental ELT. Define SQL + freshness target, and Snowflake handles the orchestration, no dbt or Airflow needed.

July 16, 2025

by Harshavardhan Yedla

· 4,180 Views · 1 Like

Engineering High-Scale Real Estate Listings Systems Using Golang, Part 1

Building scalable real estate listing systems requires integrating with inconsistent MLS data sources using RETS and RESO.

July 16, 2025

by Alireza Rahmani Khalili

CORE

· 3,571 Views · 1 Like

Making AI Faster: A Deep Dive Across Users, Developers, and Businesses

This article explores why faster AI matters and shares strategies across user, developer, and business perspectives to reduce latency and speed up delivery.

July 16, 2025

by Gunveer Gujral

· 3,275 Views · 3 Likes

Real-Time Webcam-Based Sign Language and Speech Bidirectional Translation System

AI-powered system enables real-time two-way translation between sign language and speech, improving communication for the deaf and hard-of-hearing.

July 16, 2025

by Srinivas Reddy Kosna

· 2,981 Views · 4 Likes

Data Ingestion: The Front Door to Modern Data Infrastructure

AWS offers a rich set of ingestion services. This guide provides industry use cases and a cheat sheet to help you choose the right one for your organization.

July 16, 2025

by Junaith Haja

· 2,672 Views · 3 Likes

ZapBot: Building a Chatbot With OpenAI and Gradio

Create your own AI-powered chatbot using GPT-4 and Gradio. This hands-on Python tutorial walks you through setup, code, UI, and secure API key handling.

July 16, 2025

by Vaibhavi Tiwari

· 2,972 Views · 1 Like

Compliance Automated Standard Solution (COMPASS), Part 8: Agentic AI Policy as Code for Compliance Automation With Prompt Declaration Language

Part of our compliance series—learn how Agentic AI and PDL help compliance teams turn natural language inputs into executable policy assessments at scale.

July 16, 2025

by Yuji Watanabe

· 2,176 Views · 1 Like

Decoding Database Speed: Essential Server Resources and Their Impact

Database performance hinges on CPU, memory, and storage. Monitoring and tuning these resources ensures efficient data ops like a kitchen running at peak speed.

July 15, 2025

by arvind toorpu

CORE

· 3,115 Views · 2 Likes

Dashboards Are Dead Weight Without Context: Why BI Needs More Than Visuals

BI engineers must build context-rich, code-driven dashboards that enable decisions. Actionable pipelines and semantic clarity are now the standard.

July 15, 2025

by Venkata Murali Krishna Nerusu

· 4,382 Views · 1 Like

Migrating SQL Failover Clusters Without Downtime: A Practical Guide

We migrated a live SQL failover cluster to new hardware using a rolling node replacement strategy—zero downtime, full validation, and no broken apps.

July 15, 2025

by Kishore Thota

· 2,934 Views · 2 Likes

Analysis of the Data Processing Framework of Pandas and Snowpark Pandas API

This is a process analysis of migrating existing Pandas workflows to an almost lift-and-shift approach using the Snowpark Pandas API to meet ever-growing data needs.

July 15, 2025

by Prasath Chetty Pandurangan

· 2,777 Views · 1 Like

How to Build a Real API Gateway With Spring Cloud Gateway and Eureka

Build a real-world API gateway using Spring Cloud Gateway and Eureka Server. Perfect for beginners looking to understand microservices architecture with working code.

July 15, 2025

by Vivek Rajyaguru

· 6,421 Views · 2 Likes

API Standards Are Data Standards

API standards are often created separately from an org's data standards. However, they are much more similar than many realize and, therefore, should be created together.

July 15, 2025

by Scott Sosna

CORE

· 4,209 Views · 2 Likes

Designing Configuration-Driven Apache Spark SQL ETL Jobs with Delta Lake CDC

Simplify complex ETL pipelines and enable scalable, maintainable data processing with Spark SQL and Delta Lake Change Data Capture.

July 14, 2025

by Janaki Ganapathi

· 2,091 Views

Testing Distributed Microservices Using XState

Learn how to use XState to model microservice workflows. Simplify testing, boost coverage, and debug visually using declarative state machines.

July 14, 2025

by Akash Verma

· 2,535 Views · 1 Like

Turn SQL into Conversation: Natural Language Database Queries With MCP

Exemplifies how a dedicated MCP server that is able to access databases can enable LLMs to inspect them and offer their users useful pieces of information.

July 14, 2025

by Horatiu Dan

CORE

· 3,379 Views · 8 Likes

Building Resilient Go Apps: Mocking and Testing Database Error Responses

Learn to simulate Azure Cosmos DB errors in Go to test retry logic and error handling using custom transports and policies — no real service failures needed.

July 14, 2025

by Abhishek Gupta

CORE

· 2,291 Views · 3 Likes

Understanding Time Series Databases

Time series databases (TSDBs) are specialized database systems optimized for storing, retrieving, and analyzing chronological data.

July 14, 2025

by Nikhil Makhija

· 2,962 Views · 3 Likes

Vibe Coding: Conversational Software Development — Part 2: In Practice

AI is transforming software development in Part 2 of my 'Vibe Coding' series. Explore conversational coding, code generation in practice, and developer productivity.

July 14, 2025

by Gaurav Gaur

CORE

· 2,785 Views · 4 Likes

The Latest Data Engineering Topics