Data Engineering Resources

Practical Steps to Secure the Software Supply Chain End to End

Learn practical steps to secure your software supply chain, including governance, development, deployment, identity management, and incident response.

July 18, 2025

by Karteek Kotamsetty

· 4,616 Views · 1 Like

From Drift to Discipline: Operating Model for Regaining Enterprise Cloud Control

Enterprises chasing AI and scale won’t win with more tools—they’ll win by turning infrastructure into standardized infrastructure-as-code.

July 18, 2025

by Aharon Twizer

· 3,468 Views

Implementing Event-Driven Systems With AWS Lambda and DynamoDB Streams

Learn how to build a real-time, event-driven system using AWS Lambda and DynamoDB Streams to process data changes automatically without polling or extra infra.

July 18, 2025

by Akash Verma

· 3,598 Views · 2 Likes

The Twelve-Factor Agents: Building Production-Ready LLM Applications

This article delves into the concept of the Twelve-Factor Agent, an architectural pattern designed to create robust, scalable, and maintainable applications.

July 17, 2025

by Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

· 5,937 Views · 4 Likes

Best Practices for Syncing Hive Data to Apache Doris : From Scenario Matching to Performance Tuning

Learn how to efficiently sync and analyze big data by combining Hive’s storage with Doris’s real-time analytics using various sync strategies and optimizations.

July 17, 2025

by Darren Xu

· 4,217 Views

Migrating Traditional Workloads From Classic Compute to Serverless Compute on Databricks

This tutorial explains the migration of Databricks workloads from Classic Compute to Serverless Compute for efficiency and cost effectiveness.

July 17, 2025

by Prasath Chetty Pandurangan

· 4,436 Views

Rapid AWS Prototyping With LocalStack: A Developer’s Guide to Building AWS PoCs Locally

This article shows how to use LocalStack to build and test AWS-based Proof of Concepts (PoCs) entirely on your local machine.

July 17, 2025

by Akash Verma

· 4,509 Views · 1 Like

AI-Powered Security for the Modern Software Supply Chain: Reinforcing Software Integrity in an Era of Autonomous Code and Expanding Risk

AI-driven tools enhance software supply chain security by automating threat detection, managing dependencies, and supporting compliance, while balancing risks.

July 17, 2025

by Akanksha Pathak

CORE

· 3,900 Views · 1 Like

Fraud Detection in Mobility Services With Apache Kafka and Flink

Mobility services like Uber and Grab use data streaming with Kafka and Flink for fraud detection by applying AI and machine learning in real-time.

July 17, 2025

by Kai Wähner

CORE

· 4,944 Views · 1 Like

Streamline Your ELT Workflow in Snowflake With Dynamic Tables and Medallion Design

Dynamic Tables in Snowflake bring declarative, incremental ELT. Define SQL + freshness target, and Snowflake handles the orchestration, no dbt or Airflow needed.

July 16, 2025

by Harshavardhan Yedla

· 4,631 Views · 1 Like

Engineering High-Scale Real Estate Listings Systems Using Golang, Part 1

Building scalable real estate listing systems requires integrating with inconsistent MLS data sources using RETS and RESO.

July 16, 2025

by Alireza Rahmani Khalili

CORE

· 3,787 Views · 1 Like

Making AI Faster: A Deep Dive Across Users, Developers, and Businesses

This article explores why faster AI matters and shares strategies across user, developer, and business perspectives to reduce latency and speed up delivery.

July 16, 2025

by Gunveer Gujral

· 3,539 Views · 3 Likes

Real-Time Webcam-Based Sign Language and Speech Bidirectional Translation System

AI-powered system enables real-time two-way translation between sign language and speech, improving communication for the deaf and hard-of-hearing.

July 16, 2025

by Srinivas Reddy Kosna

· 3,056 Views · 4 Likes

Data Ingestion: The Front Door to Modern Data Infrastructure

AWS offers a rich set of ingestion services. This guide provides industry use cases and a cheat sheet to help you choose the right one for your organization.

July 16, 2025

by Junaith Haja

· 2,772 Views · 3 Likes

ZapBot: Building a Chatbot With OpenAI and Gradio

Create your own AI-powered chatbot using GPT-4 and Gradio. This hands-on Python tutorial walks you through setup, code, UI, and secure API key handling.

July 16, 2025

by Vaibhavi Tiwari

· 3,134 Views · 1 Like

Compliance Automated Standard Solution (COMPASS), Part 8: Agentic AI Policy as Code for Compliance Automation With Prompt Declaration Language

Part of our compliance series—learn how Agentic AI and PDL help compliance teams turn natural language inputs into executable policy assessments at scale.

July 16, 2025

by Yuji Watanabe

· 2,288 Views · 1 Like

Decoding Database Speed: Essential Server Resources and Their Impact

Database performance hinges on CPU, memory, and storage. Monitoring and tuning these resources ensures efficient data ops like a kitchen running at peak speed.

July 15, 2025

by arvind toorpu

CORE

· 3,383 Views · 2 Likes

Dashboards Are Dead Weight Without Context: Why BI Needs More Than Visuals

BI engineers must build context-rich, code-driven dashboards that enable decisions. Actionable pipelines and semantic clarity are now the standard.

July 15, 2025

by Venkata Murali Krishna Nerusu

· 4,490 Views · 1 Like

Migrating SQL Failover Clusters Without Downtime: A Practical Guide

We migrated a live SQL failover cluster to new hardware using a rolling node replacement strategy—zero downtime, full validation, and no broken apps.

July 15, 2025

by Kishore Thota

· 3,156 Views · 2 Likes

Analysis of the Data Processing Framework of Pandas and Snowpark Pandas API

This is a process analysis of migrating existing Pandas workflows to an almost lift-and-shift approach using the Snowpark Pandas API to meet ever-growing data needs.

July 15, 2025

by Prasath Chetty Pandurangan

· 2,899 Views · 1 Like

The Latest Data Engineering Topics