Data Engineering Resources

The 4 R’s of Pipeline Reliability: Designing Data Systems That Last

In this article, learn how the 4 R’s — robust architecture, resumability, recoverability, and redundancy — enhance reliability in AI and ML data pipelines.

May 1, 2025

by Sidhant bendre

· 5,036 Views · 2 Likes

Unlocking AI Coding Assistants Part 3: Generating Diagrams, Open API Specs, And Test Data

Learn how to leverage AI assistants like LLMs to speed up development tasks such as generating UML diagrams, OpenAI specs, test data, and code.

May 1, 2025

by Gunter Rotsaert

CORE

· 6,347 Views · 1 Like

Google Cloud Document AI Basics

This simple example shows how to use a custom extractor in Google's Doc AI to process W-2s and use a PDF as part of the context to Gemini.

April 30, 2025

by Imran Burki

· 4,681 Views · 2 Likes

Docker Model Runner: Streamlining AI Deployment for Developers

Docker Model Runner is a tool introduced to simplify running and testing AI models locally, integrating seamlessly into existing workflows.

April 30, 2025

by Ram Chandra Sachan

· 23,842 Views · 18 Likes

Fixing Common Oracle Database Problems

In this article, we will discuss the issues we commonly face while working with Oracle databases and explore ways to solve them.

April 30, 2025

by Dhaval Patolia

· 6,563 Views · 1 Like

Setting Up Data Pipelines With Snowflake Dynamic Tables

This is a walkthrough of the process for building an automated data pipeline with dynamic table capabilities in Snowflake for various refresh frequencies.

April 30, 2025

by Prasath Chetty Pandurangan

· 5,016 Views

Develop a Reverse Proxy With Caching in Go

Learn to build a caching reverse proxy in Go with the standard library, featuring HTTP forwarding, in-memory caching with TTL, and compression handling.

April 30, 2025

by Rez Moss

· 3,522 Views · 2 Likes

Java’s Next Act: Native Speed for a Cloud-Native World

The Graal Stack reinvents Java for the cloud era, combining GraalVM, Micronaut and GraalOS to deliver ultra-fast, lightweight, and serverless-ready applications.

April 30, 2025

by Graziano Casto

CORE

· 8,106 Views · 7 Likes

Building Scalable and Resilient Data Pipelines With Apache Airflow

Learn to build scalable, fault-tolerant, and observable data pipelines with Apache Airflow, focusing on real-time insights and custom reporting for enterprise SaaS.

April 30, 2025

by Ramya K R Vuyyuru

· 4,660 Views · 3 Likes

The Modern Data Stack Is Overrated — Here’s What Works

The modern data stack is overrated. What works? Fewer tools, clear ownership, and systems your team actually understands.

April 29, 2025

by Rahul .

· 3,984 Views · 2 Likes

Scaling InfluxDB for High-Volume Reporting With Continuous Queries (CQs)

In this article, we improved InfluxDB query performance by using Continuous Queries to pre-aggregate high-volume Kafka data for faster, efficient reporting.

April 29, 2025

by Venkata Ashok Kumar Boyina

· 3,272 Views · 3 Likes

On-Call That Doesn’t Suck: A Guide for Data Engineers

Data pipelines don’t fail silently; they make a lot of noise. The question is, are you listening to the signal or drowning in noise?

April 29, 2025

by Tulika Bhatt

· 3,271 Views · 3 Likes

Unlocking AI Coding Assistants Part 2: Generating Code

Examine the effectiveness of AI coding assistants, highlight their potential and limitations in generating javadoc, names, and performing small coding tasks.

April 29, 2025

by Gunter Rotsaert

CORE

· 2,776 Views · 2 Likes

Build an MCP Server Using Go to Connect AI Agents With Databases

Learn to build an MCP server using Go to connect AI agents with Azure Cosmos DB, enabling seamless database interactions with Go SDK and mcp-go tools.

April 29, 2025

by Abhishek Gupta

CORE

· 3,755 Views · 3 Likes

The Transformative Power of Artificial Intelligence in Cloud Security

Explore the evolving role of AI in cloud security, the benefits it brings, and the challenges it helps organizations overcome.

April 29, 2025

by Shaik Abdul Kareem

· 13,689 Views · 2 Likes

The Role of Artificial Intelligence in Climate Change Mitigation

AI’s role in tackling climate change is set to grow significantly, driving innovative solutions through deeper analysis and machine learning.

April 29, 2025

by Dennis Helfer

· 1,670 Views · 1 Like

How To Replicate Oracle Data to BigQuery With Google Cloud Datastream

Set up Oracle 19c to BigQuery replication using Google Cloud Datastream, covering prerequisites, networking, CDC setup, and job validation.

April 29, 2025

by Karteek Kotamsetty

· 19,014 Views · 3 Likes

Filtering Messages With Azure Content Safety and Spring AI

Learn how to use Azure Content Safety with Spring AI to detect harmful input and prevent unsafe messages from reaching your chat model.

April 28, 2025

by Tamas Kiss

· 3,690 Views · 1 Like

Building Data Pipelines With Jira API

Automate Jira workflows via API with Python: tickets on pipeline failures, JQL reports, and data sync. Start with failed transfers and expand to dashboards and testing.

April 28, 2025

by Saeed Ashif Ahmed

· 4,626 Views · 1 Like

Doris: Unifying SQL Dialects for a Seamless Data Query Ecosystem

Apache Doris breaks this barrier with its powerful SQL dialect compatibility and builds a unified data query ecosystem for users.

April 28, 2025

by Darren Xu

· 3,346 Views

The Latest Data Engineering Topics