Data Engineering Resources

Talk to Your BigQuery Data Using Claude Desktop

Build an MCP server that lets Claude's desktop connect to your BigQuery data and converse with your data in natural language.

August 20, 2025

by Pavan Vemuri

CORE

· 5,571 Views · 9 Likes

Getting Started With PyIceberg: A Pythonic Approach to Managing Apache Iceberg Tables

If you're an ML Engineer and want to quickly run a feature engineering in your model, PyIceberg provides a lightweight implementation.

August 20, 2025

by Junaith Haja

· 2,522 Views · 3 Likes

Containerized Intelligence: Running LLMs at Scale Using Docker and Kubernetes

Deploy powerful LLMs easily using Docker and Kubernetes. Scale with GPUs, fine-tune models, and automate inference workloads effortlessly.

August 20, 2025

by Anjan Kumar Ayyadapu

· 12,041 Views · 5 Likes

How to Program a Quantum Computer: A Beginner's Guide

Build your first quantum programs using IBM Quantum Composer. Learn how qubits work, flip states with logic gates, and harness superposition to output 2 values.

August 19, 2025

by Kory Becker

· 3,316 Views · 5 Likes

Data Engineering for AI-Native Architectures: Designing Scalable, Cost-Optimized Data Pipelines to Power GenAI, Agentic AI, and Real-Time Insights

AI-native data platforms demand more than upgrades — learn how teams are building scalable systems for RAG, streaming, embeddings, and orchestration.

August 19, 2025

by Abhishek Gupta

CORE

· 3,380 Views · 2 Likes

What We Learned Migrating to a Pub/Sub Architecture: Real-World Case Studies from High-Traffic Systems

Kafka-powered migration of an e-commerce platform—focused on scalability, fault tolerance, and clean event-driven design.

August 19, 2025

by Ravi Teja Thutari

CORE

· 8,673 Views · 4 Likes

Building SQLGenie: A Natural Language to SQL Query Generator with LLM Integration

This article explores the journey of building SQLGenie, testing various Language Models (LLMs) to find the optimal solution for converting natural language to SQL queries

August 19, 2025

by Sangame Krishnamani

· 4,284 Views · 5 Likes

Agile AI Agents

Can ChatGPT’s Agent Mode really handle agile team tasks? I tested real use cases to see what it can do, what it misses, and where it needs improvement.

August 19, 2025

by Stefan Wolpers

CORE

· 1,942 Views · 3 Likes

Building a Secure and Unified Data Platform

Learn to build a secure GCP stack using a custom VPC, VM with private access, managed Cloud SQL PostgreSQL, and BigQuery for real-time federated queries.

August 19, 2025

by Vani Gundu

· 6,045 Views · 2 Likes

A Retrospective on GenAI Token Consumption and the Role of Caching

Caching is an important technique for enhancing the performance and cost efficiency of diverse cloud native applications, including modern GenAI applications.

August 19, 2025

by Bhala Ranganathan

CORE

· 3,180 Views · 1 Like

What’s Wrong With Data Validation — and How It Relates to the Liskov Substitution Principle

Most systems assume data is valid, causing redundant or missing checks. This article suggests modeling validated data as a separate type.

August 18, 2025

by Mykola Haliullin

· 2,129 Views · 3 Likes

Prompt-Based ETL: Automating SQL Generation for Data Movement With LLMs

Prompt-based ETL uses LLMs to convert plain English into validated, schema-aware SQL, automating data transformation and enabling self-serve analytics.

August 18, 2025

by Saisuman Singamsetty

· 6,988 Views · 4 Likes

Real-Time Analytics Using Zero-ETL for MySQL

Learn how to set up a zero-ETL integration from Amazon RDS for MySQL to Amazon Redshift using AWS CLI for real-time analytics without complex pipelines.

August 18, 2025

by Harpreet Kaur Chawla

· 2,001 Views · 2 Likes

Logging MCP Protocol When Using stdio, Part II

Learn different ways to log the MCP JSON protocol over stdio, with an intro to MCP and a deep dive into its key components and their nuances.

August 18, 2025

by Raghuraman Ramaswamy

CORE

· 3,490 Views · 5 Likes

Building AI Agents With .NET: A Practical Guide

Build autonomous, goal-driven AI agents in .NET using OpenAI’s GPT models. Learn how to integrate perception, reasoning, action, and learning in C#.

August 18, 2025

by Devdas Gupta

· 6,942 Views · 4 Likes

React Server Components in Next.js 15: A Deep Dive

RSC moves logic to the server for faster loads, less JS, and cleaner code. No useEffect, no DOM access. Start small, scale later.

August 18, 2025

by Yusup Izripov

· 4,423 Views · 4 Likes

Architecting Compound AI Systems for Scalable Enterprise Workflows

Learn how to architect modular and secure agentic workflows using compound AI systems for scalable enterprise automation.

August 18, 2025

by Srinivas Sandiri

· 2,409 Views · 2 Likes

My First Practical Agentic App: Using Firebase and Generative AI to Automate Office Tasks

A virtual office assistant that streamlines everyday office duties — drafting emails, summarizing meetings, etc., driven by Generative AI and Firebase.

August 15, 2025

by Lakshmi Narayana Rasalay

· 2,206 Views · 3 Likes

Green DevOps: Building Sustainable Pipelines and Energy-Aware Cloud Deployments

This article explores how teams are measuring energy usage, optimizing pipelines, and deploying eco-conscious applications using modern tools and techniques.

August 15, 2025

by Gideon Ali

· 3,095 Views · 2 Likes

How to Architect a Compliant Cloud for Healthcare Clients (Azure Edition)

We share how we built a secure, compliant cloud on Azure for healthcare clients using native services like RBAC, CMK encryption, and policy enforcement.

August 15, 2025

by Kishore Thota

· 3,553 Views · 3 Likes

The Latest Data Engineering Topics