Data Engineering Resources

Why Cloud Matters: Building Global, Scalable Microservices

A hands-on experience: Learn how to build a Book Management Catalog Microservice using Java, Helidon, Jakarta NoSQL, and Oracle Cloud.

February 17, 2025

by Otavio Santana

CORE

· 2,836 Views · 3 Likes

Redis as a Primary Database for Complex Applications

This article aims to answer one question: How can Redis be used as a primary database for complex applications that need to store data in multiple formats?

February 14, 2025

by Mohammed Talib

· 5,944 Views · 4 Likes

Optimizing Database Performance in Middleware Applications

Optimize database interactions in middleware with caching, parallelism, write batching, and connection pooling to boost performance and scalability effectively.

February 14, 2025

by karandeep johar

· 5,615 Views · 5 Likes

ETL Generation Using GenAI

Learn about how GenAI automates ETL pipelines, generates code, adapts to schema changes, and improves data processes with speed, efficiency, and precision.

February 14, 2025

by Ramesh Daddala

· 3,792 Views · 2 Likes

Unlocking Local AI: Build RAG Apps Without Cloud or API Keys

In this tutorial, we will use Chipper, an open-source framework that simplifies building local RAG applications without cloud dependencies or API keys.

February 14, 2025

by Santhosh Vijayabaskar

CORE

· 3,528 Views · 1 Like

Git Tags vs AWS Tags: A Tag-Tastic Showdown!

The ultimate showdown between Git Tags and AWS Tags, two contenders with nothing in common except their love for the word "tag." Let's see them go head-to-head.

February 14, 2025

by Raghava Dittakavi

CORE

· 3,747 Views

AI Summarization: Extractive and Abstractive Techniques

Learn how to create an AI-powered summarization tool using Hugging Face and OpenAI, combining extractive and abstractive methods for concise, accurate results.

February 13, 2025

by Vaibhavi Tiwari

· 2,535 Views · 1 Like

How to Backup SQL Server RDS to an S3 Bucket

Learn how to back up your SQL Server RDS database directly to an Amazon S3 bucket using msdb.dbo.rds_backup_database, with detailed examples.

February 13, 2025

by arvind toorpu

· 3,274 Views

Idempotency in Distributed Systems: When and Why It Matters

This article explains idempotency in distributed systems and ensuring consistent results regardless of multiple executions, with implementation and challenges.

February 13, 2025

by Sandeep Kumar Gond

· 2,036 Views

Goose Migrations for Smooth Database Changes

Migrate safely: Goose automates DB changes, version control, and rollbacks. Minimal downtime — ideal for CI/CD and high load.

February 12, 2025

by Ilia Ivankin

· 3,341 Views

Loading XML into MongoDB

Learn how to export XML data to MongoDB using SmartXML ETL tools, simplifying the process and ensuring efficient data handling and storage.

February 12, 2025

by Luca Sanders

· 3,552 Views

The Right ETL Architecture for Multi-Source Data Integration

Dedicated ETL pipelines are easy to set up but hard to scale, while common pipelines offer efficiency at the cost of complexity. Know which one to choose.

February 12, 2025

by Murat Balkan

CORE

· 3,806 Views · 1 Like

Cloud Security Is a Data Problem

As cloud data evolves, we need to learn how data integration, AI, and machine learning help mitigate risks in complex cloud environments and prevent breaches.

February 12, 2025

by Ryan Henrich

· 3,377 Views

Upcoming DZone Events

This article highlights DZone’s upcoming events, which offer insights into key industry trends and opportunities for professionals to learn, discuss, and grow.

Updated February 12, 2025

by Alanna Lovejoy

· 5,084 Views · 4 Likes

Your Network, Your Rules: Take Charge With Own DNS

Set up your own DNS nameserver using the same open-source tools that power today's state-of-the-art public DNS nameservers.

February 12, 2025

by Mahesh Veerabathiran

· 9,587 Views · 2 Likes

How to Reorganize and Rebuild Indexes in MS SQL?

In this article, we'll explore how to reorganize and rebuild indexes in MS SQL Server to address fragmentation and improve database performance.

February 12, 2025

by Priyanka Chauhan

· 2,961 Views · 1 Like

A Comprehensive Guide to Building and Debugging Apache Doris

Learn how to compile and debug Apache Doris from source. This guide also covers setup, dependencies, troubleshooting, and debugging tips for developers.

February 12, 2025

by haijun huang

· 2,611 Views · 1 Like

USA PATRIOT Act vs SecNumCloud: Which Model for the Future?

The USA PATRIOT Act grants U.S. data access abroad, while French SecNumCloud secures European sovereignty. The challenge: balancing security and sovereignty.

February 12, 2025

by Frederic Jacquet

CORE

· 3,233 Views

System Design of an Audio Streaming Service

In this article, we will go over the audio streaming system and key design considerations to address functional and non-functional requirements.

February 11, 2025

by Gitanjali Sahoo

· 4,715 Views · 4 Likes

From Zero to Scale With AWS Serverless

Serverless scales effortlessly from zero to peak, with AWS tools like DynamoDB and Lambda enabling cost-efficient, auto-scaling apps for data and real-time workflows.

February 11, 2025

by Ravi Laudya

· 4,528 Views · 1 Like

The Latest Data Engineering Topics