Data Engineering Resources

Idempotency in Distributed Systems: When and Why It Matters

This article explains idempotency in distributed systems and ensuring consistent results regardless of multiple executions, with implementation and challenges.

February 13, 2025

by Sandeep Kumar Gond

· 5,069 Views

Goose Migrations for Smooth Database Changes

Migrate safely: Goose automates DB changes, version control, and rollbacks. Minimal downtime — ideal for CI/CD and high load.

February 12, 2025

by Ilia Ivankin

· 9,486 Views

Loading XML into MongoDB

Learn how to export XML data to MongoDB using SmartXML ETL tools, simplifying the process and ensuring efficient data handling and storage.

February 12, 2025

by Luca Sanders

· 6,689 Views · 1 Like

The Right ETL Architecture for Multi-Source Data Integration

Dedicated ETL pipelines are easy to set up but hard to scale, while common pipelines offer efficiency at the cost of complexity. Know which one to choose.

February 12, 2025

by Murat Balkan

CORE

· 6,019 Views · 1 Like

Cloud Security Is a Data Problem

As cloud data evolves, we need to learn how data integration, AI, and machine learning help mitigate risks in complex cloud environments and prevent breaches.

February 12, 2025

by Ryan Henrich

· 4,228 Views · 1 Like

Upcoming DZone Events

This article highlights DZone’s upcoming events, which offer insights into key industry trends and opportunities for professionals to learn, discuss, and grow.

Updated February 12, 2025

by Alanna Lovejoy

· 9,238 Views · 4 Likes

Your Network, Your Rules: Take Charge With Own DNS

Set up your own DNS nameserver using the same open-source tools that power today's state-of-the-art public DNS nameservers.

February 12, 2025

by Mahesh Veerabathiran

· 12,198 Views · 2 Likes

How to Reorganize and Rebuild Indexes in MS SQL?

In this article, we'll explore how to reorganize and rebuild indexes in MS SQL Server to address fragmentation and improve database performance.

February 12, 2025

by Priyanka Chauhan

· 6,974 Views · 1 Like

A Comprehensive Guide to Building and Debugging Apache Doris

Learn how to compile and debug Apache Doris from source. This guide also covers setup, dependencies, troubleshooting, and debugging tips for developers.

February 12, 2025

by Michael Hayden

· 4,521 Views · 2 Likes

USA PATRIOT Act vs SecNumCloud: Which Model for the Future?

The USA PATRIOT Act grants U.S. data access abroad, while French SecNumCloud secures European sovereignty. The challenge: balancing security and sovereignty.

February 12, 2025

by Frederic Jacquet

CORE

· 4,910 Views · 1 Like

System Design of an Audio Streaming Service

In this article, we will go over the audio streaming system and key design considerations to address functional and non-functional requirements.

February 11, 2025

by Gitanjali Sahoo

· 6,671 Views · 4 Likes

From Zero to Scale With AWS Serverless

Serverless scales effortlessly from zero to peak, with AWS tools like DynamoDB and Lambda enabling cost-efficient, auto-scaling apps for data and real-time workflows.

February 11, 2025

by Ravi Laudya

· 5,859 Views · 1 Like

Understanding AWS Karpenter for Kubernetes Auto-Scaling

AWS Karpenter is an auto-scaling solution for Kubernetes designed to optimize cloud architectures by offering rapid scaling and cost efficiency.

February 11, 2025

by Rajesh Gheware

CORE

· 4,686 Views · 3 Likes

Secrets Security Is the Most Important Issue For Mobile Apps

The latest OWASP Top 10 For Mobile list ranks credential mismanagement as the largest concern our applications face. Let's stop letting our secrets sprawl.

February 11, 2025

by Dwayne McDaniel

· 3,083 Views · 3 Likes

Chat Completion Models vs OpenAI Assistants API

In this article, you will learn the key differences between chat completion (like those provided via the Chat Completions endpoint) and OpenAI's Assistants API.

February 11, 2025

by Mohammed Talib

· 5,915 Views · 1 Like

SQL as the Backbone of Big Data and AI Powerhouses

SQL powers Big Data and AI with tools like BigQuery, remaining a cornerstone of data-driven innovation through its simplicity and adaptability.

February 11, 2025

by Medha Gupta

· 3,483 Views · 1 Like

Perfecting CRUD Functionality in NextJS

CRUD operations are essential for data management. Learn how to manage data processing, React Query, pagination, and more efficiently with insights and examples.

February 11, 2025

by Olena Vlasenko

· 6,363 Views · 5 Likes

Indexed View for Aggregating Metrics

In this article, we will learn how to utilize the Indexed View capability of Azure SQL Server for tuning query performance and reducing latency.

February 10, 2025

by Ankur Peshin

· 4,359 Views · 1 Like

An In-Depth Guide to Threads in OpenAI Assistants API

Learn how to use OpenAI's Assistants API to manage threads and messages — create, list, retrieve, modify, delete — plus handle files, metadata, and more.

February 10, 2025

by Mohammed Talib

· 10,764 Views · 4 Likes

Spring Data Neo4j: How to Update an Entity

Several options for updating data with a Spring application. This post walks through some examples using the default save method, custom queries, and projections.

February 10, 2025

by Jennifer Reif

CORE

· 7,615 Views · 2 Likes

The Latest Data Engineering Topics