Data Engineering Resources

Handling Schema Versioning and Updates in Event Streaming Platforms Without Schema Registries

Learn how to handle schema versioning and updates in Kafka and other event streaming platforms without using schema registries through custom deserializers.

August 6, 2024

by Sanket Panhale

· 16,251 Views · 1 Like

Oracle: Migrate PDB to Another Database

Explore different approaches to move the pluggable database from source to target database such as plug/unplug, data pump, refreshable clone and DBCA relocate.

August 6, 2024

by Deval Parikh

· 4,201 Views · 2 Likes

Develop With OCI Real-Time Speech Transcription and Oracle Database NL2SQL/Select AI To Speak With Your Data

Use a combination of real-time speech-to-text, natural language to SQL, and Gen AI, to talk with your data just as you would to a person and in real-time.

August 6, 2024

by Paul Parkinson

· 2,648 Views · 3 Likes

Idempotency in Data Pipelines: Overview

Idempotency is an important concept in data engineering, particularly when working with distributed systems or databases. Learn more here.

August 6, 2024

by Harsh Daiya

CORE

· 3,004 Views · 4 Likes

Go Serverless: Unleash Next-Gen Computing

Learn how serverless architecture, with its event-driven architecture, auto-scaling ability, and security walls, frees devs from infrastructure management woes.

August 5, 2024

by Aditya Kumar

· 5,647 Views · 3 Likes

Scaling Prometheus With Thanos

Learn how to scale Prometheus using Thanos for long-term storage, global view metrics, and improved performance in large-scale Kubernetes environments.

August 5, 2024

by Ritesh Sonawane

· 3,220 Views · 5 Likes

Reimagining AI: Ensuring Trust, Security, and Ethical Use

How can we trust AI in our daily lives? Explore the balance between innovation and safety in the age of intelligent systems.

August 5, 2024

by Saigurudatta Pamulaparthyvenkata

· 3,512 Views · 2 Likes

Building an IoT-based Waste Management System: A Software Architect's Guide

Explore how IoT waste management optimizes collection, reduces costs, and enhances sustainability using smart bins, mobile apps, and analytics.

August 5, 2024

by Arun Pandey

CORE

· 4,540 Views · 2 Likes

Streaming Data Joins: A Deep Dive Into Real-Time Data Enrichment

Unravel the complexities of streaming data joins in this guide covering key concepts, design, and best practices for optimal real-time data enrichment.

August 5, 2024

by Preet Shah

· 3,783 Views · 4 Likes

Building a To-Do List With MongoDB and Golang

In this article, consider how you can write your small service in Go in a couple of hours and put everything in a database.

August 5, 2024

by Ilia Ivankin

· 2,004 Views · 1 Like

Free Tier API With Apache APISIX

A free tier is a must for any API service provider worth its salt. In this post learn how to configure such a free tier with Apache APISIX.

August 5, 2024

by Nicolas Fränkel

CORE

· 1,760 Views · 1 Like

Harnessing the Power of AWS Aurora for Scalable and Reliable Databases

Delve into the benefits of AWS Aurora and explore a real-life example of how it is used in an online social media platform.

August 5, 2024

by Vijay Panwar

CORE

· 2,429 Views · 3 Likes

Overview of Classical Time Series Analysis: Techniques, Applications, and Models

Explore what makes time series analysis unique and the specialized techniques and models needed to understand and predict future patterns or trends.

August 5, 2024

by Salman Khan

CORE

· 52,300 Views · 30 Likes

Upgrading Spark Pipelines Code: A Comprehensive Guide

Discuss the strategic importance of Spark code upgrades and explore an introduction to a powerful toolkit designed to streamline this process: Scalafix.

August 5, 2024

by Suri Nuthalapati

CORE

· 24,776 Views · 1 Like

Automation Resilience: The Hidden Lesson of the CrowdStrike Debacle

Read here to discover the as-yet uncovered culprit in this massive hunt for the responsible parties for the recent CrowdStrike debacle: automation.

August 5, 2024

by Jason Bloomberg

· 1,721 Views · 4 Likes

Industrial IoT: Exploring Microcontrollers for Robust Applications

Let's explore the features, applications, and top choices for microcontrollers in industrial IoT, achieving reliability, efficiency, and security in harsh environments.

August 2, 2024

by Carsten Rhod Gregersen

· 4,117 Views · 1 Like

Criticality in Data Stream Processing and a Few Effective Approaches

Opposite to traditional batch data processing technique, here processing works on the data as it is produced in real-time. Learn more in this post.

August 2, 2024

by Gautam Goswami

CORE

· 4,967 Views · 1 Like

Software Testing Errors to Look Out For (With Examples)

Explore the main purposes of software testing, which include identifying errors, deficiencies, or missing requirements with respect to actual requirements.

August 2, 2024

by Amir Ali

· 2,406 Views · 1 Like

The C̶a̶k̶e̶ User Location Is a Lie!!!

This post discusses the nuances surrounding location-based programming: various ways to access user location, how they can fail, and what to do about it.

August 2, 2024

by Austin Gil

CORE

· 5,547 Views · 2 Likes

The One-Pixel Threat: How Minuscule Changes Can Fool Deep Learning Systems

Learn about how a single pixel can compromise deep learning models from medical diagnostics to autonomous vehicles, and the challenges to securing our AI future.

August 1, 2024

by Frederic Jacquet

CORE

· 3,520 Views · 2 Likes

The Latest Data Engineering Topics