Data Engineering Resources

extended Berkeley Packet Filter (eBPF) for Cloud Computing

Explore eBPF, a technology that executes codes in the Linux kernel and is necessary for K8s network observability/monitoring, auditing, and traffic routing.

August 7, 2024

by Gaurav Shekhar

· 17,269 Views · 3 Likes

JMeter Plugin HTTP Simple Table Server (STS) In-Depth

Take an in-depth look to discover some of the possibilities of using the HTTP Simple Table Server a JMeter Plugin in this short manual.

August 6, 2024

by Vincent DABURON

· 6,497 Views · 4 Likes

Building an LLM-Powered Product to Learn the AI Stack: Part 1

Building a Meal Planning Bot to Deep Dive into LLMs, Product Development, and Cloud Skills. From Data Generation, to Model Training, to Deployment and App building

August 6, 2024

by Indrajit Bhattacharya

· 7,730 Views · 10 Likes

Handling Schema Versioning and Updates in Event Streaming Platforms Without Schema Registries

Learn how to handle schema versioning and updates in Kafka and other event streaming platforms without using schema registries through custom deserializers.

August 6, 2024

by Sanket Panhale

· 24,133 Views · 1 Like

Oracle: Migrate PDB to Another Database

Explore different approaches to move the pluggable database from source to target database such as plug/unplug, data pump, refreshable clone and DBCA relocate.

August 6, 2024

by Deval Parikh

· 12,546 Views · 4 Likes

Develop With OCI Real-Time Speech Transcription and Oracle Database NL2SQL/Select AI To Speak With Your Data

Use a combination of real-time speech-to-text, natural language to SQL, and Gen AI, to talk with your data just as you would to a person and in real-time.

August 6, 2024

by Paul Parkinson

· 5,211 Views · 4 Likes

Idempotency in Data Pipelines: Overview

Idempotency is an important concept in data engineering, particularly when working with distributed systems or databases. Learn more here.

August 6, 2024

by Harsh Daiya

CORE

· 3,992 Views · 4 Likes

Go Serverless: Unleash Next-Gen Computing

Learn how serverless architecture, with its event-driven architecture, auto-scaling ability, and security walls, frees devs from infrastructure management woes.

August 5, 2024

by Aditya Kumar

· 6,543 Views · 3 Likes

Scaling Prometheus With Thanos

Learn how to scale Prometheus using Thanos for long-term storage, global view metrics, and improved performance in large-scale Kubernetes environments.

August 5, 2024

by Ritesh Sonawane

· 4,172 Views · 5 Likes

Reimagining AI: Ensuring Trust, Security, and Ethical Use

How can we trust AI in our daily lives? Explore the balance between innovation and safety in the age of intelligent systems.

August 5, 2024

by Saigurudatta Pamulaparthyvenkata

· 4,809 Views · 2 Likes

Building an IoT-based Waste Management System: A Software Architect's Guide

Explore how IoT waste management optimizes collection, reduces costs, and enhances sustainability using smart bins, mobile apps, and analytics.

August 5, 2024

by Arun Pandey

CORE

· 6,313 Views · 2 Likes

Streaming Data Joins: A Deep Dive Into Real-Time Data Enrichment

Unravel the complexities of streaming data joins in this guide covering key concepts, design, and best practices for optimal real-time data enrichment.

August 5, 2024

by Preet Shah

· 5,371 Views · 4 Likes

Building a To-Do List With MongoDB and Golang

In this article, consider how you can write your small service in Go in a couple of hours and put everything in a database.

August 5, 2024

by Ilia Ivankin

· 2,821 Views · 2 Likes

Free Tier API With Apache APISIX

A free tier is a must for any API service provider worth its salt. In this post learn how to configure such a free tier with Apache APISIX.

August 5, 2024

by Nicolas Fränkel

· 2,487 Views · 1 Like

Harnessing the Power of AWS Aurora for Scalable and Reliable Databases

Delve into the benefits of AWS Aurora and explore a real-life example of how it is used in an online social media platform.

August 5, 2024

by Vijay Panwar

CORE

· 3,912 Views · 3 Likes

Overview of Classical Time Series Analysis: Techniques, Applications, and Models

Explore what makes time series analysis unique and the specialized techniques and models needed to understand and predict future patterns or trends.

August 5, 2024

by Salman Khan

CORE

· 55,454 Views · 30 Likes

Upgrading Spark Pipelines Code: A Comprehensive Guide

Discuss the strategic importance of Spark code upgrades and explore an introduction to a powerful toolkit designed to streamline this process: Scalafix.

August 5, 2024

by Suri (thammuio)

CORE

· 26,074 Views · 1 Like

Automation Resilience: The Hidden Lesson of the CrowdStrike Debacle

Read here to discover the as-yet uncovered culprit in this massive hunt for the responsible parties for the recent CrowdStrike debacle: automation.

August 5, 2024

by Jason Bloomberg

· 2,210 Views · 4 Likes

Industrial IoT: Exploring Microcontrollers for Robust Applications

Let's explore the features, applications, and top choices for microcontrollers in industrial IoT, achieving reliability, efficiency, and security in harsh environments.

August 2, 2024

by Carsten Rhod Gregersen

· 5,508 Views · 1 Like

Criticality in Data Stream Processing and a Few Effective Approaches

Opposite to traditional batch data processing technique, here processing works on the data as it is produced in real-time. Learn more in this post.

August 2, 2024

by Gautam Goswami

CORE

· 5,968 Views · 1 Like

The Latest Data Engineering Topics