Data Engineering Resources

Using Flink, Kafka, and NiFi for Real-Time Airport Arrivals and Departures

Learn to build a streaming application using the best of NiFi, Kafka, and Flink for event-driven apps. OpenSky networks rest feeds provide all the data.

September 26, 2024

by Timothy Spann

· 4,705 Views · 5 Likes

Backpressure in Distributed Systems

Learn more about backpressure, a technique in distributed systems to prevent overload and cascading failures by controlling the flow of requests.

September 26, 2024

by Rajesh Kumar Pandey

· 9,695 Views · 3 Likes

Building a Zero Trust API With ASP.NET Core: A Developer’s Guide

Secure your ASP.NET Core API with a Zero Trust model by authenticating every request, enforcing least privilege, and continuously monitoring to protect against threats.

September 25, 2024

by Aneesh Gopalakrishnan

· 6,527 Views · 4 Likes

Concurrency Issues in SQL and Distributed Systems

What happens if the first transaction must be canceled later for any reason? How should this affect the other transaction? Read on to understand!

September 25, 2024

by Adam Furmanek

CORE

· 5,513 Views · 6 Likes

Automating Data Pipelines With Snowflake: Leveraging DBT and Airflow Orchestration Frameworks for ETL/ELT Processes

This article explores the automation of data pipelines using Snowflake, dbt, and Airflow, detailing best practices for efficient data processing and orchestration.

September 25, 2024

by Harshavardhan Yedla

· 5,860 Views · 2 Likes

Feature Engineering Transforming Predictive Models

Delve into the transformative power of feature engineering in applied machine learning, and learn how carefully crafted features can elevate your models.

September 25, 2024

by Sundeep Goud Katta

· 4,356 Views · 3 Likes

Securing Your Azure Kubernetes Services Cluster

Learn about securing an Azure Kubernetes cluster with the principle of least privilege as a top priority integrated with Microsoft Entra and Kubernetes RBAC.

September 25, 2024

by Rafael Natali

· 5,166 Views · 2 Likes

Probabilistic Graphical Models: A Gentle Introduction

Explore this guide to probabilistic graphical models, which represent probability distributions and capture conditional independence structures using graphs.

September 25, 2024

by Salman Khan

CORE

· 31,243 Views · 49 Likes

The Future of Application Security: Empowering Developers in the AI Era

Explore the future of application security in the AI era. Learn how developers can integrate security into their workflow and leverage AI responsibly.

September 25, 2024

by Tom Smith

CORE

· 3,193 Views · 1 Like

How to Convert Files to Thumbnail Images in Java

This article explores the benefits of and solutions for programmatically generating thumbnail images for documents passing through our file processing systems.

September 25, 2024

by Brian O'Neill

CORE

· 7,556 Views · 3 Likes

Simplifying Multi-Cloud Observability With Open Source

As cloud workloads become increasingly sophisticated, adopting an open-source and vendor-agnostic approach is key to streamlining organizational operations.

September 25, 2024

by Siri Varma Vegiraju

CORE

· 20,066 Views · 2 Likes

Optimizing External Secrets Operator Traffic

Learn how to efficiently implement External Secrets Operator in your Kubernetes cluster to reap the benefits of External Secrets Management systems.

September 25, 2024

by Upasana Sharma

· 8,536 Views · 8 Likes

Oracle NoSQL Database: A Comprehensive Guide for Developers

In this guide to the Oracle NoSQL database, learn more about what it is, explore developer-friendly tools, how to get started, common challenges, and more.

September 24, 2024

by Otavio Santana

CORE

· 7,947 Views · 1 Like

Java Bean Validation: Applying Constraints Programmatically

Learn how to configure a Java bean validator in order to apply constraints programmatically within a Spring application.

September 24, 2024

by Maksim Kren

· 9,000 Views · 3 Likes

Efficient Transformer Attention for GenAI

Explore this overview of recent research focused on building efficient transformer models by reducing the computational complexity of self-attention.

September 24, 2024

by Anurag Paul

· 4,431 Views · 2 Likes

Practical Generators in Go 1.23 for Database Pagination

Learn techniques that have broad applications across various domains, helping to optimize system resource usage and improve overall application responsiveness.

September 24, 2024

by Nikita Melnikov

· 27,105 Views · 1 Like

What Is a Data Pipeline?

In this post, we examine data pipelines, explore their benefits, compare them to other data processes, and discuss various implementation methods.

Updated September 24, 2024

by Garrett Alley

· 53,404 Views · 18 Likes

Human Introspection With Machine Intelligence

Explore the abductive logic programming (ALP) approach within computational logic and how it can be utilized to enhance our cognitive processes and behaviors.

September 24, 2024

by vuppulapati Chandra Sekhar Naidu

· 2,375 Views · 3 Likes

Supporting Offline Mode: Key Strategies and Benefits for iOS Mobile Apps

Supporting offline mode in mobile apps enhances user experience by enabling use without an internet connection through data caching, synchronization, and more.

September 23, 2024

by Namaswi Chandarana

· 4,186 Views · 1 Like

The Future Speaks: Real-Time AI Voice Agents With Ultra-Low Latency

Open-source AI voice agents now achieve real-time, low-latency responses with under 500ms turnaround time. What will you create with it?

September 23, 2024

by Waqas Saeed

· 4,692 Views · 2 Likes

The Latest Data Engineering Topics