Data Engineering Resources

How Midjourney and Other Diffusion Models Create Images from Random Noise

Diffusion models, such as Midjourney and DALL-E 2 can produce stunning images, but their training always begins with complete noise.

April 22, 2023

by Rostyslav Demush

· 7,272 Views · 2 Likes

MLOps for Beginners: Getting Started With MLOps

New to MLOps? Learn the basics of machine learning with this MLOps for beginners guide.

April 22, 2023

by Atulpriya Sharma

· 4,796 Views · 1 Like

Producing and Consuming Avro Messages With Redpanda Schema Registry

In this article, learn how to use Redpanda’s built-in schema registry to produce and consume Apache Avro messages in five simple steps.

April 21, 2023

by Dunith Dhanushka

CORE

· 4,783 Views · 1 Like

Lambda Architecture: A Powerful Approach to Data Engineering

In this article, we will explore the Lambda Architecture in detail, understanding its key concepts, benefits, and challenges.

April 21, 2023

by Amlan Patnaik

· 5,373 Views · 6 Likes

Why Use Kong API Gateway

Unlock the power of your API infrastructure with the Kong API Gateway. Let's explore the top reasons why you should choose Kong for API management needs.

April 21, 2023

by Vaibhav Pathak

· 7,720 Views · 4 Likes

How to Stream Sensor Data to Apache Pinot for Real Time Analysis

Learn how to use Arduino, MQTT, Kafka, and Apache Pinot to stream environmental data for real-time access to sensor data.

April 21, 2023

by David G. Simmons

CORE

· 6,800 Views · 8 Likes

Test Tool Tutorial: A Comprehensive Guide With Examples and Best Practices

This test tool tutorial will give you a detailed overview of the test tools, why you should use them, their benefits, and much more.

April 21, 2023

by Harshit Paul

· 6,474 Views · 1 Like

IoT Testing: How It Works and Why You Can’t Do Without It

This article explains IoT testing, how to do it correctly, which challenges you can face along the way, and why.

April 21, 2023

by Inna Martyniuk

· 4,088 Views · 2 Likes

Consumer Group in Kafka [Video Tutorials]: Partitions and Consumers

In this video tutorial series, take a deeper look at the consumer group in Kafka with varying numbers of partitions and consumers.

April 21, 2023

by Ram N

· 6,089 Views · 2 Likes

CockroachDB TIL: Volume 12

Learn to find partial indexes, identify the DB Console URL with SQL, experiment with a lightweight pooler called pgcat, handle client disconnects, and more.

April 20, 2023

by Artem Ervits

CORE

· 3,890 Views · 1 Like

Data Encryption: Benefits, Types, and Methods

This post explains data encryption and lists its benefits, types and the common encryption methods found in different tools.

April 20, 2023

by Alex Tray

· 7,303 Views · 1 Like

Maximizing the Potential of LLMs: Using Vector Databases

How can you work around the token limit problem in Large Language Models like GPT-4 using Vector Embedding Databases and LangChain?

April 20, 2023

by Roger Oriol

· 4,941 Views · 2 Likes

Static Proxies or Rotating Proxies: Which One Should You Choose?

This article explains the differences between Static proxies and Rotating proxies. Let's find out what they are.

April 20, 2023

by Hanzala Ali

· 5,497 Views · 1 Like

Benefits of React V18: A Comprehensive Guide

This article will cover three key features of React v18: automatic batching, transition, and suspense on the server.

April 20, 2023

by Beste Bayhan

· 7,248 Views · 4 Likes

Kafka: The Basics

Kafka is a powerful tool for building streaming architectures. The article serves as an introduction to both the technology and associated data producers and consumers.

April 20, 2023

by Pavel Micka

· 8,177 Views · 6 Likes

AWS: Pushing Jakarta EE Full Platform Applications to the Cloud

In this article, readers will learn how to deploy more complex Jakarta EE applications as serverless services with AWS Fargate.

April 20, 2023

by Nicolas Duminil

CORE

· 8,286 Views · 5 Likes

Recording Badge Scans in Apache Pinot

Use a real-time analytics database to record NFC badge scans in part 1 of this 3-part project involving environmental data, real-time notifications, and more.

April 19, 2023

by David G. Simmons

CORE

· 4,824 Views · 5 Likes

Integrate Apache Spark and QuestDB for Time-Series Analytics

In this tutorial, we will explore how a Spark Application can load and save data from/to QuestDB and provide practical advice to achieve the best performance.

April 19, 2023

by Imre Aranyosi

· 4,237 Views · 8 Likes

Apache Kafka in Java [Video Tutorials]: Architecture and Simple Consumer/Producer

Explore a video tutorial series regarding Apache Kafka in Java as you explore Kafka architecture, components, producer callbacks, and more.

April 19, 2023

by Ram N

· 5,859 Views · 2 Likes

Introduction to Elasticsearch

Elasticsearch is a highly scalable and distributed search and analytics engine designed to handle large volumes of structured, semi-structured, and unstructured data.

April 19, 2023

by Rama Krishna Panguluri

· 13,299 Views · 11 Likes

The Latest Data Engineering Topics