Data Engineering Resources

Real-Time GenAI With RAG Using Apache Kafka and Flink to Prevent Hallucinations

Learn about context-specific real-time Generative AI (GenAI) with Retrieval Augmentation Generation (RAG) using Kafka and Flink to prevent hallucinations.

September 11, 2024

by Kai Wähner

CORE

· 4,834 Views · 2 Likes

Automatic 1111: Custom Sketch-to-Image API

Develop a custom Sketch-to-Image API for converting hand-drawn/digital sketches into photorealistic images using stable diffusion models powered by ControlNet.

September 10, 2024

by Tharakarama Reddy Yernapalli Sreenivasulu

· 5,725 Views · 6 Likes

Exploring Real-Time Data Ingestion Into Snowflake Using CockroachDB, Redpanda, and Kafka Connect

Explore Kafka Connect as a solution to stream changefeeds into Snowflake for greater control over how messages are delivered to Snowflake.

September 10, 2024

by Artem Ervits

CORE

· 5,985 Views · 1 Like

REST and HTTP Semantics

Discover if the benefits of using PUT and POST as described in the RFC outweigh whatever reasons for using them in a way that is not RFC-compliant.

September 10, 2024

by Greg Hall

· 6,539 Views · 3 Likes

Explainable AI (XAI): How Developers Build Trust and Transparency in AI Systems

XAI is essential for making AI systems transparent, trustworthy, and ethically compliant, with developers playing a crucial role.

September 10, 2024

by Suri (thammuio)

CORE

· 48,762 Views · 3 Likes

Transforming Continuous Delivery With Feature Flags

Integrate feature flags into CI/CD pipelines to mitigate deployment risks, ensure safer feature rollouts, and facilitate feature testing.

September 10, 2024

by Josephine Eskaline Joyce

CORE

· 7,638 Views · 8 Likes

Accelerate Your Journey to a Modern Data Platform Using Coalesce

This article will identify key challenges organizations face today in managing data platforms, and explore how advanced ETL tools can address these challenges.

September 10, 2024

by Asia Banu Shaik

· 5,947 Views · 5 Likes

Digital Garden

This article is dedicated to the concept of a digital garden — the philosophy of publicly maintaining personal notes.

September 10, 2024

by Anton Belyaev

· 5,699 Views · 2 Likes

How Can the Scalability of MySQL Be Improved for BenchmarkSQL TPC-C Testing?

Can MySQL maintain performance without collapse at tens of thousands of concurrent connections in low-conflict scenarios of BenchmarkSQL TPC-C testing?

September 10, 2024

by Bin Wang

· 3,479 Views · 2 Likes

Mastering Multi-Cloud and Edge Data Synchronization: A Retail Use Case With KubeMQ’s Java SDK

Explore how KubeMQ’s open-source Java SDK provides a powerful solution for enterprises looking to manage data across complex multi-cloud and edge environments.

September 10, 2024

by John Vester

CORE

· 48,255 Views · 3 Likes

Top Security Flaws Hiding in Your Code Right Now and How to Fix Them

This article examines the three most common injection attack types: SQL injection, Deserialization Injection, and Logging Injection — and discusses ways to prevent them.

September 10, 2024

by Jonathan Vila

· 5,011 Views · 3 Likes

Zero to AI Hero, Part 2: Understanding Plugins in Semantic Kernel, A Deep Dive With Examples

In this tutorial, explore one of the "gears" of Semantic Kernel — plugins — and discover the work they are capable of beyond chit-chatting.

September 10, 2024

by Aneesh Gopalakrishnan

· 5,431 Views · 2 Likes

Spring Boot GoT: Game of Trace!

Start with the basics of Spring Boot logs, and learn how to configure logging, test log severity, and adjust log levels using the LoggingSystem class.

September 9, 2024

by Fernando Boaglio

· 6,789 Views · 2 Likes

Model-Driven Development and Testing

Explore how open-source ApiLogicServer (ALS) and GenAI-powered development impact the deployment of microservice applications.

September 9, 2024

by Tyler Band

· 4,883 Views · 8 Likes

What Are Logging Levels?

Many things can break, but when the logging breaks, then we are completely lost. In this article, understand how to log efficiently in distributed systems.

September 9, 2024

by Adam Furmanek

CORE

· 4,607 Views · 3 Likes

Data Storage Formats for Big Data Analytics: Performance and Cost Implications of Parquet, Avro, and ORC

This article compares the performance and cost efficiency of three storage formats Parquet, Avro, and ORC on Google Cloud Platform.

September 9, 2024

by Rahul Sarabu

· 8,696 Views · 4 Likes

Mainframe Modernization: A Comprehensive Technical Blueprint

This post explores a practical hybrid approach to mainframe workload migration to the cloud using both IBM Cloud and AWS examples.

September 6, 2024

by Manjit Chakraborty

· 7,006 Views · 1 Like

Comparison of Apache Astro and Airflow

Compare Astro and Apache Airflow, explaining their architecture, features, scalability, usability, community support, and integration capabilities.

September 6, 2024

by Krishnamurty Raju Mudunuru

· 11,625 Views · 2 Likes

Principles of Modern Data Infrastructure

Explore principles of modern data infrastructure such as scalability, high availability, speed, security, maintainability, efficiency, and developer experience.

September 6, 2024

by Joe Zhou

· 5,631 Views · 1 Like

DRY Your Apache APISIX Config

DRY is an important principle in software development. This post will show you how to apply it to Apache APISIX configuration.

September 6, 2024

by Nicolas Fränkel

· 4,531 Views · 4 Likes

The Latest Data Engineering Topics