Data Engineering Resources

Integrating Apache Kafka in KRaft Mode With RisingWave for Event Stream Processing

Install and configure the latest version of Apache Kafka on a single-node cluster running on Ubuntu-22.04, and subsequently integrate it with RisingWave.

August 23, 2024

by Gautam Goswami

CORE

· 4,696 Views · 2 Likes

Parent Document Retrieval (PDR): Useful Technique in RAG

PDR allows RAG to respond accurately and nuanced to complex queries. Explore more in this step-by-step implementation using LangChain, OpenAI, and more.

August 22, 2024

by Intiaz Shaik

· 6,746 Views · 2 Likes

Setting Up CORS and Integration on AWS API Gateway Using CloudFormation

Learn about the significance of CORS, authorization including bearer tokens, and the advantages of selecting optional methods in place of standard GET requests.

August 22, 2024

by Vijay Panwar

CORE

· 6,978 Views · 2 Likes

Protect Your Alerts: The Importance of Independent Incident Alert Management

Hosting your incident alert management system separately from your primary cloud services is crucial for ensuring operational resilience.

August 22, 2024

by Judit Sharon

· 4,403 Views · 2 Likes

Kotlin Coroutines and OpenTelemetry Tracing

Explore the underlying workings of OpenTelemetry and analyze the workings of @WithSpan in general as well as within the context of Kotlin Coroutines.

August 22, 2024

by Nicolas Fränkel

CORE

· 7,190 Views · 5 Likes

Day in the Life of a Developer With Google’s Gemini Code Assist: Part 1

Explore Gemini Code Assist, its usage, and its impact beyond basic code generation for phases like bootstrapping, development, testing, deployment, and more.

August 22, 2024

by Aakash Sharma

· 7,865 Views · 6 Likes

Multicluster Gateways With Kubernetes Gateway API

In this article, discover how to use the Gateway API spec to configure gateways for multicluster setup.

August 22, 2024

by Ravi Verma

· 7,628 Views · 2 Likes

Evolution of Governance Framework With AI

Understand how a governance framework can improve productivity and increase the satisfaction of the stakeholders through AI.

August 22, 2024

by KARANVEER ANAND

· 3,525 Views · 2 Likes

Improving Snowflake Performance by Mastering the Query Profile

Take an in-depth look at how Snowflake executes queries and maximize Query performance using these advanced techniques.

August 21, 2024

by John Ryan

· 6,741 Views · 2 Likes

Automatic 1111: Sketch-to-Image Workflow

With Stable Diffusion Web UI Automatic 1111, create custom workflows for converting Sketches into photo-realistic images using diffusion and ControlNet models.

August 21, 2024

by Tharakarama Reddy Yernapalli Sreenivasulu

· 4,529 Views · 5 Likes

Data Pipeline Techniques in Action

Take a deep dive into the architectural concepts of data pipelines along with a hands-on tutorial for implementation, demonstrating the concepts in action.

August 21, 2024

by Sohil Shah

· 6,478 Views · 2 Likes

Using SQS With JMS for Legacy Applications

In this article, take an in-depth look at the steps to integrate a legacy Java application with SQS through JMS.

August 21, 2024

by Balaji Nagarajan

· 9,717 Views · 9 Likes

Unlock AI Power: Generate JSON With GPT-4 and Node.js for Ultimate App Integration

Discover how to effectively integrate OpenAI’s API with Node.js, generating JSON outputs that enhance app functionality and leverage advanced AI technology.

August 21, 2024

by Pradeep kumar Saraswathi

· 5,470 Views · 2 Likes

Preventing and Fixing Bad Data in Event Streams: Part 2

The techniques we use for dealing with bad data in event streams differ from those in the batch world. Here, learn more about overcoming bad data in streaming.

August 21, 2024

by Adam Bellemare

· 6,147 Views · 3 Likes

Four Essential Tips for Building a Robust REST API in Java

Enhance your Java REST API with consistent resource naming, maintainable versioning, robust security, and proper exception handling.

August 21, 2024

by Otavio Santana

CORE

· 11,910 Views · 8 Likes

Snowflake and dbt Integration to Enable Advanced Analytics

This guide describes how to leverage dbt and Snowflake to build high-quality data pipelines, focusing on data and analytics rather than on infrastructure.

August 20, 2024

by Rajesh Remala

· 4,047 Views · 2 Likes

OpenShift Data Foundation on IBM Cloud Using Terraform, Part 1: Deployment

Automated workloads often have difficulty in using CLI or UI way to deploy ODF. Explore the use of Terraform for deploying ODF which can be used for automation.

August 20, 2024

by Aayush Subramaniam

· 4,466 Views · 2 Likes

Mastering PUE for Unmatched Data Center Performance

PUE can make or break data center energy efficiency. Improving this figure will help your facility lower costs and become more sustainable.

August 20, 2024

by Emily Newton

· 4,027 Views · 1 Like

How Data Encryption Can Simplify Infrastructure Architecture

Learn how to leverage data encryption capabilities to improve security standards, simplify infrastructure architecture, and enhance developer velocity.

August 20, 2024

by Lulu Cheng

· 4,046 Views · 2 Likes

In-Depth Understanding of Vector Search for RAG and Generative AI Applications

Learn how to enhance LLMs with RAG using vector-based search and embeddings for more accurate responses, featuring tools like Azure AI Search.

August 20, 2024

by Mohammed Talib

· 7,807 Views · 6 Likes

The Latest Data Engineering Topics