Data Engineering Resources

Cloud Cost Optimization: New Strategies for the AI Era

Explore new strategies for cloud cost optimization in the AI era, featuring insights from Stacklet CTO Kapil Thangavelu on FinOps, open source, and automation.

October 8, 2024

by Tom Smith

CORE

· 5,052 Views · 1 Like

Understanding the Differences Between Rate Limiting, Debouncing, and Throttling

Learn the differences between rate limiting, debouncing, and throttling, with examples and best practices for managing function execution.

October 8, 2024

by Charly Poly

· 3,313 Views · 3 Likes

PostgreSQL Everywhere and for Everything

PostgreSQL is versatile and can successfully handle less popular SQL scenarios and workflows that don’t use SQL at all. Let's see where Postgres shines.

October 8, 2024

by Adam Furmanek

CORE

· 6,953 Views · 9 Likes

Secret Management and Rotation

Learn about automating KMS key rotation for asymmetric keys to raise the security bar by replacing keys at an interval.

October 8, 2024

by Pranav Kumar Chaudhary

· 3,639 Views · 2 Likes

The Shift Left Architecture: From Batch and Lakehouse to Data Streaming

Explore the Shift Left Architecture using data streaming (Kafka/Flink) that enables data products for DWH, data lake, and lakehouses like Snowflake/Databricks.

October 8, 2024

by Kai Wähner

CORE

· 3,243 Views · 1 Like

Mastering AI Agents: How Agentic Design Patterns Make Agents Smarter

Discover how Agentic Design Patterns can transform your AI workflows by breaking down complex tasks into manageable, specialized AI agents.

October 7, 2024

by Noorain Panjwani

· 3,723 Views · 4 Likes

Integrate Spring Boot With jOOQ, Liquibase, and Testcontainers

Get started with jOOQ, Liquibase, and Testcontainers: Create a basic Spring Boot application and integrate the aforementioned techniques including a test setup.

October 7, 2024

by Gunter Rotsaert

CORE

· 4,868 Views · 1 Like

Natural Language SQL Query Application

Walk through building a powerful and intuitive web application that converts natural language into SQL queries and retrieves data from a PostgreSQL database.

October 7, 2024

by Balusamy Nachiappan

· 4,142 Views · 2 Likes

In-House Model Serving Infrastructure for GPU Flexibility

In this article, learn how to create a GPU SKU-agnostic serving infrastructure for flexibility, efficiency, and resilience.

October 7, 2024

by Suruchi Shah

· 3,913 Views · 1 Like

Low Code AI Agent Using Kumologica and Anthropic AI for Customer Feedback Sentiment Analysis

The article covers the design and implementation of a low code AI agent developed using Kumologica and Anthropic AI for customer feedback sentiment analysis.

October 7, 2024

by Pranav K

CORE

· 1,820 Views · 2 Likes

The Disruptive Potential of On-Device Large Language Models

Explore the advancements of on-device AI and how it will revolutionize our daily interactions with technology, making our lives more efficient and connected.

October 7, 2024

by Rishab Mehra

· 1,998 Views · 2 Likes

Curating Efficient Distributed Application Runtime (Dapr) Workflows

With Dapr and its powerful abstractions, writing resilient workflows is easy. In this article, we will understand how to write one with an example.

October 4, 2024

by Siri Varma Vegiraju

· 6,280 Views · 4 Likes

How to Build a RAG-Powered Chatbot With Google Gemini and MyScaleDB

Learn to construct a RAG-powered chatbot using Google Gemini models and MyScaleDB, including installing dependencies, preparing data, generating embeddings,etc.

October 4, 2024

by Usama Jamil

· 2,979 Views · 2 Likes

Enhanced API Security: Fine-Grained Access Control Using OPA and Kong Gateway

Kong secures APIs and OPA handles policy decisions. This custom plugin caches user data, minimizing database checks and ensuring fast, accurate access control.

October 4, 2024

by Amrutha Paladugu

· 4,716 Views · 4 Likes

How to Create a Search Engine and Algorithm With ClickHouse and Snowflake

Explore a step-by-step guide to developing a search engine and algorithm using Clickhouse, an open-source data warehousing solution.

October 3, 2024

by Akash Mukherjee

· 3,560 Views · 2 Likes

Implementing Polling With RxJS

Learn how to implement polling with RxJS for efficient, repetitive API calls, and optimize performance with exponential backoff and advanced RxJS operators.

October 3, 2024

by Aliaksandr Mirankou

· 5,409 Views · 1 Like

Why Doesn't Postgres Use My Index?

It may happen that we create an index, but PostgreSQL doesn’t use it. What could be the reason, and how can we fix it?

October 3, 2024

by Adam Furmanek

CORE

· 4,147 Views · 7 Likes

Supercharge Your Search With GenAI: From Simple Queries to Smarter Results

Boost your search with GenAI models for smarter, more relevant results using query expansion and reformulation using Chain of Thought prompting.

October 3, 2024

by Suruchi Shah

· 2,738 Views · 4 Likes

A Comprehensive Guide to Database Sharding: Building Scalable Systems

Explore an in-depth guide to database sharding: what it is, its types, how to select shard keys, and route queries for building scalable systems.

October 2, 2024

by Suleiman Dibirov

CORE

· 6,108 Views · 6 Likes

Hyperscale Data Centers Ignite Construction Surge

Demand is growing for robust computing. However, hyperscale data center construction faces challenges due to technology demands, shortages, and regulations.

October 2, 2024

by Emily Newton

· 4,036 Views · 1 Like

The Latest Data Engineering Topics