Data Engineering Resources

Data Management With PostgreSQL Partitioning and pg_partman

PostgreSQL table partitioning divides large tables into smaller segments to boost query performance, simplify maintenance, and enable scalable data management.

February 7, 2025

by arvind toorpu

CORE

· 4,198 Views · 1 Like

Exploring Operator, OpenAI’s New AI Agent

OpenAI has introduced its new AI agent, Operator. See how it can streamline software testing, reduce manual effort, and enhance accuracy with automation.

February 6, 2025

by Kailash Pathak

CORE

· 5,503 Views · 3 Likes

Community Over Code Keynotes Stress Open Source's Vital Role

This year's Community Over Code keynotes highlighted a maturing open-source ecosystem tackling new challenges around security, regulation, and scalability.

February 6, 2025

by Brian Proffitt

· 3,632 Views · 5 Likes

Best Practices for Scaling Kafka-Based Workloads

Kafka is a famous technology with a lot of great features and capabilities. This article explains Kafka producer and consumer configurations best practices.

February 6, 2025

by Narendra Lakshmana gowda

· 6,090 Views · 1 Like

How to Maximize the Azure Cosmos DB Availability

In this article, we will learn how to improve the Azure Cosmos database availability on applications that have zero tolerance for downtime.

February 6, 2025

by Prem Kumar Mani

· 4,572 Views

Build a URL Shortener With Neon, Azure Serverless Functions

Know how to build a simple and production-ready URL shortener running on Azure with serverless API using tools like Neon, Azure serverless, and Node.js.

February 6, 2025

by Bobur Umurzokov

· 5,806 Views · 2 Likes

How Spring Boot Starters Integrate With Your Project

The article provides an overview of Spring Boot starters, which are pre-packaged sets of dependencies that simplify adding functionality to Spring Boot applications.

February 5, 2025

by Taras Ivashchuk

· 6,132 Views · 3 Likes

The Role of DQ Checks in Data Pipelines

Why are DQ checks critical for every data pipeline, and what are some of the different types of DQ alerts you can set up to enhance the reliability of your pipeline?

February 5, 2025

by Ajay Krishnan Prabhakaran

· 5,593 Views

Demystifying Sorting Assertions With AssertJ

Know the options for sorting assertions in AssertJ, including ascending/descending order, multiple attributes, null handling, and case-insensitive sorting.

February 5, 2025

by Arnošt Havelka

CORE

· 8,828 Views · 3 Likes

Detecting Patterns in Event Streams With FlinkCEP

Patterns play a big role in ESP as they help to spot important sequences or behaviors in data that keep flowing nonstop.

February 5, 2025

by Gautam Goswami

CORE

· 6,630 Views · 2 Likes

How to Use AI With WordPress

In this article, you will learn how to use AI with WordPress, as we list a couple of plugins and tools that will be useful for automating tasks and workflows.

February 5, 2025

by Nitish Singh

CORE

· 3,941 Views

Mastering the Transition: From Amazon EMR to EMR on EKS

Optimize your data processing by seamlessly transitioning from Amazon EMR to EMR on EKS. Discover best practices and tips for a smooth migration.

February 5, 2025

by Satrajit Basu

CORE

· 7,278 Views · 4 Likes

AI Regulation in the U.S.: Navigating Post-EO 14110

The Trump administration revoked EO 14110, shifting the U.S. toward a market-driven AI strategy to spur innovation and investment.

February 5, 2025

by Frederic Jacquet

CORE

· 4,620 Views · 5 Likes

Build a Stateless Microservice With GitHub Copilot in VSCode

In this guide, we will build a stateless Flask Microservice using GitHub Copilot, with its benefits, speed, and where to be cautious when using it.

February 5, 2025

by Sibasis Padhi

· 8,813 Views · 7 Likes

Teradata Performance and Skew Prevention Tips

Use unique identifiers such as Teradata Primary Index to evenly distribute data across AMPs to improve database performance by 70%.

February 4, 2025

by Sudheer Kumar Lagisetty

· 4,432 Views · 10 Likes

Leveraging Neo4j for Effective Identity Access Management

Discussing Neo4j for IAM data modeling, the use cases, the guidelines and best practices, and how Neo4j can be integrated with existing IAM systems and tools.

February 4, 2025

by Ramesh Manickavel

CORE

· 4,545 Views · 2 Likes

A Guide to Automating AWS Infrastructure Deployment

AWS CloudFormation automates infrastructure with code, supporting Elastic Beanstalk, serverless, EC2, and more for consistent deployments.

February 4, 2025

by Praveen Kumar Thopalle

· 5,472 Views · 2 Likes

Data Governance Essentials: Policies and Procedures (Part 6)

Learn how data quality, policies, and procedures strengthen data governance by ensuring accuracy, compliance, and security for better decision-making.

February 4, 2025

by Sukanya Konatam

· 5,776 Views · 4 Likes

Processing Cloud Data With DuckDB And AWS S3

DuckDB's ability to read data directly from cloud storage, such as AWS S3, makes it particularly powerful for modern data architectures.

February 4, 2025

by Anil Kumar Moka

· 14,561 Views

SOC 2 Made Simple: Your Guide to Certification

This guide breaks down the SOC 2 Type 2 certification process into practical steps, from preparation to the audit, with some tips and tools to make the journey smoother.

February 4, 2025

by Roman Misyurin

· 4,130 Views · 3 Likes

The Latest Data Engineering Topics