Data Engineering Resources

Custom SCD2 Implementation Using PySpark

Learn to implement Slowly Changing Dimension Type 2 (SCD2) in a data warehouse for tracking historical data, ensuring data integrity, and enabling scalability.

January 13, 2025

by Abhishek Trehan

· 1,780 Views

Process Mining Key Elements

This article introduces process mining, explaining its key elements and practical applications for discovering and analyzing workflows using event data.

January 13, 2025

by Puneet Malhotra

· 2,890 Views · 5 Likes

Enterprise RAG in Amazon Bedrock: Introduction to KnowledgeBases

In this post, learn details of Amazon Bedrock KnowledgeBases capability and how to build a serverless RAG pipeline using KnowledgeBases.

January 13, 2025

by Suvoraj Biswas

· 2,442 Views · 1 Like

Security in the Age of AI: Challenges and Best Practices

Key security challenges in AI and strategies to protect systems, from data breaches to adversarial attacks, to ensure robust and secure AI integration.

January 13, 2025

by Akanksha Pathak

CORE

· 2,831 Views · 3 Likes

API Logic and Workflow Integration

Generative natural language rules allow a simple way to extend an API for the developer experience in connecting transaction systems to a workflow.

January 13, 2025

by Tyler Band

· 4,258 Views · 4 Likes

A Tool to Ease Your Transition From Oracle PL/SQLs to Couchbase JavaScript UDFs

In this article, take a deep dive into an AI Translator tool used to automate the conversion of PL/SQL to JSUDF.

January 13, 2025

by Gaurav Jayaraj

· 2,455 Views · 2 Likes

Streamlining HashiCorp Cloud Platform (HCP) Deployments With Terraform

Use Terraform to automate and manage the HashiCorp Cloud Platform (HCP) for streamlining deployment and reducing operational complexity.

January 13, 2025

by Sai Sandeep Ogety

CORE

· 2,644 Views · 13 Likes

Revolutionizing Catalog Management for Data Lakehouse With Polaris Catalog

Achieve cross-query-engine interoperability and zero-data-copy architecture with Polaris Catalog, enhance data management, and streamline workflows.

January 10, 2025

by Dipankar Saha

· 2,885 Views · 1 Like

Optimizing SQL Server Performance With AI: Automating Query Optimization and Predictive Maintenance

AI enhances SQL Server performance through query optimization and predictive maintenance, boosts efficiency, reduces latency, and improves system scalability.

January 10, 2025

by Vijay Panwar

CORE

· 4,921 Views · 2 Likes

Top 5 Books to Enhance Your Software Design Skills in 2025

Discover the top 5 books to master software design, from simplicity to clean code, testing, and Java best practices — plus a bonus on design patterns.

January 10, 2025

by Otavio Santana

CORE

· 5,905 Views · 7 Likes

Maximizing AI Agents for Seamless DevOps and Cloud Success

Harness the power of AI agents in DevOps and cloud delivery for success. Explore how to use AI in DevOps and cloud environments in this article.

January 9, 2025

by Marija Naumovska

CORE

· 4,841 Views · 4 Likes

Quantum Machine Learning for Large-Scale Data-Intensive Applications

This article examines how QML can harness the principles of quantum mechanics to achieve significant computational advantages over classical approaches.

January 9, 2025

by Manasi Sharma

· 5,482 Views · 1 Like

A Hands-On Guide to Enable Amazon GuardDuty on AWS Account

This hands-on guide will help you enable Amazon GuardDuty via the AWS Console in order to monitor and detect security threats in your AWS environment.

January 9, 2025

by Sai Sandeep Ogety

CORE

· 3,806 Views · 13 Likes

Mastering Retrieval Augmented Generation: From Fundamentals to Advanced Techniques

RAG enhances LLMs by linking them to external retrieval systems, while advanced RAG addresses challenges in query formulation, embedding generation, and accuracy.

January 9, 2025

by Salman Khan

CORE

· 6,349 Views · 3 Likes

Formulating a Robust Strategy for Storage in Amazon Relational Database Service PostgreSQL Deployments

This article explores understanding the key components of RDS storage, optimizing these storage factors, and capping storage growth by utilizing retention periods.

January 9, 2025

by Vivek Singh

· 3,349 Views · 2 Likes

AI Governance: Building Ethical and Transparent Systems for the Future

This article takes a deep dive into AI governance, including insights surrounding its challenges, frameworks, standards, and more.

January 9, 2025

by Sukanya Konatam

· 6,308 Views · 2 Likes

Scholcast: Generating Academic Paper Summaries With AI-Driven Audio

Scholcast is a Python package that converts academic papers into detailed summaries in A/V format using AI, making it easier to stay updated with the research.

January 8, 2025

by Sabarna Choudhuri

· 3,090 Views · 1 Like

Visualizing the Air Quality Index (AQI) Across Punjab, Pakistan, and India

Learn how to use IQAir data for visualizing air quality in Punjab to gain insight into the widespread impact of air pollution across the region.

January 8, 2025

by Akmal Chaudhri

CORE

· 3,148 Views

Top 5 Key Features of Apache Iceberg for Modern Data Lakes

Gain insight into key Iceberg features such as time travel, schema evolution, partition evolution, and ACID transactions with clear SQL examples and diagrams.

January 8, 2025

by Rajanikantarao Vellaturi

· 3,091 Views · 2 Likes

API and Security: From IT to Cyber

Learn how to protect your APIs beyond the basics. This article explores the risks, but also the solutions, which are not necessarily limited to the API Manager.

January 8, 2025

by Thomas Jardinet

CORE

· 4,779 Views · 1 Like

The Latest Data Engineering Topics