Data Engineering Resources

Software Development Trends to Follow in 2025

In this blog, I will discuss key software development trends for 2025, from AI-powered tools to ethical AI, low-code platforms, and cloud-native advancements.

January 2, 2025

by Fawad Malik

· 6,260 Views · 6 Likes

Stop Being Afraid of Databases

Ensuring database reliability can be difficult. Our goal is to speed up development and minimize rollbacks. Achieving efficient processes requires database observability.

January 2, 2025

by Adam Furmanek

CORE

· 4,105 Views · 2 Likes

Kubernetes Deployments With DMZ Clusters: An Essential Guide

A DMZ cluster in Kubernetes secures public services from internal workloads, enhancing scalability, reducing attack surface, and ensuring controlled access.

January 2, 2025

by Sai Sandeep Ogety

CORE

· 5,331 Views · 13 Likes

Incremental Jobs and Data Quality Are On a Collision Course

Big data isn’t dead; it’s just going incremental. But bad things happen when uncontrolled changes collide with incremental jobs. Reacting to changes is a losing strategy.

January 1, 2025

by Jack Vanlightly

· 4,652 Views · 1 Like

Logical Reasoning in Network Problems

Online issues can be highly complex, requiring the capture of key information for a clear understanding. Through logical reasoning, the root cause was pinpointed.

January 1, 2025

by Bin Wang

· 6,220 Views · 3 Likes

Boosting Performance and Efficiency: Enhancing React Applications With GraphQL Over REST APIs

This article guides you through setting up a GraphQL server, defining schemas, and integrating with React, highlighting benefits and potential challenges.

January 1, 2025

by Shreyam Dutta Gupta

· 5,218 Views · 6 Likes

Driving RAG-Based AI Infrastructure

When paired with AI agents for workflow orchestration, RAG-based AI infrastructure excels in dynamic decision-making, analytics, and automation.

January 1, 2025

by Apurva Kumar

· 2,943 Views · 2 Likes

Data Lake vs. Data Warehouse vs. Data Lakehouse

The three data storage options and their pros and cons: the legacy data warehouse, the more recent data lake, and contemporary data lakehouse architectures.

December 31, 2024

by Noa Shavit

· 4,677 Views · 4 Likes

Streamlining Database Management: Running PostgreSQL in Docker Containers

You will learn how to run PostgreSQL in a Docker container with a portable, isolated, and resource-efficient database setup that is easy to manage and scalable.

December 31, 2024

by VARUNREDDY DEVIREDDY

· 5,680 Views · 3 Likes

Iceberg Catalogs: A Guide for Data Engineers

Learn about Apache Iceberg catalogs, their types, configurations, and the best-fit solutions for managing metadata and large datasets in different environments.

December 31, 2024

by Naresh Dulam

· 4,472 Views · 2 Likes

The Importance of Kubernetes in MLOps and Its Influence on Modern Businesses

The most critical challenges in MLOps involve constructing scalable and flexible infrastructure, to which Kubernetes is an efficient solution.

December 31, 2024

by Anupama Babu

· 4,219 Views · 1 Like

Enhancing DevOps in the Cloud With AI Innovation

Learn how AI-driven tools are transforming DevOps practices in cloud environments, improving automation, efficiency, and scalability by leveraging AI.

December 31, 2024

by Srinivas Reddy Bandarapu

· 3,819 Views · 3 Likes

Building a Receipt Scanner App With OCR, OpenAI, and PostgreSQL

In this article, you'll learn how to build a receipt scanner app using OCR for text extraction, OpenAI for processing, and PostgreSQL for efficient data storage.

December 31, 2024

by Balusamy Nachiappan

· 4,307 Views · 3 Likes

Agile Failure Has Strengthened, Not Weakened, Software

With psychological factors helping explain why humans struggle with Agile, interdisciplinary expertise can help save us from future software disasters.

December 31, 2024

by Junade Ali

· 3,264 Views · 2 Likes

Supercharge Your Coding Workflow With Ollama, LangChain, and RAG

Learn how to build a retrieval-augmented generation (RAG) code assistant using Ollama and LangChain to optimize your coding workflow.

December 30, 2024

by Santhosh Vijayabaskar

CORE

· 4,039 Views · 4 Likes

Setting Up a ScyllaDB Cluster on AWS Using Terraform

This article will teach you how to set up, deploy, manage, and scale ScyllaDB clusters on AWS using Terraform with this comprehensive step-by-step guide.

December 30, 2024

by Oleg Lipin

· 4,663 Views · 4 Likes

Bridging the Gap: Unlocking the Power of HDFS-Based Data Lakes With Streaming Databases

Big data technologies' quick development has highlighted the necessity of a smooth transition between real-time data analytics and batch processing systems.

December 30, 2024

by Gautam Goswami

CORE

· 2,499 Views · 1 Like

Getting Started With Snowflake Snowpark ML: A Step-by-Step Guide

This guide discusses setting up the Snowpark ML library, configuring your environment, and implementing a basic ML use case.

December 30, 2024

by Sevinthi Kali Sankar Nagarajan

· 2,485 Views · 2 Likes

Implementing CRUD Operations With NLP Using Microsoft.Extensions.AI

Learn how to implement CRUD operations using NLP in a .NET Web API application with the Microsoft.Extensions.AI library for a light management system.

December 30, 2024

by Baraneetharan Ramasamy

· 2,769 Views · 1 Like

AI in 2024: Year in Review and Predictions for 2025

The past year has been transformative for artificial intelligence. Let's review the major developments of 2024 and explore what the future might hold in 2025.

December 30, 2024

by Roger Oriol

· 1,674 Views

The Latest Data Engineering Topics