Data Engineering Resources

Web Scraping With LLMs, ScrapeGraphAI, and LangChain

In this article, learn how to use LLMs for web scraping with ScrapeGraphAI, LangChain, and Pydantic. This guide covers setup, configuration, and data extraction

January 31, 2025

by Juveria dalvi

· 4,899 Views

Creating a Service for Sensitive Data With Spring and Redis

In some cases, one cannot store user-sensitive data permanently. Let's create a simple application that handles sensitive data leveraging Spring and Redis.

January 31, 2025

by Alexander Rumyantsev

· 5,023 Views · 2 Likes

CAP and PACELC Theorems in Plain English

Data engineering fundamentals explained: the CAP theorem and PACELC — two frameworks for understanding tradeoffs in distributed systems.

January 31, 2025

by luminousmen _

· 2,670 Views

How to Build a Data Dashboard Prototype With Generative AI

This article is a tutorial that shows how to build a data dashboard to visualize book reading data taken from goodreads.com.

January 30, 2025

by Jo Stichbury

CORE

· 4,486 Views · 1 Like

Develop Microservices Using Azure Functions, API Management

Explore how to easily build cloud-deployed microservices using serverless technology with Azure Functions and connect with APIs standardized in API Management.

January 30, 2025

by Ankit Masrani

· 7,146 Views · 3 Likes

Building a Machine Learning Pipeline Using PySpark

This article discusses building an efficient ML pipeline with PySpark, covering data loading, preprocessing, model training, and evaluation for large datasets.

January 30, 2025

by Abhishek Trehan

· 5,219 Views · 1 Like

Bridging Graphviz and Cytoscape.js for Interactive Graphs

Making Graphviz static digraphs interactive and compatible with Cytoscape by converting DOT format graphs into Cytoscape JSON using Python.

January 30, 2025

by Puneet Malhotra

· 4,287 Views · 1 Like

SmartXML: An Alternative to XPath for Complex XML Files

We'll discuss SmartXML, an XPath alternative for parsing complex XML files, converting them to SQL, and loading the results into a database seamlessly.

January 30, 2025

by Luca Sanders

· 5,700 Views · 2 Likes

Commonly Occurring Errors in Microsoft Graph Integrations and How To Troubleshoot Them (Part 7)

This article discusses retrieving attachments from various SharePoint lists using the SharePoint API and Microsoft Graph API.

January 30, 2025

by Constantin Kwiatkowski

· 4,559 Views

Scaling Read Your Own Writes Consistency

This article is intended for distributed systems practitioners looking to understand and implement Read Your Own Writes consistency in production environments.

January 30, 2025

by Ganapathy Subramanian Ramachandran

· 2,816 Views

Passing JSON Variables in Azure Pipelines

Learn how to handle JSON variables in Azure DevOps pipelines, avoid escaping issues, and ensure seamless API integration with proper normalization techniques.

January 29, 2025

by Mohammed Basil

· 8,082 Views · 2 Likes

Predicting Diabetes Types: A Deep Learning Approach

Machine learning analysis of diabetes in young Indians: Deep learning vs. XGBoost (64.75% vs. 74% accuracy) using health and lifestyle data.

January 29, 2025

by Aditya Karnam Gururaj Rao

· 3,230 Views · 1 Like

Metal and the Simulated Annealing Algorithm

The Simulated Annealing algorithm described in this article demonstrates its effectiveness as a powerful tool for finding optimal solutions to complex problems.

January 29, 2025

by Vitaly Kuznetsov (Ippolitov)

· 3,212 Views · 2 Likes

Expert Guide: How to Slash Cloud Cost in 2025

In this guide, we will zero in on high-impact tips for cloud cost optimization. Cost is still a hurdle as many companies onboard to cloud computing.

January 29, 2025

by Milavkumar Shah

· 5,720 Views · 1 Like

Gemini 2.0 Flash (Experimental): A Deep Dive for Developers

Gemini 2.0 Flash — In the blog, let us see the top 10 abilities of the LLM and how developers can utilize it to develop various applications.

January 29, 2025

by Akash Lomas

· 4,022 Views · 1 Like

Scrape Amazon Product Reviews With Python

Let's learn how we can implement Python and Python scripts to scrape the Amazon website in an ethical way to extract product review data.

January 29, 2025

by Juveria dalvi

· 2,817 Views · 2 Likes

Implement RAG With PGVector, LangChain4j, and Ollama

Apply vector search and RAG experiments to enhance query results and optimize data storage for text embeddings, specifically with Bruce Springsteen's album data.

January 28, 2025

by Gunter Rotsaert

CORE

· 3,015 Views · 2 Likes

Using Spring AI to Generate Images With OpenAI's DALL-E 3

Integrate Spring AI with OpenAI's DALL-E 3 to generate images. Set up Spring Boot, configure the API integration, and customize settings easily.

January 28, 2025

by Danil Temnikov

CORE

· 10,968 Views · 12 Likes

Understanding Inference Time Compute

Inference time compute is a crucial factor in the deployment of machine learning models, where performance, efficiency, and user experience are key.

January 28, 2025

by Bhala Ranganathan

CORE

· 17,051 Views · 4 Likes

How Apache Flink and Apache Paimon Influence Data Streaming

Apache Flink is a crucial component of Apache Paimon since it offers the real-time processing power that enhances Paimon's strong consistency and storage features.

January 28, 2025

by Gautam Goswami

CORE

· 5,853 Views · 4 Likes

The Latest Data Engineering Topics