Data Engineering Resources

This article provides an analysis of the reasoning limitations that exist in today's LLMs and the approaches to overcome them.

February 4, 2025

by Bhala Ranganathan

CORE

· 6,976 Views · 3 Likes

Building a Full-Stack Resume Screening Application With AI

In this article, you'll learn how to build a smart resume analyzing application using the DeepSeek R1 open-source GenAI LLM model and save operation costs.

February 3, 2025

by Anujkumarsinh Donvir

CORE

· 10,308 Views · 9 Likes

Java Stream API: 3 Things Every Developer Should Know About

Java Stream API simplifies collection processing with lazy evaluation, parallelism, and functional programming. Use it to write cleaner, efficient, and scalable code.

February 3, 2025

by Danil Temnikov

CORE

· 15,963 Views · 16 Likes

Best Practices for API Rate Limits and Quotas

Rate limits protect your infrastructure, and quotas help you monetize your APIs. Both are key parts of a healthy API strategy.

February 3, 2025

by Derric Gilling

CORE

· 7,060 Views · 1 Like

Managing Distributed System Locks With Azure Storage

Locks in distributed systems give processes exclusive access to resources. Timeout and lease-based locks help prevent deadlocks and resource contention.

February 3, 2025

by Siri Varma Vegiraju

CORE

· 11,624 Views · 1 Like

All You Need to Know About Apache Spark

Apache Spark is a fast, open-source cluster computing framework for big data, supporting ML, SQL, and streaming. It’s scalable, efficient, and widely used.

February 3, 2025

by Abhishek Trehan

· 4,541 Views · 3 Likes

90% Cost Reduction With Prefix Caching for LLMs

Up to 70% of prompts in LLM applications are repetitive. Prefix caching can reduce inference costs by up to 90%, thus optimizing performance and saving money.

February 3, 2025

by Mahak Shah

· 5,049 Views

Building Neural Networks With Automatic Differentiation

In this post, we will write a basic DNN using simple Python. To do that, we need to understand automatic differentiation and then implement it in code.

February 3, 2025

by Mayank Gupta

· 3,191 Views · 2 Likes

Building RAG Apps With Apache Cassandra, Python, and Ollama

A brief introduction to Apache Cassandra for retrieval-augmented generation using Python and Ollama for developing applications free of cost locally or on a server.

February 3, 2025

by Varun Setia

· 4,674 Views

Pydantic: Simplifying Data Validation in Python

Pydantic is a powerful Python library that uses type annotations to validate data structures. Learn about the powerful features of Pydantic with code examples.

February 3, 2025

by Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

· 3,684 Views · 6 Likes

The Quest for HA and DR in Loki

Minimize data loss and business disruption by implementing high availability and configuring disaster recovery for Loki with AWS S3 as the object store.

January 31, 2025

by Pavan N G

· 4,255 Views · 1 Like

Building Custom Tools With Model Context Protocol

Learn how to build MCP servers to extend AI capabilities. Create tools that AI models can seamlessly integrate, demonstrated through an arXiv paper search implementation.

January 31, 2025

by Aditya Karnam Gururaj Rao

· 11,020 Views · 4 Likes

Web Scraping With LLMs, ScrapeGraphAI, and LangChain

In this article, learn how to use LLMs for web scraping with ScrapeGraphAI, LangChain, and Pydantic. This guide covers setup, configuration, and data extraction

January 31, 2025

by Juveria dalvi

· 5,037 Views

Creating a Service for Sensitive Data With Spring and Redis

In some cases, one cannot store user-sensitive data permanently. Let's create a simple application that handles sensitive data leveraging Spring and Redis.

January 31, 2025

by Alexander Rumyantsev

· 5,143 Views · 2 Likes

CAP and PACELC Theorems in Plain English

Data engineering fundamentals explained: the CAP theorem and PACELC — two frameworks for understanding tradeoffs in distributed systems.

January 31, 2025

by luminousmen _

· 2,779 Views

How to Build a Data Dashboard Prototype With Generative AI

This article is a tutorial that shows how to build a data dashboard to visualize book reading data taken from goodreads.com.

January 30, 2025

by Jo Stichbury

CORE

· 4,679 Views · 1 Like

Develop Microservices Using Azure Functions, API Management

Explore how to easily build cloud-deployed microservices using serverless technology with Azure Functions and connect with APIs standardized in API Management.

January 30, 2025

by Ankit Masrani

· 7,402 Views · 3 Likes

Building a Machine Learning Pipeline Using PySpark

This article discusses building an efficient ML pipeline with PySpark, covering data loading, preprocessing, model training, and evaluation for large datasets.

January 30, 2025

by Abhishek Trehan

· 6,121 Views · 1 Like

Bridging Graphviz and Cytoscape.js for Interactive Graphs

Making Graphviz static digraphs interactive and compatible with Cytoscape by converting DOT format graphs into Cytoscape JSON using Python.

January 30, 2025

by Puneet Malhotra

· 4,529 Views · 1 Like

SmartXML: An Alternative to XPath for Complex XML Files

We'll discuss SmartXML, an XPath alternative for parsing complex XML files, converting them to SQL, and loading the results into a database seamlessly.

January 30, 2025

by Luca Sanders

· 5,901 Views · 2 Likes

The Latest Data Engineering Topics