Data Resources

Master Production-Ready Big Data, Apache Spark Jobs in Databricks and Beyond: An Expert Guide

Refine Apache Spark performance in Databricks with strategies. Includes expert insights, PySpark examples, and diagrams for efficient data processing.

November 6, 2025

by Ram Ghadiyaram

CORE

· 2,072 Views · 1 Like

AIOps to Agentic AIOps: Building Trustworthy Symbiotic Workflows With Human-in-the-Loop LLMs

Build secure, human-in-the-loop AIOps workflows using agentic AI for smart alert triage, context summarization, and automated, safe incident resolution.

November 4, 2025

by Pratik Prakash

CORE

· 3,238 Views

5 Critical Databricks Performance Hacks That Most Engineers Miss (100x Faster Queries)

In this article, learn to boost Databricks' performance with six proven optimization strategies for UDFs, AQE, Delta Lake, broadcasts, and Photon acceleration.

November 3, 2025

by Ram Ghadiyaram

CORE

· 3,399 Views · 4 Likes

Delta Lake 4.0 and Delta Kernel: What's New in the Future of Data Lakehouses

Delta Lake 4.0 pushes the lakehouse forward with flexible schemas, stronger transactions, remote, multi‑engine access, self‑optimizing performance, and AI‑ready storage.

November 3, 2025

by Sairamakrishna BuchiReddy Karri

· 3,465 Views · 2 Likes

Streamlining Real-Time Ad Tech Systems: Techniques to Prevent Performance Bottlenecks

Ad tech platforms depend on microscopic efficiency. Profiling and disciplined logging collectively cut waste and latency, turning scale into sustainable profit.

October 31, 2025

by Jeet Nishit Mehta

· 1,975 Views · 1 Like

When Coalesce Is Slower Than Repartition: A Spark Performance Paradox

In this article, learn why repartition() can outperform coalesce() in Apache Spark — and how Catalyst optimizer pushdown can throttle your job’s parallelism.

October 30, 2025

by Janani Annur Thiruvengadam

CORE

· 2,760 Views · 3 Likes

How to Get a Frequency Table of a Categorical Variable as a Data Frame

Learn how to get a frequency table of a categorical variable as a Data Frame in R using table(), cut(), count(), and plyr — with clean, ready-to-use output.

October 30, 2025

by Chaitanya Sagar

· 1,484 Views · 1 Like

From Autocomplete to Co-Creation: How AI Changes Developing/Debugging Workflows in Engineering

AI is not a replacement for engineers but a powerful partner that accelerates coding, debugging, and learning; human judgment still drives the real innovation.

October 29, 2025

by Aishwarya Murali

· 2,500 Views · 1 Like

End of Static Knowledge Bases? How MCP Enables Live RAG

Learn why static RAG systems fail over time and how the Model Context Protocol (MCP) enables live, hybrid AI architectures that stay accurate and up to date.

October 29, 2025

by Janani Annur Thiruvengadam

CORE

· 2,431 Views · 2 Likes

Debugging a Spark Driver Out of Memory (OOM) Issue With Large JSON Data Processing

This article draws on real-world debugging experience and aims to provide insights into Spark's memory management challenges.

October 29, 2025

by Raju Ansari

· 2,393 Views · 2 Likes

HSTS Beyond the Basics: Securing AI Infrastructure and Modern Attack Vectors

HTTP Strict Transport Security (HSTS) is a web security policy mechanism that helps protect websites against protocol downgrade attacks and cookie hijacking.

October 29, 2025

by Vidyasagar (Sarath Chandra) Machupalli FBCS

CORE

· 2,165 Views · 4 Likes

Building Cloud Ecosystems With Autonomous AI Agents: The Future of Scalable Data Solutions

Autonomous AI agents transform cloud ecosystems by automating data workflows, enhancing scalability, governance, and real-time analytics for resilient operations.

October 28, 2025

by Aravind Nuthalapati

CORE

· 10,534 Views · 1 Like

Data Migration in Software Modernization: Balancing Automation and Developers’ Expertise

Learn how to balance automation tools and developer logic to ensure clean transitions, reduce time to market, and build scalable software foundations.

October 28, 2025

by Sergey Laptick

· 1,173 Views

Writing (Slightly) Cleaner Code With Collections and Optionals

See how utility methods provided by Kilo's Collections and Optionals classes can help you write more readable and maintainable Java code.

October 27, 2025

by Greg Brown

· 2,732 Views · 1 Like

Set Up Spring Data Elasticsearch With Basic Authentication

Guide to configure SSL communication with Elasticsearch via Spring Data Elasticsearch. Additionally, the communication is secured with BASIC authentication.

October 27, 2025

by Arnošt Havelka

CORE

· 2,128 Views

Anthropic’s Model Context Protocol (MCP): A Developer’s Guide to Long-Context LLM Integration

Anthropic’s Model Context Protocol (MCP) standardizes how AI applications provide context to LLMs. Learn more about MCP and its practical uses.

October 27, 2025

by Naresh Dulam

· 5,733 Views · 4 Likes

Enterprise-Grade Document Intelligence: Cloud Big Data AI With YOLOv9 and Spark on AWS

Automate document analysis with YOLOv9, Apache Spark, and AWS. Boost speed, accuracy, and fraud detection across finance, healthcare, insurance, and more.

October 27, 2025

by Ram Ghadiyaram

CORE

· 2,535 Views · 4 Likes

Data Quality at Write Time: Engineering Reliability With Delta Expectations

Ensure reliable data from the start. Learn how Delta Expectations in Databricks enforce real-time, write-time validation to prevent bad data before it lands.

October 23, 2025

by Rambabu Bandam

· 2,574 Views · 1 Like

Coarse Parallel Processing of Work Queues in Kubernetes: Advancing Optimization for Batch Processing

In the current modern distributed architecture, this article will explain how to execute a Kubernetes Job with optimized and multiple parallel worker processes.

October 20, 2025

by Tanu Jain

· 3,050 Views · 1 Like

Should You Use Azure Data Factory?

Yes for most ETL scenarios, ADF handles data movement, basic transformations, and scheduling well, just don't expect it to replace custom code for complex business logic.

October 20, 2025

by Sohag Maitra

· 6,351 Views

The Latest Data Topics