Big Data Resources

Frequently Faced Challenges in Implementing Spark Code in Data Engineering Pipelines

This article discusses some of the common challenges faced by data engineers in Pyspark applications and the possible solutions to overcome these challenges.

April 25, 2023

by Amlan Patnaik

· 2,610 Views · 2 Likes

A Critical Detail About Kafka Partitioners

Understanding the role of a partitioner in the Kafka producer is important to understand to build effective Kafka applications.

April 25, 2023

by Bill Bejeck

· 3,814 Views · 2 Likes

How To Install CMAK, Apache Kafka, Java 18, and Java 19 [Video Tutorials]

This article has video tutorials on installing CMAK, Apache, Kafka, Java 18, Java 19, and JDK on Windows, Amazon EC2, and Linux.

April 25, 2023

by Ram N

· 6,481 Views · 2 Likes

API Gateway For ChatGPT Plugins

Learn how API Gateway can be beneficial for ChatGPT plugin developers to expose, secure, manage, and monitor their API endpoints.

April 23, 2023

by Bobur Umurzokov

· 4,212 Views · 2 Likes

Producing and Consuming Avro Messages With Redpanda Schema Registry

Learn how to use Redpanda’s built-in schema registry to produce and consume Apache Avro messages in five simple steps.

April 21, 2023

by Dunith Dhanushka CORE

· 3,888 Views · 1 Like

Lambda Architecture: A Powerful Approach to Data Engineering

In this article, we will explore the Lambda Architecture in detail, understanding its key concepts, benefits, and challenges.

April 21, 2023

by Amlan Patnaik

· 4,293 Views · 6 Likes

How to Stream Sensor Data to Apache Pinot for Real Time Analysis

Learn how to use Arduino, MQTT, Kafka, and Apache Pinot to stream environmental data for real-time access to sensor data.

April 21, 2023

by David G. Simmons CORE

· 5,247 Views · 7 Likes

IoT Testing: How It Works and Why You Can’t Do Without It

This article explains IoT testing, how to do it correctly, which challenges you can face along the way, and why.

April 21, 2023

by Inna Martyniuk

· 3,556 Views · 2 Likes

Consumer Group in Kafka [Video Tutorials]: Partitions and Consumers

In this video tutorial series, take a deeper look at the consumer group in Kafka with varying numbers of partitions and consumers.

April 21, 2023

by Ram N

· 5,271 Views · 2 Likes

Static Proxies or Rotating Proxies: Which One Should You Choose?

This article explains the differences between Static proxies and Rotating proxies. Let's find out what they are.

April 20, 2023

by Hanzala Ali

· 4,365 Views · 1 Like

Kafka: The Basics

Kafka is a powerful tool for building streaming architectures. The article serves as an introduction to both the technology and associated data producers and consumers.

April 20, 2023

by Pavel Micka

· 7,113 Views · 6 Likes

Recording Badge Scans in Apache Pinot

Use a real-time analytics database to record NFC badge scans in part 1 of this 3-part project involving environmental data, real-time notifications, and more.

April 19, 2023

by David G. Simmons CORE

· 4,080 Views · 4 Likes

Integrate Apache Spark and QuestDB for Time-Series Analytics

In this tutorial, we will explore how a Spark Application can load and save data from/to QuestDB and provide practical advice to achieve the best performance.

April 19, 2023

by Imre Aranyosi

· 3,818 Views · 8 Likes

Apache Kafka in Java [Video Tutorials]: Architecture and Simple Consumer/Producer

Explore a video tutorial series regarding Apache Kafka in Java as you explore Kafka architecture, components, producer callbacks, and more.

April 19, 2023

by Ram N

· 4,565 Views · 2 Likes

Introduction to Elasticsearch

Elasticsearch is a highly scalable and distributed search and analytics engine designed to handle large volumes of structured, semi-structured, and unstructured data.

April 19, 2023

by Rama Krishna Panguluri

· 9,582 Views · 9 Likes

Apache Kafka for Data Consistency

Apache Kafka ensures data consistency across legacy batch, request-response mobile apps, and real-time streaming in a data mesh architecture.

April 18, 2023

by Kai Wähner CORE

· 2,386 Views · 2 Likes

Maximizing the Potential of LLMs: A Guide to Prompt Engineering

Carefully designing the prompts used in LLMs like GPT-4 can greatly improve the results and become very useful for a variety of use cases.

April 18, 2023

by Roger Oriol

· 3,288 Views · 1 Like

Demystifying Data Fabric Architecture: A Comprehensive Overview

This article will provide a comprehensive overview of the data fabric architecture, its key components, and how it works.

April 18, 2023

by Amlan Patnaik

· 4,444 Views · 2 Likes

Embedded Systems Security Vulnerabilities and Protection Measures

Some tips and important consideration for embedded devices, which are often connected to the internet and can be vulnerable to various types of cyberattacks.

April 17, 2023

by Sreekanth Yalavarthi

· 4,487 Views · 4 Likes

Securing MQTT With Username and Password Authentication

This article will explain how authentication works in MQTT, what security risks it solves, and introduce password-based authentication.

April 17, 2023

by Kary Ware

· 4,197 Views · 1 Like

The Latest Big Data Topics