Gautam Goswami

CORE

Founder at DataView

Bangalore, IN

Joined Sep 2020

https://dataview.in/

About

Enthusiastic about learning and sharing knowledge on Big Data, Data Science & related headways including data streaming platforms through knowledge sharing platform Dataview.in. Presently serving as Head of Engineering & Data Streaming at Irisidea TechSolutions, Bangalore, India. https://www.irisidea.com/gautam-goswami/

Stats

Reputation:	1769
Pageviews:	290.2K
Articles:	40
Comments:	2

Expertise

Big Data

Articles
Comments

Articles

How Apache Flink and Apache Paimon Influence Data Streaming

Apache Flink is a crucial component of Apache Paimon since it offers the real-time processing power that enhances Paimon's strong consistency and storage features.

January 28, 2025

· 1,764 Views

Revolutionize Stream Processing With Data Fabric

An open-source distributed event-streaming platform like Apache Kafka supports data fabric by handling real-time data streaming across various systems.

January 6, 2025

· 2,390 Views · 2 Likes

Bridging the Gap: Unlocking the Power of HDFS-Based Data Lakes With Streaming Databases

Big data technologies' quick development has highlighted the necessity of a smooth transition between real-time data analytics and batch processing systems.

December 30, 2024

· 2,453 Views · 1 Like

Which Flow Is Best for Your Data Needs: Time Series vs. Streaming Databases

While both are used to handle time-related data, their underlying technologies and main purpose are built to serve different purposes.

December 13, 2024

· 4,928 Views · 3 Likes

Protecting Your Data Pipeline: Avoid Apache Kafka Outages With Topic and Configuration Backups

Applications that are unable to publish messages to a Kafka topic or be consumed by downstream applications are considered to be experiencing an outage.

November 29, 2024

· 2,297 Views · 1 Like

The Significance of Complex Event Processing (CEP) With RisingWave for Delivering Accurate Business Decisions

Learn more about CEP, how it addresses a key challenge in real-time processing by detecting patterns in data streams, and compare FlinkCEP and RisingWave.

September 11, 2024

· 4,586 Views · 1 Like

Integrating Apache Kafka in KRaft Mode With RisingWave for Event Stream Processing

Install and configure the latest version of Apache Kafka on a single-node cluster running on Ubuntu-22.04, and subsequently integrate it with RisingWave.

August 23, 2024

· 4,540 Views · 2 Likes

Criticality in Data Stream Processing and a Few Effective Approaches

Opposite to traditional batch data processing technique, here processing works on the data as it is produced in real-time. Learn more in this post.

August 2, 2024

· 5,164 Views · 1 Like

Partitioning Hot and Cold Data Tier in Apache Kafka Cluster for Optimal Performance

Discover how by partitioning the hot and cold data tiers in the Apache Kafka Cluster, we can optimize storage resources based on data characteristics.

June 28, 2024

· 9,308 Views · 5 Likes

Real-Time Data Transfer from Apache Flink to Kafka to Druid for Analysis/Decision-Making

In this article, follow an outline of the steps to transfer processed data from Flink 1.18.1 to a Kafka 2.13-3.7.0 topic.

April 15, 2024

· 2,264 Views · 2 Likes

Streaming Real-Time Data From Kafka 3.7.0 to Flink 1.18.1 for Processing

Flink seamlessly integrates with Kafka and offers robust support for exactly-once semantics, ensuring each event is processed precisely once. Learn more here.

March 10, 2024

· 10,887 Views · 2 Likes

Why Apache Kafka and Apache Flink Work Well Together to Boost Real-Time Data Analytics

Use Flink and Kafka to create reliable, scalable, low-latency real-time data processing pipelines with fault tolerance and exactly-once processing guarantees.

February 13, 2024

· 4,355 Views · 1 Like

Integrating Rate-Limiting and Backpressure Strategies Synergistically To Handle and Alleviate Consumer Lag in Apache Kafka

Kafka Consumer Lag refers to the variance between the most recent message within a Kafka topic and the message that has been processed by a consumer. This lag may arise when the consumer struggles to match the pace at which new messages are generated and appended to the topic.

January 23, 2024

· 2,618 Views · 2 Likes

Leveraging Apache Kafka for the Distribution of Large Messages

In this article, we will explore the architectural approach for separating the actual payload (the large video file) from the message intended to be circulated via Kafka.

December 19, 2023

· 4,006 Views · 3 Likes

The Zero Copy Principle With Apache Kafka

When doing computer processes, the zero-copy technique is employed to prevent the CPU from being used for data copying across memory regions.

November 17, 2023

· 2,995 Views · 1 Like

Understanding Supervisor in Apache Druid

A supervisor is a built-in part of Druid, making it easier to ingest, analyze, and monitor data in real-time. Learn more!

October 16, 2023

· 2,775 Views · 3 Likes

Causes and Remedies of Poison Pill in Apache Kafka

A poison pill is a message deliberately sent to a Kafka topic, designed to consistently fail when consumed, regardless of the number of consumption attempts.

September 25, 2023

· 4,634 Views · 3 Likes

Apache Kafka’s Built-In Command Line Tools

I want to highlight the five scripts/tools that I believe will have the biggest influence on your development work, mostly related to real-time data stream processing.

August 21, 2023

· 2,502 Views · 2 Likes

The Significance of Deep Storage in Apache Druid

Druid’s Deep storage guarantees long-term data persistence even if data is deleted from the live cluster after compaction.

July 7, 2023

· 3,349 Views · 2 Likes

Forging Druid With Apache Kafka for Real-Time Streaming Analytics

A real-time analytics database called Apache Druid can be leveraged very effectively where real-time ingestion, fast query performance, and high uptime are crucial.

June 16, 2023

· 4,662 Views · 1 Like

Knowing and Valuing Apache Kafka’s ISR (In-Sync Replicas)

To get more clarity about ISR in Apache Kafka, we should first carefully examine the replication process in the Kafka broker.

June 1, 2023

· 3,965 Views · 1 Like

Handling Bad Messages via DLQ by Configuring JDBC Kafka Sink Connector

When an error occurs, or bad data is encountered by the JDBC Kafka sink connector, these unprocessed messages are forwarded to the DLQ.

April 11, 2023

· 4,976 Views · 1 Like

Streaming Data to RDBMS via Kafka JDBC Sink Connector Without Leveraging Schema Registry

This article covers the biggest difficulty with the JDBC sink connector: it requires knowledge of the schema of data that has already landed on the Kafka topic.

February 22, 2023

· 7,315 Views · 2 Likes

Intrinsic Aspects of Apache ZooKeeper and Their Importance

This article explores ZNodes, sessions, watches, quorum, transactions, and local storage and snapshots, all aspects of Apache ZooKeeper.

January 23, 2023

· 1,864 Views · 1 Like

Internal Components of Apache ZooKeeper and Their Importance

In this article, readers will learn about the internal components of Apache ZooKeeper. The key concept is the zNode, which be acted as files or directories.

January 20, 2023

· 4,924 Views · 2 Likes

Resolve Apache Kafka Starting Issue Installed on Single/Multi-Node Cluster

Without integrating Apache Zookeeper, Kafka alone won’t be able to form the complete Kafka cluster.

January 12, 2023

· 3,837 Views · 1 Like

Processing of Streaming Data: Kappa vs Lambda Architectures

In today’s Big Data landscape, Lambda architecture is a new archetype for handling a vast amount of data. How does it compare to Kappa architecture?

August 19, 2022

· 5,904 Views · 1 Like

The Lakehouse: An Uplift of Data Warehouse Architecture

This article highlights how an architectural pattern is enhanced and transformed into a traditional data warehouse, eventually turning it into a data lakehouse.

April 5, 2022

· 5,614 Views · 5 Likes

A Short Introduction to Apache Iceberg

This tutorial shows how to use Apache Iceberg in order to address data consistency and performance issues. Read on to see how it can help you!

August 20, 2021

· 9,176 Views · 3 Likes

Confluent’s Kafka REST Proxy, The Silk Route for Data Movement to Operational Kafka Cluster

In this article, I am going to detailing out the steps to integrate the prebuilt versions of Confluent REST Proxy with running a multi-broker Apache Kafka cluster.

June 13, 2021

· 19,195 Views · 3 Likes

Comments

Apache Kafka in a Smart City Architecture

Mar 15, 2021 · Kai Wähner

Nice read.

Install and Configure Confluent Platform (Kafka) in AWS EC2 Instance RHEL 8

Dec 01, 2020 · Enrico Rafols Dela Cruz

Nicely explained.