Gautam Goswami

CORE

Founder at DataView

Bangalore, IN

Joined Sep 2020

https://dataview.in/

About

Enthusiastic about learning and sharing knowledge on Big Data, Data Science & related headways including data streaming platforms through knowledge sharing platform Dataview.in. Presently serving as Head of Engineering & Data Streaming at Irisidea TechSolutions, Bangalore, India. https://www.irisidea.com/gautam-goswami/

Stats

Reputation:	2306
Pageviews:	350.6K
Articles:	48
Comments:	3

Expertise

Big Data

Articles
Comments

Articles

From Stream to Strategy: How TOON Enhances Real-Time Kafka Processing for AI

The TOON data format specifically targets the propagation of structured, validated, and semantically consistent data, thereby reducing ambiguity in real time.

March 27, 2026

· 1,865 Views

Hot Data: Where Real-Time Insight Begins

Processing hot data has significant value in the modern age, as it enables businesses to make instant decisions with low-latency, fault-tolerant, real-time systems.

February 27, 2026

· 2,024 Views

Is TOON the Next Lightweight Hero in Event Stream Processing With Apache Kafka?

Majorly beneficial for LLM-specific pipelines, we can use TOON to ingest stream data into an Apache Kafka topic, as it's a compact, token-efficient serialization format.

November 28, 2025

· 3,801 Views · 1 Like

Using Schema Registry to Manage Real-Time Data Streams in AI Pipelines

Learn how schema registries ensure data consistency, integrity, and scalability in real-time AI pipelines using Apache Kafka and modern streaming tools.

October 27, 2025

· 2,353 Views · 3 Likes

AI on the Fly: Real-Time Data Streaming From Apache Kafka to Live Dashboards

Real-time data streaming plays a key role for AI models as it allows them to handle and respond to data as it comes in, instead of just using old fixed datasets.

September 11, 2025

· 6,967 Views

Driving Streaming Intelligence On-Premises: Real-Time ML With Apache Kafka and Flink

This article explores how to design, build, and deploy a predictive ML model using Flink and Kafka in an on-premises environment to power real-time analytics.

June 17, 2025

· 1,876 Views · 6 Likes

Materialized Views in Data Stream Processing With RisingWave

Materialized views enhance data streaming by improving incremental computation, enabling efficient retrieval and calculation of aggregated or pre-processed data.

March 3, 2025

· 3,533 Views · 2 Likes

Detecting Patterns in Event Streams With FlinkCEP

Patterns play a big role in ESP as they help to spot important sequences or behaviors in data that keep flowing nonstop.

February 5, 2025

· 6,463 Views · 2 Likes

How Apache Flink and Apache Paimon Influence Data Streaming

Apache Flink is a crucial component of Apache Paimon since it offers the real-time processing power that enhances Paimon's strong consistency and storage features.

January 28, 2025

· 5,757 Views · 4 Likes

Revolutionize Stream Processing With Data Fabric

An open-source distributed event-streaming platform like Apache Kafka supports data fabric by handling real-time data streaming across various systems.

January 6, 2025

· 3,708 Views · 3 Likes

Bridging the Gap: Unlocking the Power of HDFS-Based Data Lakes With Streaming Databases

Big data technologies' quick development has highlighted the necessity of a smooth transition between real-time data analytics and batch processing systems.

December 30, 2024

· 3,004 Views · 1 Like

Which Flow Is Best for Your Data Needs: Time Series vs. Streaming Databases

While both are used to handle time-related data, their underlying technologies and main purpose are built to serve different purposes.

December 13, 2024

· 5,703 Views · 3 Likes

Protecting Your Data Pipeline: Avoid Apache Kafka Outages With Topic and Configuration Backups

Applications that are unable to publish messages to a Kafka topic or be consumed by downstream applications are considered to be experiencing an outage.

November 29, 2024

· 3,407 Views · 1 Like

The Significance of Complex Event Processing (CEP) With RisingWave for Delivering Accurate Business Decisions

Learn more about CEP, how it addresses a key challenge in real-time processing by detecting patterns in data streams, and compare FlinkCEP and RisingWave.

September 11, 2024

· 5,259 Views · 1 Like

Integrating Apache Kafka in KRaft Mode With RisingWave for Event Stream Processing

Install and configure the latest version of Apache Kafka on a single-node cluster running on Ubuntu-22.04, and subsequently integrate it with RisingWave.

August 23, 2024

· 5,200 Views · 3 Likes

Criticality in Data Stream Processing and a Few Effective Approaches

Opposite to traditional batch data processing technique, here processing works on the data as it is produced in real-time. Learn more in this post.

August 2, 2024

· 5,867 Views · 1 Like

Partitioning Hot and Cold Data Tier in Apache Kafka Cluster for Optimal Performance

Discover how by partitioning the hot and cold data tiers in the Apache Kafka Cluster, we can optimize storage resources based on data characteristics.

June 28, 2024

· 10,009 Views · 6 Likes

Real-Time Data Transfer from Apache Flink to Kafka to Druid for Analysis/Decision-Making

In this article, follow an outline of the steps to transfer processed data from Flink 1.18.1 to a Kafka 2.13-3.7.0 topic.

April 15, 2024

· 2,717 Views · 2 Likes

Streaming Real-Time Data From Kafka 3.7.0 to Flink 1.18.1 for Processing

Flink seamlessly integrates with Kafka and offers robust support for exactly-once semantics, ensuring each event is processed precisely once. Learn more here.

March 10, 2024

· 11,855 Views · 2 Likes

Why Apache Kafka and Apache Flink Work Well Together to Boost Real-Time Data Analytics

Use Flink and Kafka to create reliable, scalable, low-latency real-time data processing pipelines with fault tolerance and exactly-once processing guarantees.

February 13, 2024

· 4,864 Views · 1 Like

Integrating Rate-Limiting and Backpressure Strategies Synergistically To Handle and Alleviate Consumer Lag in Apache Kafka

Kafka Consumer Lag refers to the variance between the most recent message within a Kafka topic and the message that has been processed by a consumer. This lag may arise when the consumer struggles to match the pace at which new messages are generated and appended to the topic.

January 23, 2024

· 3,197 Views · 3 Likes

Leveraging Apache Kafka for the Distribution of Large Messages

In this article, we will explore the architectural approach for separating the actual payload (the large video file) from the message intended to be circulated via Kafka.

December 19, 2023

· 4,467 Views · 4 Likes

The Zero Copy Principle With Apache Kafka

When doing computer processes, the zero-copy technique is employed to prevent the CPU from being used for data copying across memory regions.

November 17, 2023

· 3,689 Views · 1 Like

Understanding Supervisor in Apache Druid

A supervisor is a built-in part of Druid, making it easier to ingest, analyze, and monitor data in real-time. Learn more!

October 16, 2023

· 3,171 Views · 3 Likes

Causes and Remedies of Poison Pill in Apache Kafka

A poison pill is a message deliberately sent to a Kafka topic, designed to consistently fail when consumed, regardless of the number of consumption attempts.

September 25, 2023

· 6,809 Views · 3 Likes

Apache Kafka’s Built-In Command Line Tools

I want to highlight the five scripts/tools that I believe will have the biggest influence on your development work, mostly related to real-time data stream processing.

August 21, 2023

· 2,959 Views · 2 Likes

The Significance of Deep Storage in Apache Druid

Druid’s Deep storage guarantees long-term data persistence even if data is deleted from the live cluster after compaction.

July 7, 2023

· 3,782 Views · 2 Likes

Forging Druid With Apache Kafka for Real-Time Streaming Analytics

A real-time analytics database called Apache Druid can be leveraged very effectively where real-time ingestion, fast query performance, and high uptime are crucial.

June 16, 2023

· 5,460 Views · 1 Like