DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations

Trending

  • Develop Hands-Free Weather Alerts To Ensure Safe Backpacking
  • Unlocking the Power of AIOps: Enhancing DevOps With Intelligent Automation for Optimized IT Operations
  • CDNs: Speed Up Performance by Reducing Latency
  • What ChatGPT Needs Is Context
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Monitoring Kafka: The Real-Time Ops Challenge

Monitoring Kafka: The Real-Time Ops Challenge

Kafka's ability to handle streaming data feeds in real-time affords real-time insights into information relevant to a business at any time and with great performance.

Jason Bloomberg user avatar by
Jason Bloomberg
·
Sep. 29, 17 · Opinion
Like (5)
Save
Tweet
Share
6.13K Views

Join the DZone community and get the full member experience.

Join For Free

Let's say you're a long-time expert on message-oriented middleware — say, an MQSeries (now IBM MQ) pro from back in the day — and someone asked you to create an entirely new middleware product following modern, cloud-native architectural principles, able to process trillions of messages per day. What would it look like?

The answer? It would probably look a lot like Apache Kafka. Kafka is an open-source stream processing platform that originated at LinkedIn, where the engineering team built Kafka to support real-time data feeds to facilitate real-time analytics.

Kafka is cloud-native, which means that its creators built it to leverage the architectural best practices of the cloud, including unlimited horizontal scale, rapid elasticity, high performance, and no single point of failure.

Kafka can, of course, run in the cloud as well — but as middleware, it can connect to both cloud or on-premises endpoints, or run fully on-premises if so desired. Regardless of such deployment details, Kafka brings the architectural advantages of cloud-native software to the middleware world.

The Real-Time Context for Kafka

Kafka's ability to handle streaming data feeds in real-time enables executives and other line-of-business personnel to obtain real-time insights into information relevant to their business at any point in time.

To achieve this level of performance, Kafka operates with low latency, processing data in-memory in a fully distributed manner, while scaling writes and reads with partitioned, distributed commit logs.

In addition, Kafka offers built-in load balancing, coupled with its rapid partitioning of data (also called sharding). In other words, the Kafka team designed the platform to be more like a distributed database transaction log than a traditional middleware system.

Following its cloud-native heritage, the Kafka team also built replication into its architecture - once again, in a horizontally scalable, elastic way. Kafka accomplishes this task with a complicated set of records, topics, consumers, producers, brokers, logs, partitions, and clusters - often with one-to-many relationships, with numerous message traffic flows among them.

The end result is both scalable as well as fault tolerant, as no node acts as a single point of failure. Traditional middleware like IBM MQ, in contrast, depends upon technologies like queues to guarantee message delivery - but with queues, failures delay the arrival of messages. As a real-time platform, the Kafka architecture routes around failures.

Monitoring Kafka

The real-time, fault-tolerant performance of Kafka, as with any piece of software, depends upon its proper configuration and the proper operation of the infrastructure that supports it. It would be foolish to assume that the platform's fault-tolerant architecture doesn't require monitoring and management.

Monitoring the performance of Kafka, however, means monitoring its various components, as well as the message flows among them, either individually or as end-to-end transactions. Furthermore, such monitoring must be in real-time in order to keep up with the real-time data flowing through the Kafka implementation.

The diagram below from Nastel AutoPilot for Kafka illustrates this challenge. On the left is a Kafka sender, which interacts with three readers, and in turn, with five topics. Note, however, that Kafka is an elastic environment, so it may spawn additional components as necessary.

Image title

Monitoring Kafka (Source: Nastel)

AutoPilot monitors each of the components and the message flows connecting them — a task that would overwhelm a traditional monitoring tool.

In fact, AutoPilot provides both operational and transactional monitoring. It also offers forensics to diagnose Kafka problems and monitors Kafka performance and availability via end-to-end stream monitoring and metrics tracking from the various Kafka components as well as Zookeeper, Kafka's configuration service.

As streaming data become increasingly common, organizations are coming to depend upon the real-time visibility at scale that such technologies provide. The real-time streaming capabilities that Kafka delivers add new value to the business and can help evolve traditional data-centric offerings, from data warehouses to business intelligence.

This broad-based trend to real-time insights raises the bar on monitoring and management, both for operations as well as the transaction management essential for business visibility.

Customers aren't going to wait for real-time performance, so the business cannot afford to wait either. Every aspect of the technology infrastructure, from the network to the middleware to the applications, must now perform in a cloud-native, real-time context — including ops management and monitoring.

Copyright © Intellyx LLC. Nastel is an Intellyx client. At the time of writing, none of the other organizations mentioned in this article are Intellyx clients. Intellyx retains full editorial control over the content of this paper.

kafka

Published at DZone with permission of Jason Bloomberg, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • Develop Hands-Free Weather Alerts To Ensure Safe Backpacking
  • Unlocking the Power of AIOps: Enhancing DevOps With Intelligent Automation for Optimized IT Operations
  • CDNs: Speed Up Performance by Reducing Latency
  • What ChatGPT Needs Is Context

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: