DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Data
  4. What You Need to Know to Test Kafka

What You Need to Know to Test Kafka

Kafka and distributed streams can come in handy when working with microservices. Read on for an introduction to testing Kafka and make sure you're on the right track.

Nirmal Chandra user avatar by
Nirmal Chandra
·
Feb. 14, 19 · Tutorial
Like (7)
Save
Tweet
Share
29.84K Views

Join the DZone community and get the full member experience.

Join For Free

1. Introduction

In this article, we will quickly explore some of the fundamental concepts of Kafka distributed streams. Also, we will learn how to test an application built using Kafka.

This will give us some insight into what we can test in a microservices architecture involving Kafka.

For more detailed about automated testing Kafka application with running examples in GitHub, please visit A Quick and Practical Example of Kafka Testing in DZone.

1.1. How Kafka is Different from Other MQs

All Message Queues(MQs) such as Kafka, ActiveMQ and RabbitMQ etc are all messaging technologies used for asynchronous communication. They all support decoupling i.e. decoupling between server and client while sending/receiving messages.

Whereas Kafka is a distributed, parallel processing, partitioned, replicated messaging service. Though it provides the functionality of a simple messaging system, at the same time it(Kafka) has been uniquely designed for a high-throughput distributed messaging system.

kafka-basics

                                                                                            Image credit:tutorial-point

2. Things We Need To Know

Kafka is a distributed messaging system. While the word "distributed" here solves loads of real-world problems, at the same time it also causes us to think of testing scenarios and challenges to address.

When we deal with Kafka applications, we need to know at least the below topics:

  1. Broker
  2. Topic
  3. Partitions
  4. Record
  5. Produce
  6. Consume

We need to know where the topic resides inside the brokers and what type of messages, a.k.a. records, are produced to the topic, as well as what happens when the messages are consumed by the listeners.

Once we know these basic things, we should be able to test any Kafka application easily.

Kafka architecture

For more details about Kafka, please visit Kafka Architecture and More.

2.1. What Does a Broker(s) Do?

A broker is a Kafka server. A broker allows the producers to send messages, and consumers to fetch messages by topic, partition, and offset.

2.2. What Is a Topic and Partitions?

According to Kevin Sookocheff, "Kafka topics are divided into a number of partitions. Partitions allow you to parallelize a topic by splitting the data in a particular topic across multiple brokers — each partition can be placed on a separate machine to allow multiple consumers to read from a topic in parallel."

Image title

2.3. Knowing the Record Format

A record is a message which can be written to or fetched from a topic. A record can be of various formats, e.g. RAW, JSON, CSV, AVRO, etc., and many others.

Records are represented in Key, Value pair,

e.g. "key":"1234", "value":"Hello World"

2.4. Producer Testing

'Produce' simply means writing one or more records (key-value pairs) to a topic. For example, we produce a record with the following.

i.e. "key":"1234", "value":"Hello World"

2.5. Consumer Testing

'Consume' simply means fetching one or more records (key-value pairs) from one or more topic(s).

Assert that the same record was present in the response (fetched record) i.e. "key": "1234", "value": "Hello World",

because we might have consumed more than one record if they were produced to the same topic before we started consuming from the topic.

3. Things We Can Test in a Kafka Applications

Put simply, we can assert whether we have successfully produced (written) a particular record or stream of records to a topic. And we can also assert the consumed (fetched) record or stream of records from one or more topic(s).

Also, we can dive in further and assert at granular levels, for instance :

  1. Whether we have produced the record to a particular partition.
  2. The type of record we are able to produce or consume.
  3. The number of records written to a topic or fetched from a topic.
  4. The offset of the record.
  5. Sending and receiving AVRO or JSON records and asserting the outcome.
  6. Assert the DLQs (Dead Letter Queue) record(s) and the record-metadata.
  7. Schema Registry for AVRO and validate records.
  8. And KSQL of querying streaming data in a SQL fashion and validate the result.

4. Conclusion

In this short tutorial, we learned some fundamental concepts of Kafka and the minimum things we need to know before proceeding with Kafka Testing. Also, we learned what all things we can cover in our testing.

For more and practical Kafka Testing, please visit A Quick and Practical Example of Kafka Testing in DZone.

kafka Record (computer science) Testing application Concept (generic programming) Stream (computing) avro microservice Data (computing)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Secrets Management
  • DevOps Roadmap for 2022
  • Cloud Native London Meetup: 3 Pitfalls Everyone Should Avoid With Cloud Data
  • Exploring the Benefits of Cloud Computing: From IaaS, PaaS, SaaS to Google Cloud, AWS, and Microsoft

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: