DZone
Big Data Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Big Data Zone > Pepper-Box Kafka Load Generator

Pepper-Box Kafka Load Generator

Pepper-Box is a Kafka load generator app that's useful in streaming analytics and data pipelines implementation. Read on to learn about its components and workflow.

satish bhor user avatar by
satish bhor
·
May. 02, 17 · Big Data Zone · Tutorial
Like (10)
Save
Tweet
13.41K Views

Join the DZone community and get the full member experience.

Join For Free

Pepper-Box is a Kafka load generator application that can be used as a plugin for JMeter or standalone utility. It allows sending plain text Kafka messages (JSON, XML, CSV, or any other custom format), as well as Java serialized objects. Pepper-Box includes a template engine and random data generation function which helps to design message in any format. If we use it with JMeter then we can use all JMeter features. Pepper-Box is very useful in streaming analytics and data pipelines implementation, where input data format is tightly coupled with business problems.

Pepper-Box includes four main components.

1. Pepper-Box Kafka Sampler

This is a JMeter Java sampler that acts as a Kafka producer. This sampler gets messages from the backend data generator and sends messages to the Kafka broker at the given throttled rate. If the Kafka broker is designed with data encryption using SSL/TLS and authentication using Kerberos, then these securities can be easily configured in this sampler. By default, we included required and tuning-related parameters, but it also provides a user interface to configure other Kafka producer parameters.

2. Pepper-Box PlainText Config Element

This JMeter config element generates plain text messages for Kafka samplers based on the input message schema designed using the template engine. This config element provides a user interface for entering the message schema template. Before the test starts, this config element takes the schema, processes it, and creates a plain text message iterator that generates millions of plain text messages per second.

3. Pepper-Box Serialized Config Element

This JMeter config element generates serialized object messages for Kafka samplers based on the input class and its field mappings with template functions. This config element takes the class name and its field mappings with random data generation functions, processes it, and creates a Java object iterator thath generates millions of serialized object messages per second.

4. Pepper-Box Console Load Generator

This is a standalone Kafka load generator and does not require JMeter. This feature currently only supports plain text message generation and Java serialized message generation is in feature scope. This console-based load generator takes the required details message schema file, producer property file, throttle rate, duration, the number of producer threads, etc. and starts generating load at given throttled rate.

You can see below sample JSON message schema with five fields and values are template functions which will be replaced with generates random values dynamically for every iteration,

{
"messageId":{{SEQUENCE("messageId", 1, 1)}},
"messageBody":"{{RANDOM_ALPHA_NUMERIC("abcedefghijklmnopqrwxyzABCDEFGHIJKLMNOPQRWXYZ", 100)}}",
"messageCategory":"{{RANDOM_STRING("Finance", "Insurance", "Healthcare", "Shares")}}",
"messageStatus":"{{RANDOM_STRING("Accepted","Pending","Processing","Rejected")}}",
"messageTime":{{TIMESTAMP()}}
}

Pepper-Box Workflow

flow.png

When we enter the input schema (the sample schema is shown above) and start the test, then Pepper-Box follows a series of steps before producing the actual load on Kafka.

  1. The schema is given as an input to the schema parser, which parses the schema and prepares a series of expressions.
  2. The schema translator then converts those series of expressions into a class.
  3. Then translated Java class is then compiled and generates a message iterator.
  4. To produce messages on Kafka, this message iterator is iterated for specified test duration.

See the sample JMeter screens. Please visit Pepper-Box at GitHub for a more detailed implementation.

Image title

Image title

kafka Plain text Schema

Published at DZone with permission of satish bhor. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • RestTemplate vs. WebClient
  • Demystifying Cloud-Native Data Management: Layers of Operation
  • What Is Data Analytics? Understanding Data Analytics Techniques
  • Secure Your WSO2 Micro Integrator Deployment

Comments

Big Data Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo