DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations

Trending

  • What Is mTLS? How To Implement It With Istio
  • Developers Are Scaling Faster Than Ever: Here’s How Security Can Keep Up
  • RBAC With API Gateway and Open Policy Agent (OPA)
  • A Data-Driven Approach to Application Modernization
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Testing, Tools, and Frameworks
  4. Apache Kafka Load Testing on AWS

Apache Kafka Load Testing on AWS

Benjamin Ball user avatar by
Benjamin Ball
·
Oct. 21, 14 · Interview
Like (0)
Save
Tweet
Share
13.60K Views

Join the DZone community and get the full member experience.

Join For Free

 A recent blog article from Jaime Alquiza talks about load testing (vs. benchmarking) Kafka clusters that are being hosted on AWS / EC2. Alquiza opens up the article with some words on what exactly the difference is between load testing and benchmarking, and how load testing doesn't necessarily characterize the same workload that your system will otherwise be handling. This ends up being an important factor as he moves forward, and he conveys pretty clearly the need to keep in mind that your a testing relative and varied workloads--and that while you'll certainly learn about cluster performance, it won't be exactly informative of your standard workload.

He identifies some of the goals of load testing in his mind:

But what performance characteristics should I expect in my particular configuration? What should I consider a bad latency spike? What's the impact of dropping a broker? The reason I'm interested in load testing is to form a collection of expectations so these questions are better understood in the reality of production infrastructure.

Alquiza created a tool for his load testing purposes that he calls Sangrenel, the purpose of which is to generate random messages of varying size and bombard the Kafka clusters and "spit out" throughput latency. In the rest of his article he sets up the testing tools and scenario, and includes a breakdown of Kafka performance, including the performance of:

  • Message size (throughput and latency)
  • Number of partitions
  • Replication cost
  • Consumer position impact
  • Scaling up broker resources
  • Varying cluster size
  • Small message trans rate maxout

Here wraps up with some closing thoughts and concludes:

I basically spend 2/3 of my work time torture testing and operationalizing distributed systems in production. There's some that I'm not so pleased with (posts pending in draft forever) and some that have attributes that I really love. Kafka is one of those systems that I pretty much enjoy every bit of, and the fact that it performs predictably well is only a symptom of the reason and not the reason itself: the authors really know what they're doing. Nothing about this software is an accident. Performance, everything in this post, is only a fraction of what's important to me and what matters when you run these systems for real. Kafka represents everything I think good distributed systems are about: that thorough and explicit design decisions win.
Read the rest of his article, and see the detailed results of his tests, at his blog.
kafka Load testing AWS

Opinions expressed by DZone contributors are their own.

Trending

  • What Is mTLS? How To Implement It With Istio
  • Developers Are Scaling Faster Than Ever: Here’s How Security Can Keep Up
  • RBAC With API Gateway and Open Policy Agent (OPA)
  • A Data-Driven Approach to Application Modernization

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: