DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • Detecting Memory Leaks From a JVM Heap Dump
  • Java and Low Latency
  • All You Need To Know About Garbage Collection in Java
  • Java Memory Management

Trending

  • Exploring Sorting Algorithms: A Comprehensive Guide
  • Spring Authentication With MetaMask
  • Modern Data Backup Strategies for Safeguarding Your Information
  • How TIBCO Is Evolving Its Platform To Embrace Developers and Simplify Cloud Integration
  1. DZone
  2. Coding
  3. Java
  4. Apache RocketMQ: How We Lowered Latency

Apache RocketMQ: How We Lowered Latency

See how latency was lowered in Apache RocketMQ.

Andy Shi user avatar by
Andy Shi
·
Jun. 07, 19 · Tutorial
Like (4)
Save
Tweet
Share
20.50K Views

Join the DZone community and get the full member experience.

Join For Free

Apache RocketMQ

Apache RocketMQ is a distributed messaging and streaming data platform open sourced by Alibaba.

Here is the typical deployment topology:

Image title

At first glance, it looks pretty similar to other products, such as Kafka.

In fact, even the NameServer in the above diagram is Zookeeper, the same as Kafka.

To end the confusion, I would like to share a couple lessons we learned in the process of shaping this project.

In this article, I’ll start with one of the major features: low latency.

Little’s Law

Image title

Little’s law states that the occupancy = latency x throughput.

If occupancy is a set number, the lower the latency, the higher the throughput. In order to maximize the throughput, we have to minimize the latency.

RocketMQ is a pure Java implementation. By “pure” I mean rocketMQ has developed its own memory, storage, and all the other modules, all of which has to be fine-tuned to avoid huge latency.

Image title

JVM Stall:

JVM stall happens a lot, such as during GC, JIT, RevokeBias, and RedefineClasses, to name a few. Many of us already know how to fine-tune GC: via adjusting heap size, changing GC timing, optimizing data structure, etc. But how do we tune the other JVM stalls?

One approach to the issue is to use -XX:+PrintGCApplicationStoppedTime with XX:+PrintSafepointStatistics and -XX: flags to isolate the trouble maker. For example, if we can identify RevokeBias is causing the stall, we can use -XX:-UseBiasedLocking to turn off the biased lock.

Memory Reclaim

When RocketMQ is applying for new memory pages, there is a chance that will cause the Linux system to get into a stage called direct reclaim, which may cause huge latency.

Image title

Another problem we run into is when the kernel reclaims the memory, if that page is loaded from swap space, it may cause file I/O, which leads to huge delay.

Image title

To solve these two issues, we played with different combinations of vm.extra_free_kbytes and vm.swappiness to get the optimal result.

In most Linux versions, the memory page size is 4K. If your application somehow creates a read/write racing condition on the same page, the latency will be a huge problem. To avoid this issue, we tried different ways to coordination memory access.

Conclusion

As they say: the devil is in the details. Many of the fine-tuning processes are trivia yet important. After the optimization, the project is able to handle very tough challenges.

The following is the result of the latency benchmark:

Image title

philosophy Memory (storage engine) Java (programming language) garbage collection Java virtual machine Throughput (business) Law (stochastic processes) Data (computing) kafka Linux (operating system)

Opinions expressed by DZone contributors are their own.

Related

  • Detecting Memory Leaks From a JVM Heap Dump
  • Java and Low Latency
  • All You Need To Know About Garbage Collection in Java
  • Java Memory Management

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: