DZone
Integration Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Integration Zone > Apache RocketMQ: How We Lowered Latency

Apache RocketMQ: How We Lowered Latency

See how latency was lowered in Apache RocketMQ.

Andy Shi user avatar by
Andy Shi
CORE ·
Jun. 07, 19 · Integration Zone · Tutorial
Like (4)
Save
Tweet
19.72K Views

Join the DZone community and get the full member experience.

Join For Free

Apache RocketMQ

Apache RocketMQ is a distributed messaging and streaming data platform open sourced by Alibaba.

Here is the typical deployment topology:

Image title

At first glance, it looks pretty similar to other products, such as Kafka.

In fact, even the NameServer in the above diagram is Zookeeper, the same as Kafka.

To end the confusion, I would like to share a couple lessons we learned in the process of shaping this project.

In this article, I’ll start with one of the major features: low latency.

Little’s Law

Image title

Little’s law states that the occupancy = latency x throughput.

If occupancy is a set number, the lower the latency, the higher the throughput. In order to maximize the throughput, we have to minimize the latency.

RocketMQ is a pure Java implementation. By “pure” I mean rocketMQ has developed its own memory, storage, and all the other modules, all of which has to be fine-tuned to avoid huge latency.

Image title

JVM Stall:

JVM stall happens a lot, such as during GC, JIT, RevokeBias, and RedefineClasses, to name a few. Many of us already know how to fine-tune GC: via adjusting heap size, changing GC timing, optimizing data structure, etc. But how do we tune the other JVM stalls?

One approach to the issue is to use -XX:+PrintGCApplicationStoppedTime with XX:+PrintSafepointStatistics and -XX: flags to isolate the trouble maker. For example, if we can identify RevokeBias is causing the stall, we can use -XX:-UseBiasedLocking to turn off the biased lock.

Memory Reclaim

When RocketMQ is applying for new memory pages, there is a chance that will cause the Linux system to get into a stage called direct reclaim, which may cause huge latency.

Image title

Another problem we run into is when the kernel reclaims the memory, if that page is loaded from swap space, it may cause file I/O, which leads to huge delay.

Image title

To solve these two issues, we played with different combinations of vm.extra_free_kbytes and vm.swappiness to get the optimal result.

In most Linux versions, the memory page size is 4K. If your application somehow creates a read/write racing condition on the same page, the latency will be a huge problem. To avoid this issue, we tried different ways to coordination memory access.

Conclusion

As they say: the devil is in the details. Many of the fine-tuning processes are trivia yet important. After the optimization, the project is able to handle very tough challenges.

The following is the result of the latency benchmark:

Image title

philosophy Memory (storage engine) Java (programming language) garbage collection Java virtual machine Throughput (business) Law (stochastic processes) Data (computing) kafka Linux (operating system)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • An Overview of 3 Java Embedded Databases
  • Is Java Still Relevant?
  • 10 Steps to Become an Outstanding Java Developer
  • 6 Things Startups Can Do to Avoid Tech Debt

Comments

Integration Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo