Over a million developers have joined DZone.

A Downside of Durable Messaging

· Performance Zone

See Gartner’s latest research on the application performance monitoring landscape and how APM suites are becoming more and more critical to the business, brought to you in partnership with AppDynamics.


Durable messaging can be very fast, as fast as non-durable messaging up to a point.

Limitations of durable messaging

Durable messaging is dependant on the size of your main memory and the speed of your hard drive.  If you have a HDD, this can be as low as 20 MB/s and as high as 60 MB/s.  A RAID set of HDD can support between 100 and 300 MB/s. An SATA SSD can support between 100 and 500 MB/s and a PCI SSD can support up to 1.5 GB/s.

Case study

Say you have 8 GB of memory, writing two million 100 bytes messages per second and a HDD which support 25 MB/s.

This works fine at the speed in bursts but you reach a point where your disk cache is full.  Depending on your OS this can be between 20% and 80% of your main memory size.  In my experience, Windows tends to be closer to 20% even if you have plenty of free memory whereas Linux tends to allow in the region of 30% of your memory in uncommitted writes.

Say you are writing two million 100 byte message per second or 200 MB/s and you have 1600 MB of disk cache. The difference in speed is 175 MB/s between the rate you are writing and the rate you are generating it so in just 9 seconds you have filled the cache.  At this point your performance plummets to the write speed of your disk which is 25 MB/second. With each messaging being 100 bytes, you are now writing 250,000 messages per second or 8x slower.

What is the solution?

  • Keep your micro-bursts to less than you can fit in disk cache e.g. in the above case this would be about 18 million messages.
  • Increase the amount of memory you have.  While memory is cheap and you can buy 32 GB for about £150, all this does in include the duration of the micro-burst you can support.
  • Increase the speed of your drive.  With SSD you can support much higher bandwidths.  SATA SSD drives support up to 500 MB/sec which is higher than Chronicle can typically serialize messages, i.e. more than enough.  The downside of this is it reduces the total number of messages you can write. A 500 MB SSD can store 5 billion 100 byte messages. A 6x4 TB RAID-5 set can support a transfer rate of over 200 MB/s which would be enough for the above case study, and can store 200 billion messages.


If you see any durable messaging solution suddenly slow down under load, you need to look at the size of your buffers and the throughput of your disk sub-system.

The Performance Zone is brought to you in partnership with AppDynamics.  See Gartner’s latest research on the application performance monitoring landscape and how APM suites are becoming more and more critical to the business.


Published at DZone with permission of Peter Lawrey, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}