Compress RabbitMQ Messages

For us developers and system administrators, compression of in-transit data is a trivial way to improve application performance. Nowadays, most of our web traffic is compressed, especially when web browsers are involved since a large part of a web page’s content is more or less eligible for compression.

However, we usually neglect compression in other, less obvious use cases, such as data exchange between microservices, for example, using message brokers. This is where I’d like to focus in this article, using RabbitMQ as an example. I also used Spring for this proof of concept to leverage its powerful simplicity.

Why Should I Compress?

It might seem counter-intuitive to compress the messages sent through a message broker: they are usually sent over a high-speed network link, so why bother? You would probably be right about the network speed and this whole idea might only be useful for those using a message broker over the internet, but you should still consider this approach: it’s a low-hanging fruit.

At the cost of a few CPU cycles used for compression, you can get:

• Improved bandwidth usage: Optimizing network utilization can be very important, especially in scenarios where multiple microservices are communicating frequently
• Reduced transfer times: Smaller payload sizes result in faster transfer times, even on high-speed networks. This can contribute to overall performance improvements and reduced latency in microservices communication.
• Lower latency: Compressed payloads can help reduce the time it takes for data to travel between microservices, resulting in improved responsiveness in the system.
• Scalability: Compressed payloads can enhance the scalability of microservices architectures by optimizing resource utilization and minimizing network bottlenecks, allowing the system to handle increasing loads more efficiently.

Oh, regarding that CPU cost: The Deflate algorithm is very light on the CPU, and you can also configure the compression level and tune the tradeoff based on your needs.

Tradeoffs

In most engineering disciplines, we continuously evaluate tradeoffs, and this is no exception. As mentioned earlier, CPU overhead is mostly insignificant due to the compression algorithms being optimized. So, the next question would be: How much effort will it take?

Not much, thanks to the Spring framework! All it takes is a RabbitTemplate with a MessageConverter.

How

Using RabbitZip, my reference implementation as an example, let’s assume you already have a Spring Boot application using a standard RabbitTemplate (I’ll explain the @Primary annotation shortly):

@Bean
@Primary
RabbitTemplate template(ConnectionFactory connectionFactory) {
    return new RabbitTemplate(connectionFactory);
}

Now, let’s create another RabbitTemplate, one that will compress its payload for us:

@Bean("compressed")
RabbitTemplate compressedTemplate(
    ConnectionFactory connectionFactory,
    CompressedMessageConverter compressedMessageConverter) {

    final RabbitTemplate rabbitTemplate = 
        new RabbitTemplate(connectionFactory);
    rabbitTemplate.setMessageConverter(compressedMessageConverter);
    return rabbitTemplate;
}

There are two differences: We use a qualifier (compressed) to distinguish this RabbitTemplate, and we also use a MessageConverter. The implementation of CompressedMessageConverter is very straightforward. It will essentially compress the message payload and add a message header Use-Gzip. This part is essential when deserializing the payload, as we need to know when a message needs to be decompressed before deserialization.

Once the implementation is in place, using it is even easier:

• To send compressed messages, you only need to inject the RabbitTemplate qualified with compressed and send messages as usual (shown here). We need the @Primary annotation on the default RabbitTemplate so that any existing injection points, which don’t use the qualifier, will still work.
• On the @RabbitListener side, you only need to define the messageConverter property (shown here)

Conclusion