RabbitMQ Consumer Retry Mechanism Tutorial

Problem Analysis 

It is important to highlight that a RabbitMQ message is immutable. This means none of the message, including the header, properties, and body, can be altered by an application unless republished as a new message. This makes it infeasible to maintain a "retry counter" with the message itself to indicate how many times the message has been retried. If there are competing consumers then the same message can be picked up by any consumer, which could be running on a separate thread or process or altogether on a different platform. This adds additional complexity to maintaining a "retry counter" on the consumer application side. 

Resolution

With RabbitMQ server version 3.8.X onwards, an additional parameter named x-death has been introduced. This parameter can be leveraged to manage the retry mechanism.

Retry "n" Times With Delayed Interval

Core Concept

With RabbitMQ server version 3.8.X onwards, RabbitMQ has added a parameter named x-death that indicates how many times the messages is traversed through "dead letter exchange." It is an array that contains various information that plays a vital role in the retry mechanism.

• Exchange: Source exchange name from which the messages was dead lettered. 
• Queue: Source queue name from where it was dead lettered.
• Reason: Cause of dead letter; could be one of the below reasons:
  • Rejected: Consumer "nacks" the message with requeue false
  • Expired: TTL expired
  • Maxlen: Message dead lettered due to max length
• Count: A number representing how many times it traverses through the dead letter path.

Queue and count are the two significant attributes that will be used to implement "n" times retry count. To implement the retry interval, TTL on the dead letter queue will be leveraged. We will go through a detailed mechanism and message flow with a simple business use case.

Simple Business Use Case

An application named OrderApp receives a request to buy a computer peripheral. It publishes a message to a queue which is already subscribed by InventoryApp. InventoryApp receives the message from the queue and validates it against the inventory store for stock availability. If the accessory is available, then it processes the message; otherwise it retries 3 times in 5 seconds apart to check if the product becomes available. If it still does not succeed, then it routes the message to an error queue. 

For this use case, we will define exchanges and queues as below. Note that in RabbitMQ, queues and exchanges can be created programmatically, so there’s no need to pre-create these entities.

• Order exchange: A topic exchange to which OrderApp publishes a customer’s order request. More on the RabbitMQ exchange type is here.  You can create any type of exchange for this use case.
• Order queue: This is bound to "Order Exchange" with routing key as "order*".

@Bean

    Queue autoOrderQueue() {

        return QueueBuilder.durable(orderQ).withArgument("x-dead-letter-exchange", EXCHANGE_RETRY) .withArgument("x-dead-letter-routing-key", RTK_RETRY_TOPIC).build();

    }

• Retry exchange: A topic exchange connected to "Order Queue" with   "Dead Letter Exchange" configuration. 
• Retry queue: A queue bound to "retry exchange" with routing key as "retry*". This queue has TTL set to the desired retry interval (in milliseconds). This queue also has a "dead letter exchange" configuration, which dead letters back to "order exchange". This configuration is important for the whole flow to work. When a message lands on this queue, it stays on the queue until it expires. On expiry it traverse the message back to "order queue". Please note that queue TTL is being used here, not message TTL. Otherwise, it will end up with a cyclic issue. 

xxxxxxxxxx

@Bean

    Queue autoRetryQueue() {

        return QueueBuilder.durable(retryQ).withArgument("x-message-ttl", 5000).withArgument("x-dead-letter-exchange", EXCHANGE_ORDER) .withArgument("x-dead-letter-routing-key", RTK_TOPIC).build();

    }

• Error exchange: This is a topic exchange where the messages will land if it still not succeed after "n" times retry.
• Error queue: This is a queue bound to "retry exchange" with routing key as "error*".

xxxxxxxxxx

@Bean

    Queue autoErrorQueue() {

        return QueueBuilder.durable(errorQ) .build();

    }

Message Flow

1. OrderApp (producer) publishes an order request (message) to order exchange with routing key as order*.
2. This message gets routed to order queue as the routing key order* matches to the routing rule attached to the exchange.
3. RabbitMQ broker dispatches the message to InventoryApp, the consumer.
4. Consumer receives the message. For simplicity, consumer just checks the "order-type" attributes from the message header. It returns success if the order type is "mouse" and throws  "no stock available" error if type is "keyboard" to trigger the retry flow. Consumer has declared a constant MAX-RETRY-COUNT as 3.
   1. If "order-type" is "mouse", the message is processed successfully  and a nack sends back to the RabbitMQ broker. This removes the message from the queue.
   2. If "order-type" is "keyboard", it throws a "no stock available" error. 
5.  Consumer verifies "count" and "queue" attributes in the x-death message property. For the entry in x-death where "queue" matches to  "order", then 
   1. If count < MAX-RETRY-COUNT, then it sends NACK (negative acknowledgement) with "requeue" as false. Broker receives this message and routes to the dead letter exchange configured for the order queue. This routes the message to the "retry" queue via "retry exchange".
   2. Otherwise, the message gets published to error exchange. 
6. The message routed to the retry queue.
7. Upon expiry of the message in the "retry" queue, it gets routed to the "order" queue due to TTL settings and "dead letter exchange" configuration on the "retry" queue expiry set. The retry interval that you need to set can be set by setting the TTL (in milliseconds) value here.  

After the third retry, the x-death value looks as below 

You may find an entry for message expiry, as well. This value represents message expiry count on "retry" queue.

Source code here.