When RabbitMQ Accepts a Message but Does Not Route It to a Queue

RabbitMQ is a message broker that applications use to communicate asynchronously. A publisher sends a message to an exchange with a routing key. The exchange compares the routing key with the binding keys and routes the message to all matching queues, which deliver it to consumers.

A successful publish means RabbitMQ accepted the message at the exchange. It does not always mean that a queue received it. If no binding matches, the message is unroutable and may be dropped or returned to the publisher, depending on the publisher settings.

RabbitMQ Exchange Types

An exchange decides where a message should go. Its type determines how it makes that decision:

• Direct: Routes when the message routing key exactly matches the binding key.
• Topic: Uses dot-separated keys and wildcards. * matches one word; # matches zero or more words.
• Headers: Routes using message headers rather than the routing key.
• Fanout: Sends a copy to every bound queue and ignores the routing key.

The default exchange is a special direct exchange. RabbitMQ automatically binds each queue to it using the queue name as the routing key.

Issue

We deployed RabbitMQ in a three-node Kubernetes environment. Several applications use it to broadcast messages to multiple consumer applications. 

Application teams create topic exchanges and bind queues with binding keys. For example, we created a topic exchange as shown below:

Publishing m.event.provisioning.started to the exchange should route a copy to every queue whose binding pattern matches it.

In some environments, messages published to topic exchanges were not routed to queues even when the expected routing key was used. We inspected the topology using the management console and rabbitmqctl commands.

These commands inspect the configured exchanges, queues, and bindings in the selected virtual host:

rabbitmqctl list_exchanges -p <vhost> name type durable arguments
  This command will list all exchanges in rabbitmq
rabbitmqctl list_queues -p <vhost> name durable type messages messages_ready consumers
  This command will list all queues declared in rabbitmq
rabbitmqctl list_bindings -p <vhost> source_name source_kind destination_name destination_kind routing_key arguments
  This command will list exchange, queue and bindings in rabbitmq

The configured topology looked correct. However, in one environment, when we tried to publish a message using the RabbitMQ management UI, we received the message: Message published, but not routed.

Impact

In a field deployment, this will impact multiple customer workflows. Customers can experience the failure even though the publisher reports a successful publish.

•  A provisioning or configuration workflow can remain in a pending state because the service waiting for the event never receives it.
• Different services can hold different views of the same customer operation. For example, one service may record that a change started while another service never receives the event needed to update inventory, status, or audit information.
• Customer onboarding, device activation, policy changes, or software upgrades can be delayed or fail when an expected event is lost between services.
• Operators may retry the operation or manually intervene. That can increase recovery time and, if the original event is later delivered, create duplicate processing or confusing status changes.
• The failure can look intermittent because only some bindings or queues are affected. The management UI may still show a healthy declarative topology, which makes the issue harder to identify as a broker-routing problem.

Root Cause

RabbitMQ persists cluster metadata, including topology such as exchanges, queues, and bindings. During startup, the broker loads that metadata into runtime structures used for message routing.

We observed this in a Mnesia-based deployment. Mnesia is a distributed internal database used by RabbitMQ to store cluster metadata.

And RabbitMQ loads topic exchange configuration into these internal tables: rabbit_topic_trie_edge and rabbit_topic_trie_binding.

rabbit_topic_trie_edge

This internal runtime table represents topic-binding segments as a trie. For a binding such as m.event.provisioning.#, the dot-separated segments form edges, and each edge refers to its parent. The root edge has no parent.

The command to list the contents of the rabbit_topic_trie_edge table is: rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_edge)'. 

Below is sample output:

rabbitm@rabbitmq-0:/$ rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_binding).'
[{topic_trie_binding,
     {trie_binding,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         <<48,148,14,244,108,180,212,249,183,91,230,141,99,163,177,115>>,
         {resource,<<"/">>,queue,<<"provisioning-events">>},
         []},
     const},
 {topic_trie_binding,
     {trie_binding,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         <<67,242,234,144,205,89,103,148,212,234,75,22,120,194,157,225>>,
         {resource,<<"/">>,queue,<<"audit-events">>},
         []},
     const},
 {topic_trie_binding,
     {trie_binding,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         <<205,242,142,92,26,182,127,217,6,198,131,59,35,208,63,252>>,
         {resource,<<"/">>,queue,<<"exact-event">>},
         []},
     const}]
root@my-rabbit:/# rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_edge).'
[{topic_trie_edge,
     {trie_edge,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         root,"m"},
     <<217,70,178,13,233,108,102,215,235,100,37,39,215,68,241,133>>},
 {topic_trie_edge,
     {trie_edge,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         <<210,2,101,2,238,34,177,160,57,102,64,37,37,22,188,199>>,
         "#"},
     <<48,148,14,244,108,180,212,249,183,91,230,141,99,163,177,115>>},
 {topic_trie_edge,
     {trie_edge,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         <<210,2,101,2,238,34,177,160,57,102,64,37,37,22,188,199>>,
         "completed"},
     <<205,242,142,92,26,182,127,217,6,198,131,59,35,208,63,252>>},
 {topic_trie_edge,
     {trie_edge,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         <<217,70,178,13,233,108,102,215,235,100,37,39,215,68,241,133>>,
         "event"},
     <<245,28,154,53,247,20,217,197,203,52,13,103,28,112,252,226>>},
 {topic_trie_edge,
     {trie_edge,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         <<245,28,154,53,247,20,217,197,203,52,13,103,28,112,252,226>>,
         "#"},
     <<67,242,234,144,205,89,103,148,212,234,75,22,120,194,157,225>>},
 {topic_trie_edge,
     {trie_edge,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         <<245,28,154,53,247,20,217,197,203,52,13,103,28,112,252,226>>,
         "provisioning"},
     <<210,2,101,2,238,34,177,160,57,102,64,37,37,22,188,199>>}]

rabbit_topic_trie_binding

This internal table associates the topic exchange and destination with the final edge in the topic trie.

The command to list the contents of the rabbit_topic_trie_binding table is: rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_binding).'

Below is sample output:

rabbit@rabbitmq-0:/$ rabbitmqctl eval 'ets:tab2list(rabbit_topic_trie_binding).'
[{topic_trie_binding,
     {trie_binding,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         <<48,148,14,244,108,180,212,249,183,91,230,141,99,163,177,115>>,
         {resource,<<"/">>,queue,<<"provisioning-events">>},
         []},
     const},
 {topic_trie_binding,
     {trie_binding,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         <<67,242,234,144,205,89,103,148,212,234,75,22,120,194,157,225>>,
         {resource,<<"/">>,queue,<<"audit-events">>},
         []},
     const},
 {topic_trie_binding,
     {trie_binding,
         {resource,<<"/">>,exchange,<<"provisioning.exchange">>},
         <<205,242,142,92,26,182,127,217,6,198,131,59,35,208,63,252>>,
         {resource,<<"/">>,queue,<<"exact-event">>},
         []},
     const}]

We developed a diagnostic utility that reconstructed binding paths from these internal runtime tables and compared them with the topology returned by rabbitmqctl. The comparison showed that some configured bindings were not correctly represented in the runtime routing state.

How We Investigated the Problem

We first treated the symptom as a normal binding problem. We checked that the publisher and the consumer were using the same virtual host, confirmed that the exchange was a topic exchange, and compared the binding keys character by character. We also checked the queue consumer count and published a test message directly from the management UI. The message was accepted, but the UI reported Message published, but not routed.

Next, we used rabbitmqctl list_exchanges, rabbitmqctl list_queues, and rabbitmqctl list_bindings to capture the topology. Those commands showed the exchange, queues, and bindings that we expected. We repeated the same checks after connecting to different RabbitMQ pods and did not find a spelling, virtual-host, or wildcard mismatch.

At that point, we stopped changing application routing keys and compared the configured topology with the broker's runtime topic-trie state. The configured binding was present in the management output, but the corresponding path was incomplete in the runtime tables on the affected deployment. This explained why recreating the binding changed the behavior without changing the application code.

The publisher confirmed, and the mandatory publish option helped us narrow the problem further. A publisher confirm told us that RabbitMQ accepted the publish operation. The mandatory option made an unroutable message visible to the publisher, but it did not tell us which intended consumer was missing. We still had to verify delivery at each expected queue.

What the Publish Result Actually Tells Us

RabbitMQ accepts a publish at the exchange before it evaluates all of the topic bindings. For a topic exchange, the routing key is split into dot-separated words and matched against each binding pattern. A single-word wildcard (*) matches exactly one segment, while the multi-word wildcard (#) can match zero or more segments. A small difference such as m.event.provisioning.started versus m.events.provisioning.started is enough to produce no match.

This was the important distinction in our investigation: a successful publish was not proof of delivery. When the publish is not mandatory, a message that matches no queue can be discarded after the exchange accepts it. In our case, the binding appeared in the declarative topology but was not fully available to the runtime routing path, so the result looked like an ordinary unroutable message even though the management view appeared correct.

Changes We Made After the Incident

We kept the exchange, queue, and binding declarations idempotent, but added an explicit verification step after broker restarts, pod rescheduling, and upgrades. We also added a utility that reads the rabbit_topic_trie_edge and rabbit_topic_trie_binding tables, reconstructs the binding paths, and compares them with the exchange, queue, and binding information returned by rabbitmqctl. This gives us a direct way to identify a binding that exists in the configured topology but is missing or incomplete in the runtime topic trie.

In addition, we enabled detailed metrics at the exchange level so that we can see the unrouted message count at each exchange level. We correlate those metrics with the virtual host, routing key, queue, node, and timestamp captured by the utility. The combination of runtime-table comparison and exchange-level metrics makes it easier to determine whether the problem is a routing mismatch, a missing runtime binding, or a consumer that received the message but did not process it.

We now use the application logs, exchange-level metrics, and the utility output as the first evidence after restarts, pod rescheduling, and upgrades. We correlate an actual failed event with the expected exchange and binding path instead of injecting another message into a customer environment. The expected topology, comparison utility, and diagnostic commands are kept with the deployment so that an operator can compare the intended state with the live state during an incident.

Observed Workaround

For each affected binding, we captured the virtual host, exchange, queue, binding key, and arguments, then deleted and recreated the binding through the RabbitMQ API. We compared the runtime topic-trie state with the topology reported by rabbitmqctl, retried the affected workflows, and confirmed delivery at every expected queue.

This was the workaround for our deployment, not a universal fix. Do not edit Mnesia or ETS files directly; internal tables are version-specific and direct changes can leave the cluster inconsistent.

Lessons Learned

Management and CLI topology output are necessary first checks, but they do not replace an end-to-end routing test. After restarts, pod rescheduling, or upgrades, publish representative routing keys and verify that every expected queue receives the message. Publisher confirms, and mandatory publishing can also make unroutable messages visible to applications.

We observed this issue only with topic exchanges.