MSMQ Basics: Queues, Messages, Transactions

Microsoft Message Queue Server (MSMQ) is a Message-Oriented Middleware that allows applications to communicate among them using queues. In this post, we’ll go over some of the MSMQ basics: queues, messages, and transactions.

Queues

The queue is one of the basic concepts of MSMQ. It is just a container that stores messages, decoupling the sender from the receiver. MSMQ queues are not necessarily FIFO (First In, First Out) because messages can be prioritized.

Queues can be transactional or nontransactional. Transactional queues can only receive messages sent within a transactional context. Nontransactional queues can only receive messages sent outside of a transactional context. Messages sent in a transactional context are processed in the order in which they were sent.

There are two categories of queues: application queues and system queues.

Application Queues

Application queues are used by applications to exchange messages. These are the different types of application queues:

• Public queues (MachineName\QueueName) are destination queues published in Active Directory. This means that the queue’s properties (not contents) are replicated.
• Private queues (MachineName\Private$\QueueName) are destination queues that are registered on the local machine.
• Administration queues store the system-generated acknowledgment messages. These are not generated by default. You need to set the Acknowledgment type and the Administration queue on the message. Administration Queues use a correlation identifier to match the acknowledgment with the original message. They cannot be transactional.
• Response queues are used for Request/Response. These use a correlation identifier to match the response to a request.
• Report queues track the progress of a message through the queuing infrastructure. They store system-generated report messages.

System Queues

System queues are created by the queue manager. These are the types of system queues:

• Dead-Letter Queues store messages that cannot be delivered, so you don’t lose them. This is MSMQ’s implementation of the well-known Dead Letter Channel pattern. There are two types of dead-letter queues: transactional (MachineName\XactDeadletter$) and nontransactional (MachineName\Deadletter$). When MSMQ puts a message in the dead-letter queue, it also adds a reason (i.e., timeout expired, authorization error).
• Outgoing Queues store messages for a remote destination queue. This is an implementation of the store and forward technique. MSMQ stores the message locally, so it can send messages even when the receiver is down. When the receiver reconnects, the queue manager puts the message on the destination queue and then removes it from the outgoing queue. It’s important to know thatoutgoing queues are not created when sending a message to a local queue. For a more detailed explanation about how outgoing queues work, read John Breakwell's blog posts.
• Journal Queues store copies of messages. After sending a message, MSMQ can put a copy of the message in the Computer Journal (MachineName\Journal$). After removing a message from the destination queue, a copy of the message is optionally sent to the Queue Journal (MachineName\QueueName\Journal$).
• Private System Queues are created by MSMQ internally for administration purposes, so you cannot open them.  Here are the different types of private system queues:
  • admin_queue$ stores administrative messages.
  • order_queue$ stores messages and ensure in order delivery of transactional messages.
  • msmqadminresp stores administrative response message (i.e., MQPing responses).
  • msmqtriggersnotifications receives a message when a trigger is created/updated/deleted.
  • notify_queue$ stores notification messages.

Accessing Queues

You can access a queue by using its path name, format name, direct format name or its properties. The Path Name specifies the computer that will store the queue, the private$ keyword if it’s a private queue and the name of the queue. This is required when creating the queue. It’s important to know that the path name is case insensitive and queue names longer than 124 characters are not supported.

Messages

Messages are just envelopes that are used to send data through the queues. They can be application-generated or system-generated. Application-generated messages are sent by queued messaging applications.

MSMQ automatically creates the following types of system-generated messages:

• Acknowledgment messages: used to inform the sender about the delivery status of a message. There are two types of acknowledgment messages: positive and negative. A positive acknowledgment messages is sent when a message reaches the destination queue. If a message cannot reach its destination queue, a negative acknowledgment message is sent.
• Report messages: used for tracing purposes. The queue manager generates report messages when application-generated messages are routed through MSMQ.

Express or Recoverable

Messages are stored locally until they are sent to the destination queue:

• Express messages are stored in RAM during routing and delivery. This means that if the machines go down, the messages are lost. Their main advantage is that they provide better performance.
• Recoverable messages are stored on disk. This means that messages are not lost when the host machine goes down. Of course, this comes with a performance penalty.

Transactional or Nontransactional

You can send or receive messages from transactional queues in a transactional context. Transactional messages have to be recoverable. Transactions incur some overhead, so you can expect less throughput when using this type of messages.

Timers

MSMQ provides two timers, so you can automatically expire messages and, optionally, send a copy of the message to the sender’s dead-letter queue:

• Time-To-Reach-Queue is the time that a message has to reach the destination queue. If the timer has elapsed when the message has arrived, the queue manager discards the message.
• Time-To-Be-Received specifies a maximum time span between sending the message and removing the message from the destination queue. This includes the time it takes to reach the queue and the time the message stays in the destination queue. Because of this, Time-To-Be-Received should be longer than Time-To-Reach-Queue.

Messages properties

The MSDN documentation contains a list of properties a message can have. It’s important to know that the maximum message size is 4MB and times (i.e., arrived time, sent time, time to reach queue) are adjusted to time zones.

Transactional Messaging

MSMQ supports internal and external transactions.

Internal (single) transactions are bound to MSMQ. You can send one or many messages in the same transaction. MSMQ guarantees that if multiple messages are sent to the same queue, the messages will be received in the same order. As a side note, you can to do a nontransactional receive on  a transactional queue. The downside of this approach is that in case of a rollback, the message will be lost.

External transactions can include other Resource Managers (RM), like MS SQL Server. Microsoft Distributed Transaction Coordinator (MSDTC) coordinates these transactions. This coordination overhead comes with a big performance cost.

It’s important to know that MSMQ does not propagate the transaction context between the sender and the receiver. This is to be expected, as the purpose of asynchronous messaging is to decouple the sender from the receiver. The MSMQ protocol clearly defines the transaction boundaries for end to end scenarios:

• Transacted Send
• Reliable Transfer
  • If the receiver queue manager is different than the sender queue manager (i.e., queues on different machines), the sending queue manager reliably transfers the message to the receiving queue manager. This step is not required when both sender and receiver queues are on the same machine.
• Transacted Receive

Message Delivery Assurance

MSMQ provides the following levels of delivery assurance:

• At-most-once
  • Best-effort express delivery through Express Messages. These messages are volatile and can get lost in transit (e.g: transient network failure, MSMQ failure) or at rest (system restart).
  • Best-effort delivery through Recoverable Messages. These messages can get lost in transit, but persist through system restarts.
• Exactly-once through transactional messages.