Handling Poison Messages With Glassfish

Poison messages are basically delivery deadlocks caused by a continuous redelivery of a message to a JMS Queue or Topic. That usually happens due to a code bug or configuration problems in the project.

How to reproduce poison messages

The easiest way of reproducing the poison messages issue is to create a Message Driven Bean and then to throw an exception in its onMessage method, like the example below.

@MessageDriven(activationConfig = {@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue")}, mappedName = "MyQueue")
public class RegistrationMessageBean implements MessageListener {
	@Override
	public void onMessage(Message registration) {
		throw new RuntimeException("poison message");
	} 
}

Fixing the JMS deadlock

What happens is that JMS relies on transactions to guarantee that all messages in a Queue will be delivered despite any temporary problems in the message consumer. If a message consumer (MDB) throws an exception, the JMS server tries to redeliver the message (transaction rollback). Only if a method consuming the message finishes without error the transaction will be committed and then the message will be removed from the Queue (acknowledged). In the sample above, the message consumer will always throw an exception and due to that the server will always redeliver the message - a deadlock.

The workarounds to fix such a problem are:

1. Fix all bugs from your code: that's the general best solution for poison messages, but as you know bugs are intrinsic to any software and it is not a surprise the time you will loose fighting against the poison messages :)
2. Try-Catch and digest the exceptions: for the paranoid, a good choice is to surround the whole onMessage code with a try{...} catch(Exception e){ ... }. Even if you strongly believe your code is sound, it is a recommended practice to do that. So, rewriting our sample code in a safe way, it looks like this:

   @Override
   public void onMessage(Message registration) {
   	try {
   		throw new RuntimeException("poison message");
   	} catch(Exception error) {
   		logger.severe("I am ignoring the JMS exception: " + e.getMessage());
   	}
   }

The second solution is a robust way of guaranteeing the consume of a message despite any problems. Not so elegant, but without exceptions in the onMessage method, the Message will be acknowledged and the JMS transaction will be committed.

A more complicated scenario with sub-transactions

The previous solution works for the general case, a simple Java code inside the onMessage method. Problem is, JMS uses the Java Transaction API (JTA) to control the message transactions and the JTA API supports sub-transactions. So imagine if the onMessage calls a JPA transactional method:

@PersistenceUnit(name = "arenapuj")
protected EntityManagerFactory emf;

@Override
public void onMessage(Message registration) {
	try {
		create(new MyJpaEntity());
	} catch(Exception error) {
		logger.severe("I am ignoring the JMS exception: " + e.getMessage());
	}
}

public MyJpaEntity create(final MyJpaEntity entity) throws Exception {
	EntityManager manager = emf.createEntityManager();
	try {
		manager.persist(entity);
		manager.flush();
		return entity;
	} finally {
		if (manager != null && manager.isOpen()) {
			manager.close();
		}
	}
}

The robust try-catch block is still there but guess what: if the method JPA Transaction of the method create is rolled back, the JMS transaction is also rolled back, causing the poison message problem despite the try-catch block on the onMessage code. So, the comfortable robustness provided by the try-catch block is actually a trap, a silent killer in the JMS sub-transactions scenario.

How to avoid poison messages caused by sub-transactions?

You should annotate the method with the first sub-transaction with @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) to avoid the dependencies between the JMS Transaction and its sub-transactions. Notice that only the first sub-transaction needs to be decoupled from the JMS one to avoid poison messages.

@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public MyJpaEntity create(final MyJpaEntity entity) throws Exception {
	EntityManager manager = emf.createEntityManager();
	try {
		manager.persist(entity);
		manager.flush();
		return entity;
	} finally {
		if (manager != null && manager.isOpen()) {
			manager.close();
		}
	}
}

Done, now your JMS method is decoupled from the new JPA transaction and if the JPA code rolls back, the JMS message will be acknowledged anyway.

Disclaimer: I am abusing the terms "JMS transaction" and "JPA transactions" for the sake of clarification here. Actually there are no such things, we have only the transactions defined in the Java Transaction API (JTA). For me it seems simpler to visualize the problem thinking about the methods and their transactions scope separately but after all it is all about JTA Transactions :)

Handling problems outside the code

Other common JMS scenario is to have resources problems, like connection failures, database down, etc. If we have a message consumer down and a message producer working, the producer will try to send a message, fail and try to send the message again. It would cause another type of deadlock, but the Java EE containers provide a set of configuration options to prevent such problems. The ActivationSpec for Message Driven Beans specifies two annotations to workaround activating problems:

1. endpointExceptionRedeliveryAttempts: Number of times to redeliver a message when MDB throws an exception during message delivery
2. sendUndeliverableMsgsToDMQ: Place message in dead message queue when MDB throws a runtime exception and number of redelivery attempts exceeds the value of endpointExceptionRedeliveryAttempts? If false, the Message Queue broker will attempt redelivery of the message to any valid consumer, including the same MDB.

You may check those configurations in your code review, but it is not so critical since their default values are reasonable for the common scenarios. And you need to know container specific attributes. The Glassfish V3 activation properties changed a bit, and I suppose for the other containers we will find different names. Fortunately we can expect that all defaults work fine and we are not forced to dig in product-specific details all the time.

Summary

JMS is one of the most powerful java EE resources available for developers and architects, but it is very important that anyone designing such applications knows in deep the specification and also some implementation tricks. Poison messages can make your server and eventually the whole host machine to hang for hours - forcing a machine to restart. It is a common problem, in my opinion weakly supported by the containers and a problem we should know about. The goal of this blog entry is just to give you a chance to identify the poison messages problem of your application. to know more about JMS and its details, you need to read more and the links below seem to be a good starting point:

• Creating Robust JMS Applications - The Java EE 5 Tutorial
• JMS Messaging Using GlassFish by Deepa Sobhana
• Bitter Messages: Java* Messaging Anti-Patterns by Bruce A. Tate
• Thread: how can i setup glassfish to deal with "poison message"

Aknowledgment: a special thanks for Marina Vatkina and Nigel Deakin for their friendly support through the Glassfish's mailing list. And a kudo to "The Professor" and his wise hints on this topic.

From http://weblogs.java.net/blog/felipegaucho