EclipseLink JPA2 Distributed Cache Coordination

In larger systems where processing of user requests is distributed you need some sort of cache coordination otherwise the results may be incorrect and/or user experience bad. In Qualitas system I have a front-end Spring web application which sends AMQP messages to a broker which are later on picked up by Apache Camel route. I had a problem with caching because the web application was showing old data. To fix this I decided to use distributed cache coordination. Read below to find out how to configure and run distributed cache coordination using EclipseLink JPA2. At the end of the article there is a link to working example.

The problem and the fix 

So, in my case both web and integration applications use the same data access layer (JPA2 and EclipseLink). But while Apache Camel route was updating statuses with every successfully completed step, the web application was still showing old statuses. I decided to use EclipseLink cache coordination over JMS (RMI and CORBA are supported as well). Ideally I would use AMQP instead of JMS, but still JMS is a better choice than RMI :)

Prerequisite

You need a JMS broker. We all know that ActiveMQ is the best option so go ahead and download it from here: http://activemq.apache.org/download.html. Below I used default settings so all you have to do is just start the ActiveMQ broker.

Updating persistence.xml

I had to add the following 3 lines to my persistence.xml file:

<property name="eclipselink.cache.coordination.protocol" value="jms" />
<property name="eclipselink.cache.coordination.jms.topic" value="jms/Qualitas.EclipseLinkCacheTopic" />
<property name="eclipselink.cache.coordination.jms.factory" value="jms/Qualitas.EclipseLinkCacheConnectionFactory" />

And that was all. Almost all, as then I had to configure JNDI contexts on both web application (easy) and Apache Camel (not that easy) ends.

Configuring JNDI context in Jetty

I'm using Jetty 7.6. First I had to create WEB-INF/jetty-env.xml file and define my resources:
 

<?xml version="1.0"?>
<Configure class="org.eclipse.jetty.webapp.WebAppContext">
 <New id="eclipseLinkCacheConnectionFactory" class="org.eclipse.jetty.plus.jndi.Resource">
  <Arg>jms/Qualitas.EclipseLinkCacheConnectionFactory</Arg>
  <Arg>
   <New class="org.apache.activemq.ActiveMQConnectionFactory">
    <Arg>nio://localhost:61616</Arg>
   </New>
  </Arg>
 </New>
 <New id="eclipseLinkCacheTopic" class="org.eclipse.jetty.plus.jndi.Resource">
  <Arg>jms/Qualitas.EclipseLinkCacheTopic</Arg>
  <Arg>
   <New class="org.apache.activemq.command.ActiveMQTopic">
    <Arg>Qualitas.EclipseLinkCacheTopic</Arg>
   </New>
  </Arg>
 </New>
</Configure>

  Then I had to add proper mappings to my web.xml file:

<resource-ref>
 <res-ref-name>jms/Qualitas.EclipseLinkCacheConnectionFactory</res-ref-name>
 <res-type>javax.jms.QueueConnectionFactory</res-type>
 <res-auth>Container</res-auth>
</resource-ref>
<resource-env-ref>
 <resource-env-ref-name>jms/Qualitas.EclipseLinkCacheTopic</resource-env-ref-name>
 <resource-env-ref-type>javax.jms.Topic</resource-env-ref-type>
</resource-env-ref>

 As promised, configuring JNDI context in web application was easy. To be sure it actually worked I ran the web application and opened ActiveMQ web console. I saw Qualitas.EclipseLinkCacheTopic and I saw one active consumer.

Configuring JNDI in Apache Camel

This took me some time. To cut long story short, in order to configure JNDI and bind objects to it, you have to follow these steps:

1. Create factory and topic In your Spring configuration file copy and paste:

<amq:topic id="eclipseLinkCacheTopic" physicalName="Qualitas.EclipseLinkCacheTopic" />
<amq:connectionFactory id="eclipseLinkCacheConnectionFactory" brokerURL="nio://localhost:61616" /> 

2. Create JNDI context

To cut long story short. I first wanted to try Sun's reference implementation (as there are many sites in google that refer to it), but it's not a part of JDK any more. I also wanted to try Apache OpenEJB (as I'm a huge fan of Apache Geronimo), but I was scared with all the dependencies I saw in my console. After some googling I found a lightweight JNDI implementation from... Jetty :) Here's how to configure Spring's JndiTemplate with Jetty's JNDI implementation:

<bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate">
 <property name="environment">
  <props>
   <prop key="java.naming.factory.initial">org.eclipse.jetty.jndi.InitialContextFactory</prop>
   <prop key="java.naming.factory.url.pkgs">org.eclipse.jetty.jndi</prop>
  </props>
 </property>
</bean>

 3. Bind connection factory and topic to JNDI context

I had JNDI directory, next I had to bind my connection factory and topic to JNDI context. I wrote a pretty naive implementation using Spring's autowiring (it was naive because it would fail when there would be more autowire candidates, but was OK for me):

@Component
public class JndiBinder {
    @Autowired
    private JndiTemplate jndiTemplate;
    @Autowired
    private ConnectionFactory eclipseLinkCacheConnectionFactory;
    @Autowired
    private Topic eclipseLinkCacheTopic;
    @PostConstruct
    protected void bindEclipseLinkCacheResources() throws NamingException {
        jndiTemplate.bind("jms", new InitialContext());
        jndiTemplate.bind("jms/Qualitas.EclipseLinkCacheConnectionFactory", eclipseLinkCacheConnectionFactory);
        jndiTemplate.bind("jms/Qualitas.EclipseLinkCacheTopic", eclipseLinkCacheTopic);
    }
}

 That was all.

Running the sample 

When I started Jetty and Apache Camel, in ActiveMQ web console I saw two consumers attached to Qualitas.EclipseLinkCacheTopic. And when I uploaded a sample WS-BPEL process to my web application I saw UPLOADED status, but after a few splits of a second I saw PROCESSING status and after a few seconds I finally saw INSTALLED status. All worked like a charm.

If you're interested in full source code, Qualitas is hosted on both googlecode: http://code.google.com/p/qualitas/source/browse/ and GitHub: https://github.com/lukasz-budnik/qualitas. For now it's in master branch, but in one week time it will be a part of 0.0.5-SNAPSHOT release and will be tagged as 0.0.5-SNAPSHOT. This tag will be of course stable. Master branch may not be always stable. Before cloning master branch make sure CouldBees' Jenkins is saying that it's safe to do so: https://qualitas.ci.cloudbees.com/job/Qualitas/modules. Qualitas is a distributed system and a few things have to be configured in order to run it. Please refer to http://code.google.com/p/qualitas/wiki/BuildingTheProject and http://code.google.com/p/qualitas/wiki/RunningTheProject wiki pages.

Summary

Any comments most welcomed. It would be nice to have AMQP-based cache coordination in EclipseLink. I'm thinking about volounteering to implement this feature :)