Open Source Integration With Apache Camel and How Fuse IDE Can Help

Take any integration project and you have multiple applications talking over multiple transports on multiple platforms. As you can imagine, in large enterprises, applications like this can get complex very fast. Much of the complexity stems from two issues:

1. Dealing with the specifics of applications and transports
2. Coming up with good solutions to integration problems

Making your applications speak transports and APIs is relatively easy on its own. I'm sure everyone knows how to send JMS messages to their broker of choice; though it still requires in-depth knowledge of the JMS specification, which many developers may not have. On top of that, what happens when you want to route that JMS message to another application? You then have to take care of mapping the JMS message to the application plus handle any new concepts related to the application. Add a dozen other applications into the mix and you've got quite a headache on your hands.

Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high-level design of how applications interact. Fortunately, most solutions to enterprise integration problems have been formalized already. Gregor Hohpe and Bobby Woolfe's book, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, boils down years of experience from enterprise architects into a set of sixty-five Enterprise Integration Patterns (EIPs). This is great, but we still have to hand-code all parts of these patterns; these are not packaged solutions, only recommendations.

Apache Camel was created with the intention of addressing these two issues. In this article, I'll first give you a quick overview of what Camel is. I’ll then guide you through how Camel solves a sample integration problem. Finally, I’ll show off some new graphical tooling that makes it even easier to solve integration problems with Camel.

What Is Camel?

Apache Camel is an open source Java framework that focuses on making integration easier and more accessible to developers. It does this by providing:

• Concrete implementations of all the widely used EIPs

• Connectivity to a great variety of transports and APIs

• Easy-to-use Domain-Specific Languages (DSLs) to wire EIPs and transports together

Figure 1 shows how these three items actually map to Camel concepts. To give you a good understanding of how Camel is organized, we will discuss Components, Endpoints, Processors, and DSLs. There is, of course, a lot more going on here under the hood, but we'll leave that for another discussion.

from ("file:/tmp").to("jms:aQueue");

<route>
  <from uri="file:/tmp"/>
  <to uri="jms:aQueue"/>
</route>

from "file:/tmp" -> "jms:aQueue"

Figure 3: This shows the solution to Rider Auto Parts integration problem using notation from the Enterprise Integration Patterns book.

So we have several patterns in use here.

1. There are two Message Endpoints; one for FTP connectivity and another for HTTP.

2. Messages from these endpoints are fed into the incomingOrders Message Channel

3. The messages are consumed from the incomingOrders Message Channel and routed by a Content-Based Router to one of two Message Translators. As the EIP name implies, the routing destination depends on the content of the message. In this case, we need to route based on whether the content is a CSV or XML file.

4. Both Message Translators convert the message content into a POJO, which is fed into the orders Message Channel.

The whole section that uses a Content-Based Router and several Message Translators is referred to as a Normalizer. This composite pattern has a unique graphic to depict it but was left out here in favor of its sub-patterns to make things clearer.

Implementation Using Camel

As mentioned before, Camel has a small core set of components included by default. The rest of the components exist as separate modules. In applications that require many types of connectivity, it is useful to figure out what Camel modules to include. Listing 1 shows the dependencies using Apache Maven for the Camel implementation of the Rider Auto Parts example. Of course, you don't need to use Apache Maven for dependencies - it is just the easiest way to rapidly add new dependencies to your applications. The list of dependencies includes support for core Camel, ActiveMQ, JAXB marshaling, CSV marshaling, and HTTP. To make the example easier to try out, I've opted to use the File endpoint instead of the FTP. If we were using the FTP endpoint, we would need to add a dependency on the camel-ftp module as well.

Listing 1: Maven dependencies for the Camel implementation

<dependencies>
    <!-- Core Camel -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-core</artifactId>
      <version>${camel-version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-spring</artifactId>
      <version>${camel-version}</version>
    </dependency>

    <!-- Embedded ActiveMQ broker -->
    <dependency>
      <groupId>org.apache.activemq</groupId>
      <artifactId>activemq-core</artifactId>
      <version>${activemq-version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.xbean</groupId>
      <artifactId>xbean-spring</artifactId>
      <version>${xbean-spring-version}</version>
    </dependency>

    <!-- ActiveMQ connectivity for Camel -->           
    <dependency>
      <groupId>org.apache.activemq</groupId>
      <artifactId>activemq-camel</artifactId>
      <version>${activemq-version}</version>
    </dependency>

    <!-- Add support for JAXB marshaling -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-jaxb</artifactId>
      <version>${camel-version}</version>
    </dependency>

    <!-- Add support for CSV marshaling -->   
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-bindy</artifactId>
      <version>${camel-version}</version>
    </dependency>    

    <!-- Add support for HTTP -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-jetty</artifactId>
      <version>${camel-version}</version>
    </dependency>

  </dependencies>

While it is perfectly legitimate to use Camel as a standalone Java application, it is often useful to embed it in a container. In this case, we will be loading Camel entirely from Spring. In fact, the entire solution (except for the domain POJO and Maven build script) fits neatly into a single Spring XML file. This is because we are implementing our routing rules using the Spring XML configuration rather than one of Camel’s other DSLs. There are pros and cons to each, of course, but in this case, I mainly wanted to show off some cool tooling that currently works with the XML based routing rules only. The Spring XML file is shown in Listing 2 below.

Listing 2: Complete Spring XML file that configures an embedded ActiveMQ broker and initializes a Camel Context with three routes.

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:amq="http://activemq.apache.org/schema/core"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
       http://activemq.apache.org/camel/schema/spring
       http://activemq.apache.org/camel/schema/spring/camel-spring.xsd
       http://activemq.apache.org/schema/core
       http://activemq.apache.org/schema/core/activemq-core.xsd">

  <amq:broker brokerName="localhost" persistent="false" useJmx="false"/>

  <bean id="jms" class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="brokerURL" value="vm://localhost" />
  </bean>

  <camelContext xmlns="http://camel.apache.org/schema/spring">
    <route id="FileToJMS">
      <from uri="file:target/placeorder" />
      <to uri="jms:incomingOrders" />
    </route>

    <route id="HTTPtoJMS">
      <from uri="jetty:http://0.0.0.0:8888/placeorder" />
      <inOnly uri="jms:incomingOrders" />
      <transform>
        <constant>OK</constant>
      </transform>
    </route>

    <route id="NormalizeMessageData">
      <from uri="jms:incomingOrders" />
      <convertBodyTo type="java.lang.String" />
      <choice>
        <when>
          <simple>${body} contains '?xml'</simple>
          <unmarshal>
            <jaxb contextPath="org.fusesource.camel" />
          </unmarshal>
          <to uri="jms:orders" />
        </when>
        <otherwise>
          <unmarshal>
            <bindy packages="org.fusesource.camel" type="Csv" />
          </unmarshal>
          <to uri="jms:orders" />
        </otherwise>
      </choice>     
    </route>
  </camelContext>
</beans>

In this file, we first start an embedded Apache ActiveMQ broker and connect Camel to it. Next comes the camelContext element where we define our routing rules. Looking back at Figure 3, we need to receive orders from an FTP (substituted with File in this example) and HTTP endpoint, formatted as shown in Listing 3. In the Spring XML configuration, we can specify these incoming endpoints with two from elements. The “FileToJMS” and “HTTPtoJMS” routes start off with these from elements. Both from elements are connected to a producer (to and inOnly elements) using the “jms:incomingOrders" URI, which will send the messages to a queue named incomingOrders on the ActiveMQ broker.

Listing 3: Incoming message formats; XML on the left, CSV on the right.

In the case of the HTTP endpoint, there are a couple of extra things to mention. First off the HTTP client will be expecting a response from the application so we have to handle that. In Camel, we have full control over what the client gets back from the HTTP endpoint. Each response is determined by the last method in our current route definition. In our case, we use the transform method to set the response to the constant string "OK". Since we handle the response ourselves, we don’t want any response to come from the JMS incomingOrders queue. To send to this queue in a fire-and-forget fashion we use the inOnly element rather than the to element.

The “NormalizeMessageData” route in Listing 2 specifies the Normalizer, complete with Content-Based Router and two Message Translators. First, we specify that we want to consume messages from the incomingOrders queue on the ActiveMQ broker. The content-based routing of the messages is done with the choice, when, and otherwise elements. In our case, we want to send CSV messages to one Message Translator and XML messages to another. To check what type of message data format we have we use a Simple expression which checks the message body for the “<?xml” start tag. Simple is an expression language built into Camel. Of course, this is demonstration code only. For production cases, you would want to add a more thorough checking of content types.

To unmarshal the XML and CSV payloads into an Order object, we use Camel’s data formats. Listing 2 shows how these data formats are defined within the unmarshal elements. As shown in Listing 5, the Order object has JAXB and Camel Bindy annotations to describe the mapping to XML and CSV.

Listing 5: The Order domain class with JAXB/Bindy annotations for easy mapping to and from XML/CSV.

@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
@CsvRecord(separator = ",", skipFirstLine = true)
public class Order implements Serializable {
    @XmlAttribute
    @DataField(pos = 1)
    private String name;

    @XmlAttribute
    @DataField(pos = 2)
    private int amount;

   public Order() {
    }

    public Order(String name, int amount) {
        this.name = name;
        this.amount = amount;
    }
}

At this point, successfully normalized messages are sent to the orders queue for processing by some other application at the Rider Auto Parts business.

Implementation using Fuse IDE for Camel

As we’ve seen, Apache Camel certainly makes it easy to create integrations. The DSLs in various languages are designed to be concise, powerful and easy to read – all with the goal of making you, the developer, more productive. With improving productivity in mind, FuseSource has created an IDE for Camel development called Fuse IDE for Camel.

Fuse IDE is an Eclipse-based tool that allows you to drag and drop enterprise integration pattern (EIP) icons onto a canvas and connect them together to form routing rules. The backend of the IDE generates Camel routing rules in Spring XML for you so you don’t ever have to learn the specifics of Camel’s DSLs. Essentially, you can create a picture like Figure 3 and have Fuse IDE generate the Camel application for you.

Figure 4 shows what the Fuse IDE designer looks like when we loaded our example configuration from Listing 2.

Figure 4: Spring XML file in Listing 2 loaded into Fuse IDE’s designer.

From the designer view, we can see the graphical representation of each of the routes we created in Spring XML beforehand. From this point, we can change how the EIP icons are connected, endpoint URI properties, add additional steps to the routes, and so on. Also, as of version 1.0, Fuse IDE will support all EIPs and components available in Apache Camel — so there is no waiting for version 2.0 to use your favorite Camel features.

In this article, we started first by creating the Spring XML configuration by hand. This should not mislead you, however, there is no requirement to do this. In fact, since Fuse IDE supports 1-to-1 round-tripping between Camel Spring XML configuration and the designer view, you can use whichever you prefer. 

When you have finished constructing your routes, you can then take advantage of another neat feature included in Fuse IDE: that is the ability to generate a JUnit test case from any Camel Spring XML file. Once you are satisfied with your testing, you can then debug in a local Spring container or deploy to your container of choice — all from within the IDE. Of course, much more is planned for Fuse IDE in the future, so Camel development will only get easier.

Summary

In this article, I've shown two common problems that an integration developer may face: dealing with the specifics of applications and transports and coming up with good solutions to integration problems. The Apache Camel project provides a nice answer to both of these problems. As the example has shown, solving integration problems with Camel is straight forward and results in very concise route definitions. Building on that, we saw how Fuse IDE for Camel makes Camel development even easier.

Links

1. Apache Camel – http://camel.apache.org
2. 2FuseSource distribution of Apache Camel – http://fusesource.com/products/enterprise-camel
3. Fuse IDE for Camel - https://tools.jboss.org/features/fusetools.html
4. Camel in Action book - http://www.manning.com/ibsen
5. Jon’s Blog – http://janstey.blogspot.com
6. Article source code - http://repo.fusesource.com/maven2/org/fusesource/examples/rider-auto-spring/2.0/rider-auto-spring-2.0.zip

Originally published on 7/1/15