Navigating the Integration Landscape - Claus Ibsen on Apache Camel

 DZone had a chance to sit down with Claus Ibsen, author of the recently published Refcard Enterprise Integration Patterns with Apache Camel. In this interview, Claus, a Principal Engineer at Progress Software in the FUSE team, discusses best practices and pitfalls associated with application integration, and how Apache Camel addresses some of these challenges.  He surveys the open source ESB landscape, describes where Camel fits in and gives some valuable advice on how to properly evaluate and select the right ESB solution.

Claus Ibsen: Yeah sure. I hold a position as Principal Engineer at Progress Software in the FUSE Team (http://fusesource.com), where we specialize in open source integration. I am very fortunate to work full time on Apache Camel and our FUSE empowered products and tools.

I started in the IT industry back in 1998 as a traditional developer on the AS/400 platform. I later changed to develop software on the PC platform doing integration to the AS/400. And in the year 2000 I jumped on board the Java platform developing middleware on top of the AS/400 platform. So I started my career doing software integration using legacy and proprietary protocols such as EDI, MQ, socket, batch files etc. The dot com era took me to the land of the J2EE and web. I was technical architect and lead developer on several major projects using the latest J2EE stack money could buy. The bubble burst and in 2003 I was contracted as a technical architect and consultant to work in China (Shanghai), on a new integration infrastructure for custom clearance for the local government in Shanghai. The system handles a very high load of transactions per day as its handling payments between governments, banks and transport agencies for to entire Shanghai region. It was a great experience to live and work in a different culture than the western. A year later I returned to Denmark, and the time thereafter I have mostly been a client side consultant helping clients with their integration needs.

In terms of open source, my “career” started as many others by trying frameworks and contributing back patches and fixes. As I am curios I downloaded the source code and started digging into the internals. I helps me better understand the framework and also lear new skills. That lead my into as a committer for both Apache Camel and many years ago on the WebWork/Xwork projects.

I have also developed a plugin for IDEA and released it as open source. I started using IDEA in 2002 and at that time I needed it to be able to dump logging information for my model objects and therefore I developed the generate toString() plugin. This plugin is now in safe hands as its bundled out of the box with IDEA.

DZone: How did you become involved in the Apache Camel project?

Claus: It started, when I was working as a consultant for my last client, as we where looking for a new integration platform, to replace Sun JCAPS. My client was facing difficulties to find talented people that had or wanted to work with this platform. And the platform itself has a very poor productivity and ease of use. They where looking for something that was much more agile, flexible, leaner and most importantly can run in their existing platform and infrastructure. Something that SUN had promised them, but they couldn't keep.
As I used my spare time to track emerging technologies and try out frameworks that potentially can make my day work easier, I spotted several open source frameworks such as Apache Camel, Apache ServiceMix and Mule that could fit in my clients organization. I started digging into Camel and along the way I submitted patches for bug fixes and new features etc. 3 months later James Strachan invited me in as a committer and I think the Camel Community can testimony that I have been active and devoted since. 

DZone: Apache Camel is in the line with some other Apache Foundation integration projects including ServiceMix, and Apache Synapse. Is there any relation between these two projects and Apache Camel?

Claus: Camel grew out of our experiences on ServiceMix. There was a component in ServiceMix called the eip-component which acted as the inspiration for Camel; we basically decided to spin off a separate EIP implementation using a very simple JBI-like API that could be used within ServiceMix to route between JBI endpoints - but also be usable in any other environment such as a web application, a JMS broker or web services gateway. 

DZone: It would appear that many much of the functionality that you cover in the Camel Refcard can be implemented, in a fairly non-intrusive way, using a message mediator like Apache Synapse. Why would I choose to go with Camel instead of Apache Synapse?

Claus: Here's a link to a comparison.

Camel is way more popular with a bigger more active and diverse community - its almost as popular as Struts and way more popular than all of the Apache Web Services projects put together (so more popular than Axis + Synapse et al): http://www.nabble.com/Apache-f90.html

Camel is developed first and foremost around the Enterprise Integration Patterns as a first class citizen - that is the logical model in Camel which you can work with in the DSL of your choice: currently either Java, XML, Scala, Groovy, Ruby - or Java beans with annotations. So it tries to raise the abstraction level to that of the patterns rather than be low level plumbing like previous integration technologies tended to be.

For example Synapse started life as a mediation engine for Axis which is a very different focus. 

DZone: Other than Apache Synapse, are there any other projects out there that provide the level of integration that Camel provides?

Claus: I would say that Mule and Apache ServiceMix are the only projects I am aware of that are on par with the level we have in Camel. Spring Integration is a fairly new project and it has a lot of catch up to do.

DZone: Performance is always a big concern for integration professionals. Could you share some of your experiences with Camel performance in the context of high throughput systems?

Claus: Camel is a lightweight framework; it typically doesn't affect performance in and of itself a whole lot - what does affect performance greatly is the actual endpoints and their configuration. e.g. if you are using JMS then the pooling and concurrency configuration is way more relevant to whether you use Camel or native JMS etc.

Another common performance gotcha is when doing lots of XML/XPath/XQuery routes its often advisable to transform payloads into some XML format (DOM or JAXB bean or ADS data objects) once to avoid multiple parses for each expression evaluation.

Finally another trick to boost performance when doing transacted messaging; maybe with XA - is to use transaction batches which can boost latency or perceived throughput for a single thread (though if you are running many parallel processing threads it often doesn't make a huge difference to overall system throughput). 

DZone: What 'best practices' would you advocate for folks using Camel on integration projects?

Claus: Start simple and test early using our testing framework.

James Strachan has really put great innovation into this kit that we use extensively to test Camel itself. It works well with Spring Testing or regular Junit classes. You can use spring property placeholders for the actual server names so you can build a single integration route as skeleton for all your environments.

Camel is so powerful and easy to get things done; most patterns are a single line of Java code - that folks often run before they can walk. By testing early and iteratively adding more features & requirements its easier to spot why things go wrong.

You can mix the Spring XML configuration with the power of the Java DSL.

Decouple your business and integration logic from the Camel API using Camels powerful Bean Binding. This allows you to have all your logic with no dependency at the Camel API at all.

A recent addition is the Camel web console.

That is used for runtime visualization of routes. You can browse the messages on the endpoints and you can also use it to send in new messages. Its also useable to control  routes by stopping, starting or even edit them at runtime.

And for old fashioned logging in log files we have the Tracer.

That will trace all messages and how they are routed at runtime. If it all goes to fast for you to keep track you can globally reduce the pace of the routing and thus at runtime tail a log and easier spot what happens.

DZone:  What are the most common pitfalls when using Apache Camel?

Claus: A given problem will often have multiple solutions in Camel. So choosing which solution to use in your situation or what is the best practice can lead to a bit confusion at first. Folks don't tend to use the EIP designer and debugger that often so when things go wrong its often a bit hard to visualize the routes, put breakpoints at points in the flow and debug the route in an EIP way. Its trivial to add a bean/processor within a route and use a regular Java debugger; but its way simpler using the FUSE tooling. 

DZone: When evaluating an ESB product, what are the main comparison criteria?

Claus: It tends to depend on what you need from an integration solution. Do you want a web services intermediary; a general purpose integration platform and framework or just a way to develop distributed SOA applications?

Write down your requirements. You are looking for a product that meets your requirements and not which products that has most features marked in a feature matrix.

An in the open source space you should look for a product that has a professional organization backing it. You need someone to engage for support, training, and being active maintaining and growing the community around the product.

And you should definitely look at the community. So look for how big and active is the community? How diverse it is. For instance if its a single VC backed company behind it, then who knows if it'll get dropped as a product soon?

You should also look for if the product is actively maintained and developed. The world in the IT landscape changes very rapidly, so you don't want to end up running your business on a product that gradually decline to nothing. Look for activities in the forums, mailing lists, commit log, issue tracker, blog entries, articles and books and the likes.

And when you try out the product put questions out on the forum or mailing lists to see if there is any help and response. If the product lack a feature or you have found a glitch then submit a ticket to the issue tracker and check the response. If your feature requests isn't in the roadmap for the product maybe the product isn't moving in a direction that suits your business.

Then the product itself should support standards at both the API and network layers. Standard APIs such as: REST, WS-*, JMS/Stomp/AMQP for messaging, XMPP and other of the new ones.
The product should be modular and plugin friendly. It should be possible to provide your own modules/components to the product so they work seamless with the product as if they are shipped of the out of the box.

And last but not at least about the product itself, how lightweight and flexible is it? Can you use any of your existing platform and infrastructure as runtime, or do you need to introduce a new platform to the operations division? They would sure love if it can be hosted seamless into their existing mix of servers and infrastructure.

And often overlooked but one of the key points in my optic - Power back to the developer. At any given problem it should be possible for the developer to handle a problem. I use the picture - there should be a “sweat spot” where the cursor is blinking and the developer feel home where he can write code that solves the problem. I have often seen integration with legacy system where you need to cater for all kind of corrupt data it spit out. Its often harder and more expensive to get the legacy vendor to admit and fix the problem, than have your own integration team capable of dealing with it. Standards are fine but it should be possible to deal with systems that unexpected differs from the standard.

DZone:  What advice would you give to our members that are currently involved in integration projects?

Claus: No shoe fit all sizes. You should definitely take your time to try the different products. Take responsibly for your business and integration, your vendor does not run your business and your problems might not be on their priority list. That is why I think you should build a team with experienced resources that is capable of taking up this responsibility. Its is naive to let your business in the hands of the vendor and its army of endless consultants. Use less $$$ in the hardware and platform itself and more on bringing in the right talent to your team, and/or hire independent consultants in the role as your guard dog, capable of playing in the ballparks with the vendors. Integration is more about having the right team than the platform and hardware itself. No vendor have build the nirvana product that can substitute talent and experience.

If you want to learn a bit more about what Camel is and what it can do then take time and read this excellent article by Jonathan Anstey a fellow Camel committer.

And my last advice. Take a ride on the Camel, you wont regret it.