Jeff Davis on Open Source SOA
DZone recently sat down with Jeff Davis, author of Manning's Open Source SOA to discuss the various factors that need to be considered when selecting an open source, SOA-enabling framework. Jeff's book discusses several SOA frameworks from the open source universe such as Apache Tuscany, an open source SCA implementation; Apache Synapse, a message mediator; JBoss JBPM a Java-based, open source BPM suite; and JBoss Drools, an open source rules engine.
Dzone readers can get 30% off Open Source SOA by using the code "dzone30" while checking out with any version of the book at http://www.manning.com
Jeff: I've been working in software development for about 20 years, with the past 10 or so pretty much exclusively with the Java platform and enterprise software. About 5 years ago I joined HireRight as their System Architect, and we faced some enormous challenges in trying to integrate with a wide variety of partners and clients. At that time, open source ESBs weren't all that mature (really Mule and ServiceMix were just getting established, so we went the commercial route. After a few years, we realized that there was a multitude of very high quality open source offerings, ranging from ESBs, BPM products, business rules management systems and the like. The real challenge is how to integrate these excellent open source applications into a comprehensive suite and levering the strengths of each of the products into what I call a SOA platform. Turns out that was a pretty big challenge, if only because there is often overlap between the products in what they can do and the challenges surrounding the different libraries that are used etc. Being a huge fan of FOSS, I wanted to share some of my experiences with the larger community as a way of giving back to it.
DZone: There are many SOA related products and some of them are well designed and implemented. What criteria did you use in deciding which frameworks you would cover in your book?
Jeff: This clearly was the biggest challenge I faced in the book. In particular, some of the product areas I covered have many quality open source products in which to choose, such as in the ESB space. Others, such as complex event processing and business rules, there are fewer choices (but the products that are available in open source are outstanding). In part, the selections I made were based on my own experience, and also how well I thought they could be integrated together. For example, Apache Synapse is a very lightweight ESB, and for many organizations, I suspect that's all that is required, especially when you are using a service framework such as Apache Tuscany, which I cover in the book. I wanted to pick the products in each category such as business rules, BPM, ESB, service/component frameworks, and business rules that I thought fit well together, and demonstrate how they could be tightly integrated. In doing so, I think it provides organizations the foundation for a very robust open source SOA platform.
DZone: I believe CEPs and ESPs have not found their place in the community yet, and some problems which can easily be solved using frameworks like Esper are resolved using RDBMS and messaging. You have introduced Esper in your book. What are the possible use cases for Esper and how does your book help readers understand event-processing?
Jeff: You're right, Esper is one of the real jewels of open source SOA products, but doesn't seem to be very widely adopted. I suspect it's because you have to weave CEP into the design of your solution. If you try and bolt-on event processing after-the-fact, it's pretty tedious and difficult to do. However, if you design for it up front, it's really very simple and incredibly powerful. For example, each node in a BPM or BPEL process can be configured, if designed up-front, to generate events that can be easily consumed by a CEP engine such as Esper. This enables you to gain real-time visibility into what's happening within your organization. Unlike the data warehouse approach, trends and problems can be identified immediately, and steps taken to correct any issues that may otherwise fall undetected. For example, you could use CEP to very effectively monitor whether any unusual activity is taking place in any public APIs that your organization has published. Maybe you normally get 500 API hits an hour, but now you are seeing unusual ordering volume. Or, perhaps bottlenecks are occurring in order processing that is done through a BPM process. These can be readily identified using CEP. Granted, all this can be often be done by coding some logic within the individual application, but it's far better from a consistently and maintenance standpoint to manage it using CEP. Using a CEP, you can simple fire off events from the particular application such as an ESB, and let the event processor manage the logic and filtering of them. A RDBMs is really not suitable for handling and filtering the volume of messages that a CEP can manage.
DZone: You covered SCA in your book by introducing the Tuscany project, and none of the projects you covered implements JBI. Wherever we see SOA, to some extent we see ESBs implementing JBI. What are your thoughts on the JBI and SCA story? Are they competitive or complementary technologies?
Jeff: I used to be a big fan of JBI, but sadly, it seems as though it is one of those standards that never really gained a lot of acceptance. Maybe it is due to the complexity of building JBI type components (thought products like ServiceMix have greatly streamlined it through Maven etc). In any event, Mule and other non-JBI ESBs make a pretty good case that their solutions are entirely based on open source technologies, but just don't adhere to JBI.
As for comparing JBI and SCA, I really seem them as fulfilling different types of needs. With SCA, it's a framework for building reusable services and components. In that sense, it's more similar to OSGi, but on broader scale (i.e., not tied to a JVM instance). JBI, on the other hand, is really more for creating components specific to an ESB. I think when ESBs first emerged, people embraced them as the solution for SOA, but much of that thinking has matured, I think. An ESB, in my opinion, is really best suited for integration, not for building services and components, which is the suite-spot for SCA.
DZone: jBPM is the most well known open source known BPM implementation in the Java community. How does jBPM tie in with SCA?
Jeff: I think jBPM and SCA are really complementary, as I point out in the book. jBPM is really about building orchestrations or business processes, and those are constructed from services. Those services can be nicely developed using the SCA framework. Since SCA allows you to publish services in so many different protocols (SOAP, REST/POX, JMS, RSS, etc), it gives you great flexibility for interacting with them. In the book I demonstrate how jBPM nodes can use SCA as a client to call external services, and also how to expose jBPM processes as a service. Thus, you can launch a jBPM process using JMS, SOAP, REST or any of the other available interfaces that SCA provides. The real beauty of SCA is that you aren't coding protocol-specific logic into your Java code (or the various scripting languages supported such as Ruby, Groovy etc) -- instead, how your expose your services is managed declaratively through a pretty simple XML configuration.
DZone: Apache Synapse is a well known message mediator which can be used to mediate messages in a non-intrusive way. How do you see the future of message mediation in general and Apache Synapse in particular?
Jeff: The thing I really like about Synapse is how lightweight and easy-to-configure it is to use. I'm not sure that your services that are exposed to external customers and partners should be directly provided through something like SCA, JAX-WS etc. Instead, by proxying service requests through Synapse, you can more easily manage the security and provide greater transparency to service endpoints (i.e., you can shuffle around the location where there service is actually fulfilled, all the while keeping that shielded from the client/consumer of the service). Synapse also has some pretty nifty capabilities when it comes to managing service levels and governance, so you can restrict or limit the number of calls someone can make into a particular service. WSO2 is doing a lot of work with their ESB product with OSGi, and I presume some of that will flow into the next Synapse release (WSO2 is the sponsor behind Synapse, and it's the basis for their ESB offering, which is very good).
DZone: Covering JBoss Drools in your book make the book a complete guide of Open Source SOA, can you explain more about rule engines in general and JBoss Drools in particular? How well does JBoss Drools scale and how flexible it is?
Jeff: Drools is really an amazing product, and JBoss has done an outstanding job in how it has continually advanced the product. Sadly, I don't see a lot of enterprises using it. The real challenge, as I point out in the book, is how do you effectively integrate a rules engine into your application? Like Esper, it really needs to be done up-front. One of the possible approaches I present in the book is treating rules as a decision service. Thus, you invoke it when a decision needs to be rendered, but the decision service itself is decoupled from the application, insofar as it doesn't actually perform any business logic directly. This approach is consistent with SOA, and treats rules as reusable service. Couple that with a rules management system such as Guvnor that comes with Drools, and you can manage all your operation business rules centrally. They represent, after all, the IP of most organizations. Burying those rules in code makes changing them tedious and inflexible. In the book, I have a case study that demonstrates how a Drools decision service can be exposed using the SCA framework. Thus, you can easily invoke the decision service through multiple protocols.
DZone: Have you come across any interesting, real-world use cases that employ Drools?
Jeff: A recent example I've seen where someone is using the Drools framework I've described in the book is for creating a validation service. In this case, they are developing interface files that are generated from an ERP type system that are sent to various health care providers. A lot of validation rules must be applied to "scrub" that data prior to it being sent, to ensure that it is processed correctly when it is received. Using Drools, these validation rules can be expressed in a fairly easy-to-understand form digestible by the subject matter experts. The validation service is called prior to sending the files, and any discrepancies found can be fixed up-front. Historically, these rules would be expressed in code, but doing so can very complicated and messy. Now, with this validation service, they can easily test the rules using testing tools such as soapUI, and it can be invoked as a stand-alone service.
I also describe in the book about Drools RuleFlow. It's a very interesting technology, which basically allows you to create simple business processes from within the rules engine directly (it incorporates jBPM technology). This may seem counter-intuitive, but business rules are often what drive a business process, so it makes a lot sense. For many organization, Drools and its RuleFlow capabilities may obviate the need for a full-fledged BPM.
DZone: SOA is a very broad topic and everyday we see a new door opened and a new piece of software introduced to the community to address some part of the requirements. In your opinion, how mature is the open source SOA market?
Jeff: Well, maybe I am just biased, but I struggle to see any real areas where their isn't a good open source solution available. However, it may be a niche product, and the struggle always is, how do you fit it together. This is where historically commercial products have excelled -- they provide a complete integrated solution. The products are there in the open source space, but sometimes, you have to work a bit to make them all fit together nicely. Hopefully, my book can help with that challenge.
DZone: Can you give us a few tips and advice on selecting the right SOA frameworks along with pitfalls which we should avoid in choosing such products?
Jeff: One of the most important things to do is assess how vibrant the community is for a given open source product. Check the forums and mailing lists to check for activity. Also, check out the source code and look through the unit tests that are available -- it's a fantastic way to learn more about the product. Lastly, check the licensing and open source model used. A lot of products are touted as open source, but when you get under the covers, you find that the real enterprise features are saved for some commercial alternative. I tend to frown on companies using that approach, but understand they are trying to survive in a difficult market.
DZone: Any final words for our members, Jeff?
Jeff: I continue to be amazed at the quality of many open source products. For example, I've recently began using XAware, and it is a fantastic ETL-type product. If you use an open source product, please give back to it in some fashion, either through financial support, contributing back to the code base (or documentation), or other means. Let's continue to support this community so that we can all benefit!