The Open eHealth Integration Platform (IPF) is an extension of the Apache Camel routing and mediation engine and comes with comprehensive support for message processing and connecting information systems in the healthcare sector. It is available under the Apache License version 2.0. IPF and Camel both focus on a domain-specific language (DSL) to implement and combine Enterprise Integration Patterns (EIPs) in integration solutions. IPF leverages the Groovy programming language for application development and for extending the Apache Camel DSL. One example of a healthcare-related use case of IPF is the implementation of interfaces for transactions specified in IHE profiles. The use of IPF to implement actor interfaces for the IHE PIX and PDQ profiles was tested successfully at the 2009 Connect-a-thon. The actual actors were proprietary systems that were IHE-enabled via IPF-based integration solutions. It is important to note that usage of IPF is not just limited to the healthcare domain. It can also be used to build integration solutions for several other domains. However, this article will focus primarily on the healthcare-specific features of IPF. Riding the Camel IPF is built on top of the Apache Camel routing and mediation engine. This section deals briefly with the concepts behind Camel. For a more detailed overview, read Jonathan Anstey's excellent introductory article Apache Camel: Integration Nirvana. Apache Camel is about Enterprise Integration Patterns (EIPs). EIPs have previously been described by Gregor Hohpe and Bobby Woolf in their book Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. They represent experiences of integration architects and can be used as building blocks for designing integration solutions. These patterns were published as recommendations without implying a concrete implementation. This is where Apache Camel comes in. Camel was created to provide implementations for EIPs. With Apache Camel, integration solutions are developed by combining integration patterns using a domain-specific language (DSL). In addition to EIP implementations, Camel also provides more than 60 components for connecting to a great variety of transports and APIs. The four major concepts in Camel are components, endpoints, processors and the domain-specific language (DSL). Components provide connectivity to other systems. From components, endpoints are created for sending and receiving messages. Endpoints are uniquely identified by their URIs. Processors are used to route and transform messages between endpoints. To wire endpoints and processors together, Camel provides a DSL for Java. Consider a simple example: from("jms:queue:validated") .filter().xpath("/person[@name='Martin']") .to("http://www.example.com/camel"); In this route definition, messages are read from a JMS queue (identified by the jms:queue:validated endpoint URI) and sent through a filter. The filter criterion is an XPath predicate. Any message that passes the filter will be POSTed to the http://www.example.com/camel endpoint. Camel also supports a Spring-based XML configuration for creating message processing routes (not shown here). Riding IPF Any feature provided by Camel can be used in combination with any other features provided by IPF. IPF makes its features available via extensions to the Camel DSL. These extensions have been implemented via Groovy meta-programming. For these extensions to be used in an IPF application, route definitions must be written in Groovy. Here is an example of a route definition that uses IPF's DSL extensions: from('file:input') // read HL7 message from file .unmarshal().ghl7() // create HL7 message object .validate().ghl7() // validate HL7 message object .filter {exchange -> // filter message using criteria defined by closure def msg = exchange.in.body // use exchange.in.body to access HL7 message object msg.PID[8].value == 'F' // use HL7 DSL to access and compare PID-8 field value } .to('http://...') // send message to destination (via HTTP In this route, an HL7 message is read from a file, validated and then filtered based on the value of the 8th field in the PID segment. To obtain this field value, the filter closure uses IPF's HL7 DSL. This DSL is explained in detail in the next section. If the message passes the filter it is forwarded to its destination via HTTP. The route definition uses the DSL elements from, unmarshal, filter and to from Camel directly, all other DSL elements are extensions provided by IPF, including the use of closures for filter criteria. IPF DSL extensions can be used as if they are native Camel DSL elements. Furthermore, the DSL used in applications is not just limited to the DSL provided by Camel and IPF. Application-specific DSL extensions can also be written using IPF's DSL extension mechanism. This mechanism also supports modularization of DSL definitions. Any component can contribute its own DSL extensions. Consequently, the scope of the overall DSL depends on which components were deployed for the application. Inside an OSGi environment, IPF can even automatically detect and activate DSL extensions provided by different bundles. The following sections will explore the healthcare-specific features of IPF in greater detail. For a detailed description of all IPF features consult the IPF reference documentation. For a more hands-on guide on how to write IPF applications, the IPF tutorials are a good starting point. HL7 message processing HL7 (Health Level 7) is an ANSI-accredited standards developing organization (SDO) in the healthcare sector. Among other standards it focuses on developing messaging standards for the clinical and administrative domain. This standard is often referred to as the HL7 messaging standard. There are several versions of this standard. HL7 version 2.x is widely used, the most recent update was version 2.6 in 2007. It differs significantly from the XML-based HL7 version 3. The following subsections focus on IPF's support for HL7 version 2.x. Support for HL7 version 3 is currently under development. IPF uses the open source HAPI library as its basis for HL7 version 2 message processing. HL7 version 2 at a glance HL7 version 2 messages have a hierarchical structure and are organized into groups, segments, fields and datatypes which can also repeat. Message elements are referenced by their path within the hierarchy (see navigation syntax in the following figure). HL7 DSL The core of IPF's HL7 message processing capabilities is the HL7 DSL. This is a domain-specific language for accessing and manipulating HL7 version 2 messages. The IPF library that implements this DSL (modules-hl7dsl) can also be used standalone (i.e. independent from other IPF components) in any Groovy application. Seamless integration into the IPF DSL is provided as well. The following examples presume a certain degree of familiarity with HL7 version 2 message structures and terms. Constructing messages The entry point to the HL7 DSL is IPF's MessageAdapter. To create a MessageAdapter instance from an HL7 file we use the load method from the MessageAdapters utility class. import org.openehealth.ipf.modules.hl7dsl.MessageAdapter import org.openehealth.ipf.modules.hl7dsl.MessageAdapters MessageAdapter message = MessageAdapters.load('ADT-A01.hl7') This loads the HL7 file ADT-A01.hl7 from the classpath. Accessing message content The following example obtains the PID segment contained in the PATIENT group which itself is contained in the first repetition of the repeatable PATIENT_RESULT group. // repeating elements are indexed from 0 to n def segment = message.PATIENT_RESULT(0).PATIENT.PID The function call operator () is used to refer to an element in a repetition. The element index is passed as argument. Obtaining fields is similar to obtaining groups and segments except that fields are often referred to by index rather than by name. To obtain the MSH-3 field from a message we can write: // fields are indexed from 1 to n def composite = message.MSH[3] If the field is a composite we access the second component with: // components of a composite field are indexed from 1 to n def primitive = message.MSH[3][2] or, equivalently, def primitive = message.MSH.sendingApplication.universalIDType As shown above, navigation is also possible using field names instead of indices. Care must be taken, because along with the change of internal message structures, individual field names change between HL7 versions, even when they refer to the same position of the field in a segment. If the version of the HL7 message is not known in advance, it is better to use the more concise index notation. Fields may also repeat. To obtain a given element of a repeating field the function call operator () is used, just like with groups and segments. In the next example we obtain the first element of the repeating NK1-5 field. Because NK1 is a repeating segment, the function call operator is used on segment-level, too. def field = message.NK1(0)[5](0) def fieldList = message.NK1(0)[5]() Omitting the repetition index returns a list of repeating elements. Omitting the function call operator, the first repetition of a group, segment or field is assumed. This is called smart navigation: assert message.NK1(0)[5](0)[1].value == message.NK1[5](0)[1].value assert message.NK1(0)[5](0)[1].value == message.NK1[5][1].value assert message.PATIENT_RESULT(0).PATIENT.PID[5][1] == message.PATIENT_RESULT.PATIENT.PID[5][1] If a component is omitted, the first component or subcomponent of a composite is assumed assert message.NK1(0)[5](0)[1].value == message.NK1[5].value assert message.NK1(0)[5](1)[1].value == message.NK1[5](1).value assert message.NK1(0)[2][1][1].value == message.NK1[2].value Using smart navigation, the expressions are usually shorter and less error-prone. Moreover, in many cases the same expression can be used for different HL7 versions, making the DSL more portable. Modifying message content A segment of one message can be assigned the segment value of another message using the assigment operator (=). The following example copies the EVN segment from message2 to message1. message1.EVN = message2.EVN To change a field value we navigate to that field (either by name or index, as shown above) and assign it a string or another field value. def msh = message.MSH def nk1 = message.NK1(0) msh[5] = nk1[4][4] msh[5] = 'abc' Composite fields may also be changed by assigning other composite fields. In the following example we copy the composite NK1(0)[4] field from message2 to message1. message1.NK1(0)[4] = message2.NK1(0)[4] Externalizing messages The left-shift operator (<<) appends the string-represenation of a MessageAdapter object to a writer. In the following example, we write an HL7 message to System.out. MessageAdapter message = ... System.out << message To obtain the string-representation directly, we can use the MessageAdapter.toString() method. MessageAdapter message = ... def rendered = message.toString() Usage in route definitions This section shows a few examples how to combine the HL7 DSL with IPF's route definition DSL. The unmarshal().ghl7() extension should be used to create a MessageAdapter object from an external HL7 message representation. The created MessageAdapter object can then be used in subsequent processors. from('file:input') .unmarshal().ghl7() .process {exchange -> MessageAdapter message = exchange.in.body ... } ... The reverse operation marshal().ghl7() marshals a MessageAdapter object into a stream. This is often needed for transmitting HL7 messages over a variety of transports such as JMS, for example. from('file:input') .unmarshal().ghl7() ... .marshal().ghl7() .to(jms:queue:validated) Another usage example is content-based routing. In the following example a message is routed to different destinations depending on the content of the MSH[4] field. Here we use closures in combination with Camel's when DSL element for implementing routing rules. The it variable inside closures represents a message exchange. from(...) .unmarshal().ghl7() ... .choice() .when { it.in.body.MSH[4].value == 'ABC' } .to(...) .when { it.in.body.MSH[4].value == 'DEF' } .to(...) .otherwise() .to(...) HL7 validation IPF also adds support for specifying validation rules in a way that is easy to write and simple to understand. It facilitates the definition of custom validation rules by providing a dedicated validation DSL. Like the HL7 DSL, the validation DSL can also be used standalone but also integrates well into the IPF DSL for defining message processing routes. The IPF component that implements HL7 validation is modules-hl7. Validation rules are defined by extending the ValidationContextBuilder class of IPF. The validation rules DSL is provided by the RuleBuilder class. The following example defines a subset of segments from the HL7 version 2.2 specification. package example import ca.uhn.hl7v2.validation.ValidationContext import org.openehealth.ipf.modules.hl7.validation.builder.RuleBuilder import org.openehealth.ipf.modules.hl7.validation.builder.ValidationContextBuilder class SampleRulesBuilder extends ValidationContextBuilder { RuleBuilder forContext(ValidationContext context) { new RuleBuilder(context) .forVersion('2.2') .message('ADT', 'A01').abstractSyntax( 'MSH', 'EVN', 'PID', [ { 'NK1' } ], 'PV1', [ { INSURANCE( 'IN1', [ 'IN2' ] , [ 'IN3' ] )}] ) } } The sequence and cardinality of groups and segments is defined in a syntax that is very closely related to the HL7 Abstract Message Syntax. The message must contain the segments MSH, EVN, PID and PV1; it may contain zero or more NK1 segments and it may contain a repeatable INSURANCE group. In a next step we configure our SampleRulesBuilder in a Spring application context along with a ValidationContextFactoryBean. The ValidationContextFactoryBean auto-detects any beans of type ValidationContextBuilder and adds their validation rules to a ValidationContext that is used in route definitions. The validate().ghl7() DSL extension, which we have already seen, can be configured with a custom validation profile using the profile() DSL extension. The validation context is looked up from the Spring application context via bean(ValidationContext.class). import ca.uhn.hl7v2.validation.ValidationContext import org.apache.camel.spring.SpringRouteBuilder class SampleRouteBuilder extends SpringRouteBuilder { void configure() { from(...) .unmarshal().ghl7() .validate().ghl7().profile(bean(ValidationContext.class)) ... .to(...) } } Code mapping HL7 message processing often involves mapping between code systems i.e. from one set of codes into a corresponding different set of codes. For example, HL7 version 2 and HL7 version 3 use different code systems for most coded values such as message type, gender, clinical encounter type, marital status codes, address and telecommunication use codes, just to mention a few. IPF defines a mapping service that provides the mapping logic. This may be a simple map but it can also be a facade to a remote mapping or terminology service. IPF's default mapping service implementation, the BidiMappingService, supports bidirectional mappings and reads custom mapping definitions from mapping files. An instance of the BidiMappingService can be created with the following bean definition. The mapping service references a mapping file example.map on the classpath. If there is more than one mapping file, a list can be provided via the mappingScripts property. Here is the content of example.map: mappings = { encounterType(['2.16.840.1.113883.12.4','2.16.840.1.113883.5.4'], E : 'EMER', I : 'IMP', O : 'AMB' ) } The example mapping file is a Groovy script and contains a single mapping with three entries for encounter type codes. Also defined are the ISO Object Identifiers (OIDs) for the key and value code systems. The mapping service can now be accessed either directly or via methods on java.lang.String. The String.map() method maps the codes on the left side to the right side. The identifier for the mapping can either be passed as argument assert 'E'.map('encounterType') == 'EMER' assert 'X'.map('encounterType') == null assert 'X'.map('encounterType', 'DEFAULT') == 'DEFAULT' or as part of a method name. assert 'E'.mapEncounterType() == 'EMER' assert 'X'.mapEncounterType() == null assert 'X'.mapEncounterType('DEFAULT') == 'DEFAULT' A dynamic dispatch is used to select the mapping definition from the method name. The method names must therefore correspond to the registered mappings. Mapping in the reverse direction is equally possible. assert 'EMER'.mapReverse('encounterType') == 'E' assert 'EMER'.mapReverseEncounterType() == 'E' Code systems are often associated with a globally unique identifier, usually in form of an OID. The identifier of both sides of a mapping can be obtained as follows. assert 'encounterType'.keySystem() == '2.16.840.1.113883.12.4' assert 'encounterType'.valueSystem() == '2.16.840.1.113883.5.4' Code mapping methods may also be used in combination with the HL7 DSL. MessageAdapter message = ... assert message.PV1.patientClass.value == 'I' assert message.PV1.patientClass.map('encounterType') == 'IMP' assert message.PV1.patientClass.mapEncounterType() == 'IMP' Response messages HL7 messaging often requires the return of an HL7 response message to the sender. With IPF, positive (ACK) or negative (NAK) acknowledgments to messages can be generated. Acknowledgments are in the same HL7 version as the original message and are populated with arguments to the ack() method. MessageAdapter message = ... def ack = message.ack() def nak1 = message.nak('Reason for failure') def nak2 = message.nak(new HL7Exception('Reason for failure', 204)) Generating acknowledgments is, however, only one special case of generating a response to an original message. For responses other than acknowledgements, a response message prototype can be created via the respond(eventType, triggerEvent) method. The MSH and MSA segments of the response message are then populated as required by the HL7 specification. def rsp = msg.respond('RSP','K21') // generates a RSP_K21 message More features So far, the main focus has been on HL7 message processing. IPF has many more features and services to support the development and the operation of production-quality integration solutions. Some of them are briefly described in the following list. A detailed description would far exceed the scope of this paper. For a complete overview refer to the IPF reference documentation. Core features. Collection of domain-neutral message processors and DSL extensions usable for general-purpose message processing including support for Groovy XML processing and support for using closures with Camel DSL elements. OSGi support. Support for running IPF and its services inside an OSGi environment and for the development of OSGi-ready IPF applications. Flow management. A service for monitoring and managing message flows through IPF applications. The flow manager also supports a replay of messages for e.g. recovery from failures. Large message support. Allows for memory-efficient processing of large messages. Event infrastructure. An infrastructure for publishing and consuming system and application events. Can be used e.g. for separating logging, audit or statistics concerns from application-specific route definitions. Can also be used to integrate with complex event processing (CEP) engines. Outlook Upcoming IPF releases will provide support for CDA (Clinical Document Architecture) and IHE (Integrating the Healthcare Enterprise). CDA is an XML-based document markup standard that specifies the structure and semantics of a clinical document for the purpose of exchange. IHE is an initiative of healthcare professionals and industry to improve the way computer systems in the healthcare sector share information. The goal of IPF is to make it as easy as possible for developers to implement the CDA, IHE (and other clinical) standards in their applications. Here are a few examples of what to expect in the next IPF release. CDA support IPF's CDA support will focus on building CDA documents using a domain-specific language. This DSL supports the creation of structurally correct CDA documents by enforcing CDA-relevant schema definitions but without dealing with low-level XML details. The DSL is implemented by a custom Groovy builder, the CDABuilder. In the following code snippet we use the CDABuilder to create a CDA document using CDA-specific terms such as clinicalDocument, code, title, recordTarget and so on. For printing the created document to stdout we use the left-shift (<<) operator. // Create a CDA builder CDABuilder builder = new CDABuilder() // Create a new CDA document def document = builder.build { clinicalDocument { id(root:'2.16.840.1.113883.19.4', extension:'c266') code( code:'11488-4', codeSystem:'2.16.840.1.113883.6.1', codeSystemName:'LOINC', displayName:'Consultation note' ) title('Good Health Clinic Consultation Note') recordTarget { patientRole { id { extension="12345" root="2.16.840.1.113883.19.5" } patient { name { given('John') family('Doe') } birthTime('19320924') } //... } //... } //... } //... } // Write document XML to stdout System.out << document CDA support will also include support for selected CDA profiles from IHE, HL7/ASTM and HITSP specifications, for example XPHR and CCD. Profile-specific CDA DSL extensions will enforce the constraints imposed by the profile specifications. As an example, predefined CDA sections could then be added without knowing their templateID-OIDs or their exact nested XML structure. DSL support for parsing, validating, transforming and rendering CDA documents will complete the feature set. IHE support IPF's IHE support is a framework for creating actor interfaces as specified in IHE profiles. Most likely, support for the XDS profile will be the first to come. XDS stands for Cross-Enterprise Document Sharing and deals with registration and distribution of, and access to clinical documents across health enterprises. Central to this profile are the actors document registry and document repository. A number of document management systems could in principle act as registry and/or repository in the XDS profile. Most of these, however, do not support the XDS actor interface specifications right out of the box. This is were IPF comes in. It helps developers to build IHE actor interfaces for existing information systems. Consider this example: from('ihe:xds.b:iti-41?port=8080') .process { exchange -> def document = exchange.in.body // do further document processing here ... } // communicate with your document management system .to('http://...') // notify about availability of new document .to('ihe:nav:iti-25:[email protected]') This route starts a server to receive documents according to the ITI-41 transaction of the XDS.b IHE profile. ITI-41 is the Provide and Register Document Set transaction in XDS.b. XDS.b requires documents to be transported via SOAP and ebXML standards. To free developers from having to deal with low-level SOAP/ebXML handling, these communication details are hidden inside an ihe component (eventually there may be more than one). Subsequent processors can access the transported document without having to deal with ITI-41 details. After processing, the incoming document is uploaded to a document management system and, finally, [email protected] is notified about the availability of a new document. Notifications are sent according to the IHE NAV profile where NAV stands for Notification of Document Availability. This example is of course oversimplified (for instance it does not address responses, etc) but it still gives an idea of the abstraction level on which IHE interfaces can be implemented for existing systems. Conclusion Apache Camel is a good answer to many of today's integration problems. It provides a DSL for implementing Enterprise Integration Patterns and offers developers a simple and efficient way to deal with the diversity of applications and transports in distributed systems. IPF brings the power of Apache Camel to the healtcare domain and makes healthcare IT standards usable by means of a domain-specific language that closely resembles the language of domain experts. The DSL extension mechanism permits the evolution of even more specialized healthcare DSLs. IPF's support for DSL modularization makes these DSLs and their implementing components reusable in different integration scenarios. Author Martin Krasser is a software architect and engineer working for InterComponentWare AG. He focuses on distributed systems, application integration and application security. Martin is the founder and project lead of the open source IPF project.

JSF has a simple Inversion-of-Control (IoC) container called the JSF Managed Bean Facility (MBF). Although it has a verbose XML syntax, and is not as robust as the Spring BeanFactory, PicoContainer, or the JBoss Microcontainer, the MBF does have the basics of an IoC container, and offers features like dependency injection. When a POJO is managed by the JSF MBF, it is typically referred to as a managed-bean. But if you're going to create a maintainable JSF webapp/portlet, it is necessary to distinguish between different kinds of managed-beans. This practice will also preserve the clean separation of concerns that JSF provides by implementing the Model-View-Controller (MVC) design pattern: Managed-Bean Type Nickname Typical Scope Model Managed-Bean model-bean session Description: This type of managed-bean participates in the "Model" concern of the MVC design pattern. When you see the word "model" -- think DATA. A JSF model-bean should be a POJO that follows the JavaBean design pattern with getters/setters encapsulating properties. The most common use case for a model bean is to be a database entity, or to simply represent a set of rows from the result set of a database query. Backing Managed-Bean backing-bean request Description: This type of managed-bean participates in the "View" concern of the MVC design pattern. The purpose of a backing-bean is to support UI logic, and has a 1::1 relationship with a JSF view, or a JSF form in a Facelet composition. Although it typically has JavaBean-style properties with associated getters/setters, these are properties of the View -- not of the underlying application data model. JSF backing-beans may also have JSF actionListener and valueChangeListener methods. Controller Managed-Bean controller-bean request Description: This type of managed-bean participates in the "Controller" concern of the MVC design pattern. The purpose of a controller bean is to execute some kind of business logic and return a navigation outcome to the JSF navigation-handler. JSF controller-beans typically have JSF action methods (and not actionListener methods). Support Managed-Bean support-bean session / application Description: This type of bean "supports" one or more views in the "View" concern of the MVC design pattern. The typical use case is supplying an ArrayList to JSF h:selectOneMenu drop-down lists that appear in more than one JSF view. If the data in the dropdown lists is particular to the user, then the bean would be kept in session scope. However, if the data applies to all users (such as a dropdown lists of provinces), then the bean would be kept in application scope, so that it can be cached for all users. Utility Managed-Bean utility-bean application Description: This type of bean provides some type of "utility" function to one or more JSF views. A good example of this might be a FileUpload bean that can be reused in multiple web applications. Now... One of the main benefits in making fine distinctions like this is loose coupling. What's that you ask? Well let's first take a look at an example of tight coupling, where MVC concerns can be smashed/confused into a single managed-bean: public class ModelAndBackingAndControllerBean { private String fullName; // model-bean property private boolean privacyRendered; // backing-bean property // model-bean getter public String getFullName() { return fullName; } // model-bean setter public void setFullName(String fullName) { this.fullName = fullName; } // backing-bean getter public boolean isPrivacyRendered() { return privacyRendered; } // backing-bean setter public void setPrivacyRendered(boolean privacyRendered) { this.privacyRendered = privacyRendered; } // backing-bean actionListener for UI support logic public void togglePrivacySection(ActionEvent actionEvent) { privacyRendered = !privacyRendered; } // controller-bean business logic public String submit() { System.out.println("fullName=" + fullName); return "success"; } } The problem here is that the bean would have to be kept in session scope because of the model-bean property. Additionally, what if we wanted to do some unit testing with mock model data? Can't do it. So in order to fix these problems, and to promote loose coupling, we would have three separate Java classes: public class ModelBean { private String fullName; public void setFullName(String fullName) { this.fullName = fullName; } public String getFullName() { return fullName; } } public class BackingBean { private boolean privacyRendered; public void setPrivacyRendered(boolean privacyRendered) { this.privacyRendered = privacyRendered; } public boolean isPrivacyRendered() { return privacyRendered; } public void togglePrivacySection(ActionEvent actionEvent) { privacyRendered = !privacyRendered; } } public class ControllerBean { private ModelBean modelBean; public ModelBean getModelBean() { return modelBean; } public void setModelBean(ModelBean modelBean) { // Dependency injected from the JSF managed-bean facility this.modelBean = modelBean; } public String submit() { System.out.println("fullName=" + getModelBean().getFullName()); return "success"; } } Now that the beans are found in different classes, they can all be kept in their appropriate scopes. The model-bean can be kept in session scope, and the backing-bean and controller-bean can be kept in request scope, thus saving memory resources on the server. Finally, we can use the dependency injection features of the JSF MBF in order to inject the model-bean into the controller-bean. This can be seen in the following WEB-INF/faces-config.xml example, where the #{modelBean} Expression Language (EL) binding is used: modelBean myproject.ModelBean session backingBean myproject.BackingBean request controllerBean myproject.ControllerBean request modelBean #{modelBean} From http://blog.icefaces.org/

Apache Camel is a powerful open source integration platform based on Enterprise Integration Patterns with Bean Integration. This Refcard provides you with eleven of the most essential patterns that anyone working with integration must know. This Refcard is targeted for software developers and enterprise architects, but anyone in the integration space can benefit as well. Download Now! About the Author: Claus Ibsen is a passionate open-source enthusiast who specializes in the integration space. As an engineer in the FuseSource Open Source Division he works full time on Apache Camel, FUSE Mediation Router (Apache Camel Enterprise) and related projects. Claus is very active in the Apache Camel and Fuse communities, writing blogs, twittering, assisting on the forums, irc channels and is driving the Apache Camel roadmap.

Take any integration project and you have multiple applications talking over multiple transports on multiple platforms. As you can imagine, in large enterprise applications this can get complex very fast. Much of the complexity stems from two issues: 1. dealing with the specifics of applications and transports, and 2. coming up with good solutions to integration problems. Making your applications speak transports and APIs is relatively easy on its own. I'm sure everyone knows how to send JMS messages to their broker of choice; though it still requires in depth knowledge of the JMS specification, which many developers may not have. On top of that, what happens when you want to route that JMS message to another application? You then have to take care of mapping the JMS message to the application plus handle any new concepts related to the application. Add a dozen other applications into the mix and you've got quite a headache on your hands. Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high level design of how applications interact. Fortunately, most solutions to enterprise integration problems have been formalized already. Gregor Hohpe and Bobby Woolfe's book, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, boils down years of experience from enterprise architects into a set of sixty five Enterprise Integration Patterns (EIPs). This is great but we still have to hand code all parts of these patterns; these are not packaged solutions, only recommendations. Apache Camel was created with the intention of addressing these two issues. In this article I'll show you how it actually does this. What is Camel? Apache Camel is an open source Java framework that focuses on making integration easier and more accessible to developers. It does this by providing: • concrete implementations of all the widely used EIPs • connectivity to a great variety of transports and APIs • easy to use Domain Specific Language (DSL) to wire EIPs and transports together Figure 1 shows how these three items actually map to Camel concepts. To give you a good understanding of how Camel is organized, we will discuss Components, Endpoints, Processors, and the Domain Specific Language (DSL). There is of course a lot more going on here under the hood but we'll leave that for another discussion. Figure 1: High level view of Camel's architecture. Components are the extension point in Camel to add connectivity to other systems. The core of Camel is very small to keep dependencies low, promote embeddability, etc. and as a result contains only 12 essential components. There are over 60 components outside the core. To expose these systems to the rest of Camel, Components provide an Endpoint interface. By using URIs, you can send or receive messages on Endpoints in a uniform way. For instance, to receive messages from a JMS queue aQueue and send them to a file system directory "c:/tmp", you could use URIs like "jms:aQueue" and "file:c:\tmp". Processors are used to manipulate and mediate messages in between Endpoints. All of the EIPs are defined as Processors or sets of Processors. As of writing, Camel supports 41 patterns from the EIP book, 6 other integration patterns, and many other useful Processors. To wire Processors and Endpoints together, Camel defines a Java DSL. The term DSL is used a bit loosely here as it usually implies the involvement of a compiler or interpreter that can process keywords specific to a particular domain. In Camel, DSL means a fluent Java API that contains methods named like terms from the EIP book. Its best explained with an example from("jms:aQueue") .filter().xpath("/person[@name='Jon']") .to("file:c:\tmp"); Here we define a routing rule in a single Java statement that will consume messages from the "jms:aQueue" Endpoint, send them through a Message Filter Processor, which will then send on messages passing the XPath condition to the "file:c:\tmp" endpoint. Messages failing the condition will be dropped. You can also configure your routes in a XML-based Spring configuration file. This configuration file is a lot more verbose and less auto complete friendly than the Java DSL; many prefer it though because of its direct access to Spring concepts and no requirement for compilation after changes. Here is what the earlier example would look like in Spring: /person[@name='Jon'] These are the concepts that Camel was built upon. Since then many other interesting features have been added. Details of these are left up to the reader to investigate. To get you started, some of these include: • Pluggable data formats and type converters for easy message transformation between Artix Data Services, CSV, EDI, Flatpack, HL7, JAXB, JSON, XmlBeans, XStream, Zip, Camel-bindy, etc. • Pluggable languages to create expressions or predicates for use in the DSL. Some of these languages include: EL, JXPath, Mvel, OGNL, BeanShell, JavaScript, Groovy, Python, PHP, Ruby, SQL, XPath, XQuery, etc. • Support for the integration of beans and POJOs in various places in Camel. • Excellent support for testing distributed and asynchronous systems using a messaging approach • and much more... Example A motorcycle parts business, Rider Auto Parts, supplies parts to motorcycle manufacturers. Over the years they've changed the way they receive orders several times. Initially, orders were placed by uploading CSV files to an FTP server. The message format was later changed to XML. Currently they provide a web site to submit orders as XML messages over HTTP. All of these messages are converted to an internal POJO format before processing. Rider Auto Parts states to any new customers to use the web interface to place orders. However, because of existing agreements with customers, they must keep all the old message formats and interfaces up and running. Solution using EIPs Rider Auto Parts faces a pretty common problem; over years of operation businesses acquire software baggage in the form of transports/data formats that are popular at the time. Using patterns from the EIP book we can envision the solution as something like Figure 2. Figure 2: This shows the solution to Rider Auto Parts integration problem using notation from the Enterprise Integration Patterns book. So we have several patterns in use here. 1. There are two Message Endpoints; one for FTP connectivity and another for HTTP. 2. Messages from these endpoints are fed into the incomingOrderQueue Message Channel 3. The messages are consumed from the incomingOrderQueue and routed by a Content-Based Router to one of two Message Translators. As the EIP name implies, the routing destination depends on the content of the message. In this case we need to route based on whether the content is a CSV or XML file. 4. Both Message Translators convert the message content into a POJO, which is fed into the orderQueue Message Channel. The whole section that uses a Content-Based Router and several Message Translators is referred to as a Normalizer. This composite pattern has a unique graphic to depict it but was left out here in favor of its sub-patterns to make things clearer. Implementation using Camel As mentioned before, Camel has a small core set of components included by default. The rest of the components exist as separate modules. In applications that require many types of connectivity it is useful to figure out what Camel modules to include. Listing 1 shows the dependencies using Apache Maven for the Camel implementation of the Rider Auto Parts example. Of course, you don't need to use Apache Maven for dependencies - it is just the easiest way to rapidly add new dependencies to your applications. The list of dependencies includes support for core Camel, ActiveMQ, JAXB marshaling, CSV marshaling, and HTTP. To make the example easier to try out, I've opted to use the File endpoint instead of the FTP. If we were using the FTP endpoint we would need to add a dependency on the camel-ftp module as well. Listing 1: Maven dependencies for the Camel implementation org.apache.camel camel-core ${camel-version} org.apache.camel camel-spring ${camel-version} org.apache.activemq activemq-camel ${activemq-version} org.apache.camel camel-jaxb ${camel-version} org.apache.camel camel-csv ${camel-version} org.apache.camel camel-jetty ${camel-version} org.apache.activemq activemq-core ${activemq-version} org.apache.xbean xbean-spring ${xbean-spring-version} While it is perfectly legitimate to use Camel as a standalone Java application, it is often useful to embed it in a container. In this case, we will be loading Camel from Spring. The Spring beans XML file is shown in Listing 2. First we start an embedded Apache ActiveMQ broker and connect Camel to it. We also load up some helper beans that we will reference from the DSL. Finally, the camelContext element tells Camel to look for routes in the org.fusesource.camel package. Routes are Java classes that extend the RouteBuilder class in Camel. Listing 2: Spring XML file that configures an embedded ActiveMQ broker, several beans used in the Camel route, and initializes the Camel Context to search for routes in the org.fusesource.camel package. org.fusesource.camel The real meat of the Camel implementation lies in the OrderRouter class (shown in Listing 3). This class extends RouteBuilder, so it will be automatically picked up and loaded by Camel's runtime. Looking back at Figure 2, we need to receive orders from an FTP (substituted with File) and HTTP endpoint, formatted as shown in Listing 4. In the DSL we can specify these incoming endpoints with two from elements. Both from elements are connected to a to("jms:incomingOrderQueue") element, which will send the messages to a queue on the ActiveMQ broker. Listing 3: Route definitions for the example. The routing rules are specified using a fluent API, referred to as Camel's DSL. public class OrderRouter extends RouteBuilder { @Override public void configure() throws Exception { JaxbDataFormat jaxb = new JaxbDataFormat("org.fusesource.camel"); // Receive orders from two endpoints from("file:src/data?noop=true").to("jms:incomingOrderQueue"); from("jetty:http://localhost:8888/placeorder") .inOnly().to("jms:incomingOrderQueue") .transform().constant("OK"); // Do the normalization from("jms:incomingOrderQueue") .convertBodyTo(String.class) .choice() .when().method("orderHelper", "isXml") .unmarshal(jaxb) .to("jms:orderQueue") .when().method("orderHelper", "isCsv") .unmarshal().csv() .to("bean:normalizer") .to("jms:orderQueue"); } } In the case of the HTTP endpoint, there are a couple of extra things to mention. First off the HTTP client will be expecting a response from the application so we have to handle that. In Camel, we have full control over what the client gets back from the HTTP endpoint. Each response is determined by the last method in our current route definition (each Java statement is a route definition). In our case we use the transform method to set the response to the constant string "OK". Since we handle the response ourselves, we don’t want any response to come from the JMS incomingOrderQueue. To send to this queue in a fire-and-forget fashion we add the inOnly modifier. It is important to note at this point that when writing Camel DSL in a modern Java IDE, selection of the next processing step is easy because of auto complete. The auto complete feature basically gives you a list of processors (i.e. EIPs) to choose from at any point in your route. Since fluent APIs chain methods together, the only method you need to remember is the from; all other methods are shown via auto complete. Listing 4: Incoming message formats; XML on top, CSV below. "name", "amount" "brake pad", "2" The next section of DSL in Listing 3 specifies the Normalizer, complete with Content-Based Router and two Message Translators. First we specify that we want to consume messages from the incomingOrderQueue on the ActiveMQ broker. The content based routing of the messages is done with the choice and when methods. In our case, we want to send CSV messages to one Message Translator and XML messages to another. To check what type of message we have we will be using a simple Java bean shown in Listing 5. Of course, this is demonstration code only; for production cases you would want to add more thorough checking of content types. Listing 5: Java bean that contains helper methods to be used in the DSL. public class OrderHelper { public boolean isCsv(String body) { return !body.contains("> body) { List orderHeaders = body.get(0); List orderValues = body.get(1); return new Order(orderValues.get(0), Integer.parseInt(orderValues.get(1))); } } At this point, successfully normalized messages are sent to the orderQueue for processing by some other application at the Rider Auto Parts business. Conclusion In this article I've shown two common problems that an integration developer may face: dealing with the specifics of applications and transports, and coming up with good solutions to integration problems. The Apache Camel project provides a nice answer to both of these problems. As the example has shown, solving integration problems with Camel is straight forward and results in relatively concise code. In my opinion it is the closest thing to integration nirvana that we have today. Links Apache Camel – http://camel.apache.org FUSE Mediation Router (based on Apache Camel) – http://fusesource.com/products/enterprise-camel Enterprise Integration Patterns – http://www.enterpriseintegrationpatterns.com Jon’s Blog – http://janstey.blogspot.com Camel in Action book - http://www.manning.com/ibsen Article source code - http://repo.fusesource.com/maven2/org/fusesource/examples/rider-auto-example/1.0/rider-auto-example-1.0.zip Author Jonathan Anstey is a senior engineer working for Progress Software Corporation specializing in the enterprise integration space. Jon focuses mostly on Apache Camel and its Progress endorsed likeness, FUSE Mediation Router. He also works on the Apache ActiveMQ and Apache ServiceMix projects

The concept of a software factory describes a practical work-product approach to governing an efficient service factory - a software engineering-based approach to defining, developing, testing, deploying, and operating functional services and automated business processes. All services follow a similar lifecycle of analysis, followed by design, development, deployment, and ongoing management. Because the service creation process is repetitive, a production engineering approach to automating software development can be used. The Production Engineering method required a significant effort up front, creating a specialized production or assembly line that can then mass-produce the product efficiently and in quantity In effect, we are building a Services Factory: much of the purpose of SOA governance is to define how that factory can operate most effectively. In the following excerpt from the book "SOA Governance: Achieving and Sustaining Business and IT Agility" [REF-1] we will take a look specifically at service development lifecycle control points. This article authored by Clive Gee and Robert Laird and published in SOA Magazine on Feb 23, 2009 Introduction While most organizations have some form of a system development lifecycle (SDLC), the nature of creating shared services is best guided by an SDLC with sufficient governance control points to ensure quality of service. This article discusses and explains the key concepts of a governance control point, as applied specifically to the service development lifecycle. Service Development Lifecycle Control Points Most organizations already have some type of system development lifecycle (SDLC) and a methodology that is used to perform development, although we often see in practice a lack of enforcement of that approach across different business units, and even if a set of best practices, standards, policies, and patterns has been defined, they are not always enforced. Effectively enforcing best practices and a consistent SDLC provides a reasonable entry point for real governance, while not being a huge stretch from what is already being performed via the SDLC. At the same time, if the governance maturity level of the organization can be increased to the degree that it is able to govern the SDLC, the organization is then in a much better position to proceed to the next phase of the SOA governance cycle and create program and organization governance. The danger here for even initial attempts at SOA governance is that often some key individuals view the imposition of any process or governance as being something that might apply to other people but not to them personally. For them, it's an over-engineered, useless exercise that just gets in the way of meeting their own deadlines. So, many governance processes are simply bypassed, or they're followed in a less than an enthusiastic manner. The main reason for this is that governance is imposed from the outside and the execution is onerous. What would happen if governance were mostly automated, easy, and added value to the development process and actually helped with project deadlines? Would the skeptics be more willing to take the medicine if it genuinely eased their pain? To adequately govern the SDLC, there is a need to establish measurements, policy, standards, and control mechanisms to enable people to carry out their governance roles and responsibilities as efficiently as possible, without introducing overly bureaucratic procedures. Governance of the SDLC may be characterized by the sorts of decisions that need to be made at certain "control points" within the process of services development. A control point is a decision checkpoint that provides an opportunity to measure adherence to the established processes, whether you are on track to meet the targets and goals you have established, and then decide whether the way the processes are executed or managed needs adjusting. Knowing what decisions involved in the process are critical, when to make them, and understanding what measurements are needed to monitor those processes are all essential aspects of governance. Certain activities within a process may be associated with a control point. At the end of each identified activity, there is a control point at which the governance function decides whether the program is ready to move to the next activity. Each of these milestones is a control point. At its essence, the governance of the SDLC provides a way to identify control points and to define the governance rules. At each control point, it is necessary to identify the following: • The roles for who does what at the control point • The policies to be applied at the control point • Measurements at each control point that should be applied and collected for later governance vitality actions • The proof of compliance records to be created and archived A control point will be created where there is a demonstrated advantage weighing the standardization and efficiency provided versus the time, effort, and possible project delay. The control point enables SOA governance the opportunity to ascertain progress, to communicate this progress, to forecast efforts for subsequent phases of the SDLC based on scope and issues found, to review and report compliance, and to facilitate the injection of expertise and qualified review of the artifacts, process, or decisions made by the development team. Control points don't have to consist of huge formal meetings. Services and most automated business processes are smaller entities than projects, and there are many more of them. Therefore, the existing governance approach has to be streamlined or it might grind to a halt. We've found in practice that effective control point reviews can be made during regular - typically weekly - sessions of a subset of the SOA enablement. A real productivity aid in performing these control point reviews is the use of previously completed checklists, signed off by one or more senior professionals as certification that one or more tasks has been completed successfully, and that the service, process, or other work product is fit for purpose and ready in all respects for the next task in the development process. These checklists should be viewed as contracts between different experts in the service development process. The most important part of the checklist is the signature block to show who exercised approval authority; people tend to be careful about the quality of anything that carries their personal reputation with it. Another productivity aid is the use of automated tooling. As much of the governance control point as possible should be automated. This aids in better near real-time feedback to the developers and provides an easy method to recheck work that has been updated. In addition, human beings are busy and will tend to apply governance in an inconsistent manner. Machines are consistent but not usually as flexible as needed. The combination of the two provides an optimal governance mix. Let's look at the control points needed to govern a generic development lifecycle, at least at a high level. Figure 1 represents a "governance dashboard" monitoring a typical SDLC with an eye toward the key concepts and the points where they must be addressed by SOA governance. Figure 1: Software Development Lifecycle Governance Dashboard As mentioned previously, we need a streamlined process that can handle the large number of services and automated processes that we need to implement to have real impact on business agility and flexibility. However, that streamlined process must not sacrifice the quality of governance just because of the need for extra speed. That would be an unacceptable trade-off. Some organizations deal with highly regulated processes that have mission-critical or life-critical products and need to apply highly formal, auditable governance to manage the risks involved. Other organizations have processes with lower associated risks that can be more lightly regulated. We have found in practice that the same governance process can handle both these extremes perfectly well. If there is a need for stricter governance, it can be met with tighter policies at the control points together with more stringent policy enforcement and compliance measurement. If less-strict governance is more appropriate, the same process can be used with less restrictive policies, fewer audits, and lower levels of checklist signoff required. Even within a single organization, different processes may require different styles of governance. Some processes, such as service certification, require stricter governance than other processes, such as solution architecture. Different organizational cultures require different levels of autonomy in decision making. Good governance requires good judgment. First, let's update our Figure 1 with the location of these control points so that you have a visual representation in mind as you read their descriptions. Figure 2 shows where the control points occur in that development cycle. Figure 2: Software Development Lifecycle with SOA Control Points Here are descriptions of these control points. Business Requirements and Service Identification Control Point For an SOA approach, there is an emphasis on creating services that provide agility and reuse for the business. This first business requirements and service identification control point consists of a high-level review to determine that services are being identified in accordance with services selection and prioritization policies. This first business requirements and service identification control point should address the following types of questions: • Business goals. What are the business goals that the business seeks to attain and how do we measure the benefits or progress toward the business goals via key performance indicators (KPIs)? • Do the requirements as we currently understand them clearly support those goals, and do they align with an existing "business heat map"? • Are those requirements sufficiently understood and agreed to? Are they presented in a form such as use cases, business process models, sequence diagrams, or class diagrams that are consistent with the SOA development approach? • How do we provide traceability of the requirements so that we can ascertain that those requirements have been met during the development process? Have those requirements been entered into an enterprise-wide requirements and business rules catalog? Is there any conflict with existing entries in that catalog? • Which of those requirements could be translated into good candidate services, either because they represent functionality that may be needed by multiple consumers or that might be needed for process automation? Which requirements could be better supported by deploying applications, automated processes, or manual processes? • Where we have identified candidate services, have we identified potential consumers, and determined whether any of them have specific requirements that should be considered? • Given finite IT resources, what development priority should we assign? Is ownership of any new candidate IT asset defined, and is outline funding available for its development? Solution Architecture Control Point Different IT developers and groups, if left to make all design decisions on their own, would invariably use completely different platforms, coding languages, tools, styles, methods, and techniques. This variation adds cost and complexity to the ability of the business to make future changes, and makes future maintenance very hard and costly. Further, it reduces the reliability, stability, and interoperability of the organization's IT assets. We have seen this problem at many organizations that we have visited. Simply put, the purpose of the solution architecture control point is to prevent that expensive multiplicity of approaches from occurring ever again. Essentially, any proposed IT artifact that makes it past this control point is part of the IT build plan. For the area of solution architecture, the governance should control for a series of criteria the following: • Do the proposed standards, policies, and reference architectures - the solution architecture - identify the standards, policies, and design patterns to be followed in the service implementation? This will include reference architectures, platform standards for hardware, and software-usage standards. • Have any reusable assets been identified and assessed for suitability? Has the service sourcing policy been followed? • Have the nonfunctional requirements been identified and assessed? This includes the number of transactions per time unit, a busy hour analysis, the service performance required, presentation access to the service functionality, data managed by the service, space required for the installation of the service, and any dependency and configuration requirements. Governance must validate that all these are considered and addressed. • Governance must validate that all security policies are being considered and addressed. • Governance must validate that all legal and regulatory policies are considered and addressed. • By this stage in the development of IT assets such as services or automated processes, the technical IT staff involved should have a pretty good idea about the complexity of the tasks involved, and the probable level of resources required to complete development. Should development of the asset be confirmed, the scope reduced, or the asset abandoned? Service Specification Control Point A service specification should be created for each service whose development has been approved. Best practices for service design must integrate both an IT and business perspective for the design of the interface and the responsibilities of each service. Because the service specification is, in effect, the organization's face to business partners, customers, and other stakeholders, the service externals - those details of a service that are to be made public - become an important part of the overall business design. The design should take into account the requirements of all potential service consumers (within reason), and be created at a granularity that maximizes business value. For the area of service identification and specification, the governance should control for a series of criteria the following: • Does the service identified make sense, is at the right granularity, and is not duplicating an existing service? • Does the service specification follow all SOA standards and policies? • Does the service specification follow the messaging model? If not, should an exception be granted? Service Design Control Point After the service solution architecture has been turned over to the design team, a number of design elaboration decisions must be made. Collectively, these form the service internals - a set of design models, notes, and advice that will guide the service developers as they create and test the service code. For the area of service design, the governance should control for a series of criteria the following: • Has a service architect confirmed that the design should be able to meet the nonfunctional and functional requirements for this service? • Have the service designer and data architect agreed that the service can be made to conform to the signature (that is, inputs and outputs) described in the service externals? • If a service is wrapping an existing or planned application, are the necessary interfaces to that application well defined and stable (that is, won't change if a new version of that application is installed)? • Have the monitoring metrics (for example, usage, quality of service [QoS] levels) been established? • In the case of automated processes, have the monitoring requirements been defined and planned? • In the case of long-running automated processes, have all the necessary actions to handle recovery from process errors or technical failures been addressed? • Is the overall quality and level of completeness of the service specification package good enough that the service developers or process developers can complete development without further input? Service Build Control Point After the service design has been turned over to the service build team, a number of implementation decisions need to be made before development of the code or executable model. In the interests of consistency and quality, we strongly recommend the use of code walkthrough reviews, where peers (that is, other service developers or process developers) review the work in progress and offer constructive criticism. The service build control point is effectively the last of these code walkthroughs, and should be performed with slightly more formality than the others. Questions that should be addressed include the following: • Was the asset coded in accordance with the design? • Does the code follow the accepted coding standards? • Have all the associated artifacts (for example, load libraries, metadata files, resources) been defined to create a transportable build? Have the versions of each of those artifacts been checked to see that there are no version conflicts with services already in production? Service Test Control Point Service testing is different from testing complete IT solutions or applications. Because services and automated processes do not have their own user interface, it is not possible to perform user acceptance testing directly on services or automated processes. Code frameworks or specialized tools are needed to exhaustively test services and automated processes thoroughly to avoid uncovering problems during later formal user acceptance testing when the rest of the IT solution that uses those services or processes has been completed. SOA governance must ascertain that the services test is being performed in a manner conducive to a services approach, and that exhaustive functional and nonfunctional tests have been passed before releasing any SOA asset to production. The service test team must create and use the right service test environment with tools and data to affect a comprehensive test. This should include the following: • Using the optimum set of service test tools and frameworks. • The use of an automated build and test environment that can enable fast changes of the tested software and regression testing. This environment must closely resemble the production environment. • A load/stress test tool to test nonfunctional requirements, specification, creation, and loading of realistic but artificial test data. • A test management reporting tool to keep management apprised of the testing status. • Trace the test case to the original user requirements. Service Certification and Deployment Control Point The objectives of the deployment are to migrate the services to the production environment while minimizing client downtime and impact on the business. This process is subject to many errors if performed manually. It is vital that the correct version of the services be deployed and that any deployment binding with other services and applications be performed quickly and correctly. Areas for governance to validate include the following: • The use of a tool that automates the deployment and back-out process. • Final certification checks have been made against the services to verify compliance with all policies and standards and being able to demonstrate that what was tested matches not only the requirements but what was delivered, and that no corrective changes made during testing have invalidated other test results. • IT operations have completed acceptance testing and have formally accepted the asset, signifying their confidence in being able to operate it within the terms of the QoS specified for it. • The service registrar and business service champion have reviewed the service description in the service registry and approved it. Certification of a service or automated process is a formal "passing out" ceremony, and granting of certification should signify that the SOA enablement team is happy for their reputation to be associated with the performance of the new asset. Service Vitality Control Point Service vitality takes place periodically as part of SOA governance to check up on and update the governance processes, procedures, policies, and standards in reaction to the results of the real world. This involves examining any and all lessons learned in any of the SOA planning, program control, development, or operations activities. It also includes such things as comments and feedback from all stakeholders and an examination of any common patterns (for example, common exemption requests or common reasons for failure to pass one or more control points) that need remedial action. Metrics in the efforts required for each stage of the development process can show trends that indicate improvements or declines in their vitality. A formal service vitality control point review should be conducted every three to six months to determine whether the SOA transition remains on track, and whether the level and style of governance is optimal. Individual service or automated processes should be reviewed every 6 to 12 months. Usage data of all versions of each service can determine any "stale" versions that can be deprecated or deleted, and whether the deployment options taken and decisions on who should own and who should access each service are optimal. Conclusion We have focused in this extract on SOA Governance service development control points as a method to create a software engineering capability of a service factory. The factory is a production line for services. All services pass through a common, repeatable series of development, deployment and management steps. Quality and governance is built-in throughout the entire process. References [REF-1] "SOA Governance: Achieving and Sustaining Business and IT Agility" by William A. Brown, Robert G. Laird, Clive Gee, Tilak Mitra (IBM Press, ISBN 0137147465, Copyright 2009 by International Business Machines Corporation. All rights reserved.) This article was originally published in The SOA Magazine (www.soamag.com), a publication officially associated with "The Prentice Hall Service-Oriented Computing Series from Thomas Erl" (www.soabooks.com). Copyright ©SOA Systems Inc. (www.soasystems.com)

As its name implies Amazon Simple Queue Service (Amazon SQS) is a simplified version of a Messaging system. The great thing about it is its reliability and high scalability features. Amazon SQS makes it easy to build an automated workflow, working in close conjunction with the Amazon Elastic Compute Cloud (Amazon EC2) and the other AWS infrastructure web services. You can learn more about Amazon SQS here. The Problem My colleagues (Tom Purcell and Rod Biresch) and myself have been working on a demo application for Chariot Solutions' Emerging Technologies for the Enterprise conference and the need for a transport component to simplify the access of Amazon SQS arose. We have decided to use FUSE ESB(ServiceMix) 4 as the services integration platform for the demo application, and knowing that Apache Camel collaborates very well with ServiceMix, we decided to use it for the routing and transformation needs. Currently, neither ServiceMix, nor Camel provide a component to access Amazon SQS. Besides this limitation the remaining capabilities provided by these products still made it a great choice for the demo application. To learn more about the demo application, please visit Tom Purcell's blog. I also recommend attending to the Tom Purcell's presentation titled "Bus In the Cloulds" where he'll be speaking about Could Computing use cases and how Open Source Integration technologies can help resolve issues encountered while integrating with business partners. Another important part in production applications is management and monitoring. To learn more about how to monitor this application, please visit Rod Biresch's blog. A Solution Apache Camel makes it relatively easy to compose your own components, so the next logical step was to try to create one. Having seen Dave Kavanagh's Typica API in action and how easy it makes the communication with SQS, we decided to try and come up with such Camel component using this API as the back-bone. The Camel Amazon SQS Component The basic idea is to create a component that could be used in the following fashion: The above example shows a Camel route that sends data found in files placed in the data directory to a SQS queue named "dataQueue". In the to URI "sqs:sqsProducer?accessId=....", the protocol "sqs" will tell Camel what component to configure and attributes will be mapped the properties of either the Endpoint, or the Producer class. The documentation on writing Camel Components can be found here. Let's dive into the design of the Camel SQS component. The main steps for writing a Camel Component are: Write a POJO which implements the Component interface. The simplest approach is just to derive from DefaultComponent. In our case, the class SQSComponent was created. This class extends DefaultComponent. To support auto-discovery of your component add a file to META-INF/services/org/apache/camel/component/sqs where 'sqs' is the URI scheme for your component and any related endpoints created on the fly. The sqs file will contain the class attribute with the name of Class that implements the Component: class=com.chariotsolutions.soalab.camel.component.sqs.SQSComponent The Camel SQS Component is composed of the following artifacts: The SQSComponent Class This class serves as factory of Endpoints. When the component is being configured by Camel, the method of the createEndpoint is called. This method creates and configures the Endpoint (SQSEndpoint). The SQSEndpoint Class This class serves as a factory for Producers, Consumers, and Exchanges. After the SQSComponent.createEndpoint is execute, an instance of the class SQSEndoint is created. This class contains the properties used by the SQSConsumer class. These properties are: private String accessId; private String secretKey; private String queueName; This class extends ScheduledPollEndpoint and is used to create instances of the SQSProducer and SQSConsumer. The use of the ScheduledPollEndpoint is with the intention to use polling facilities it provides to query the SQS Queue at a specified time intervals. The SQSConsumer Class This class extends the ScheduledPollConsumer class to indicate that it will be excueted in poll intervals. The poll intervals are configured in the Component URI using the consumer.initialDelay and consumer.delay attributes like so: The poll() will be executed based on the configuration specified above.This method uses the Typica API to interact with the Amazon SQS Web services. If a message was found in the SQS Queue, it will e forward to the handleMessage. This method will set the message in a wrapper class called SQSObject, place it in the SQSExchange and send it flow of the route using the synchronous Processor by default. The attribute asyncProcess=true would cause it to use the AsyncProcess. The SQSProducer Class This class extends the DefaultProducer class. It processes the message coming from the Camel flow and sends it to the SQS queue, then it sets the assigned message id to the SQSObject, which is later added to the SQSExchange in case other components need it down the road. The SQSExchange Class This The base message exchange interface providing access to the request, response and fault instances. The SQSObject Class This is convenient POJO that wraps the payload and properties coming from the SQS Queue. Taking the Camel SQS Component For A Ride To see the Camel SQS Component in action, I created a small sample application. The sample application can be found in here. Here are the steps to build and run the application: If you haven't already done so, create an Amazon SQS account using the "Sign Up" button in this page. Extract the sample code to a directory of your choice. For example, /tmp/sqs-sample. You should see 2 directories in there, camel-sqs (camel sqs component) and camel-sqs-route (sample camel route). Navigate to the camel-sqs directory and build the code using: mvn clean install Navigate to the camel-sqs-route directory. Edit the file src/main/resources/aws.properties. Specify aws.accessId and aws.secretKey for your Amazon SQS account. Edit the file src/main/resources/META-INF/spring/camel-context.xml. Find the file endpoints and modify them to point to the directories of your choice. The File-In endpoint is where you would supply the data files to be sent to the SQS queue. The File-Out endpoint is the directory where the files containing the contents of the SQS queue will be placed on. Build the code using: mvn clean install Run the code using: mvn camel:run You should be able to place your sample text files directory pointed by the File-In endpoint. After a few seconds the output files will be placed in directory pointed by File-Out. Conclusion Amazon SQS combined with ServiceMix and Camel are a great platform for building your IaaS (Infrastructure as a Service) projects. Hopefully future versions of Apache Camel will include such component. The Camel CAMEL-1432 JIRA issue has been created for this reason.Happy Cloud Computing! Recommended Sites Tom Purcell's Blog Rod Biresch's Blog Amazon SQS FUSE ESB (ServiceMix) 4 Apache Camel Chariot Solutions

Recently I needed to make my Maven2 web project communicate with an old style RPC encoded web service. We run on GlassFish which comes with JAX-RPC RI built-in, so I was hoping to find a way to use it without having to bundle another runtime such as Axis into the project. Specifically I was looking for a way to make my build-script generate client stubs from a WSDL file in the project tree. I want the client stubs to use standardized JAX-RPC APIs which are serviced by the implementation provided by GlassFish. I found that the JAX-WS RI has a Maven2 plugin but not the JAX-RPC RI. I did a lot of googling, and posted messages in a few mailing lists. Surprisingly there is almost nobody using JAX-RPC on Maven2 projects because there are barely any examples online and nobody could answer my question. I'm surprised because there are a lot of legacy systems out there that developers need to integrate with, so I can't be the only one still needing JAX-RPC support. Some people suggested that I update the web service to modern RPC literal to make client-side development easier with JAX-WS, but that is not an option. Others suggested that I use Maven's antrun plugin to call out to an ant build script that does JAX-RPC work. Finally I found an all Maven solution that works for me: org.apache.axis axis 1.4 compile org.apache.axis axis-jaxrpc 1.4 compile commons-discovery commons-discovery 0.4 compile wsdl4j wsdl4j 1.6.2 compile org.codehaus.mojo axistools-maven-plugin 1.3 wsdl2java com.mycompany.service.client src/main/resources/META-INF/wsdl target/generated-sources/wsdl2java private MyServiceSEI getMyServicePort() throws ServiceException { MyServiceLocator locator = new MyServiceLocator(); MyServiceSEI port; Stub stub; port = locator.getMyServiceSEIPort(); // OPTIONALLY configure URL and HTTP BASIC authentication stub = (Stub) port; stub._setProperty(Stub.ENDPOINT_ADDRESS_PROPERTY, "http://hostname/ContextRoot/ServicePort"); stub._setProperty(Stub.USERNAME_PROPERTY, "user"); stub._setProperty(Stub.PASSWORD_PROPERTY, "secret"); return port; } It works, but I think the client stubs are dependent on the Axis 1 runtime instead of using the GlassFish provided JAX-RPC runtime through the standardized JAX-RPC APIs. I've wasted enough time on this so I moved on. If you know a better way to do this, please let me know. No RESTful services rants please! I think SOAP and REST both have their uses depending on what you are doing. From my perspective, JAX-WS SOAP development is effortless and requires significantly less code than REST because I don't have to write the building/parsing code. After examining raw SOAP messages without extra WS-* headers added in, I decided that overhead is not a valid argument against SOAP. When you need more advanced features they are available to you without having to re-invent the wheel. I think most Java developers' negative impressions on SOAP comes from the JAX-RPC and Axis days, and because they haven't tried JAX-WS yet. From http://www.ryandelaplante.com/

The industry is recognizing the fact that performance testing & engineering should be part of the project execution road map starting from the requirements gathering phase. At many times during project executions, performance engineering related activities are executed based on customer need or slow response time of application after development phase gets completed. Glassbox can be leveraged (by developers/testers/business users) during and after the development cycle to monitor the response times of requests with-out being aware of underlying application structure and code details. Analysis generated by Glassbox gives direct pointers on where is the bottleneck which causes slow response time for that particular request/page/URL. About Glassbox Glassbox is an open source web application which aid in performance monitoring and troubleshooting of multiple web applications deployed in container. Troubleshooting It contains the built-in knowledge repository of common problems which are used to pinpoint the issues and suggestions on causes as Java code executes. Performance Monitoring It monitors the requests as Java code executes and provides details about response times. Glassbox web client (AJAX GUI) provides nice summary dashboard view which contains various attributes like (server-name, application name, operation/request-URL, average time, no. of executions, status (slow / OK) and analysis details). By default, an operation that takes more than 1 sec execution time is marked as SLOW status. Such SLA can be modified using Glassbox properties file. Analysis part describes the problem precisely and very clearly in plain English words, rather than displaying large code/exception trace. This definitely increases developer productivity by reducing developer’s time spent in log files and using IDE debuggers. Internals The two main components of Glassbox are Monitor and Agent. Monitor uses Aspect-Oriented Programming (AOP) to monitor the JVM activity. Agent diagnoses and presents the monitoring results and uses knowledge repository to cross reference the problem with suggestions/solutions. Glassbox agent supports viewing of the analysis results using JMX (eg. Java 5 JConsole) Consoles. Glassbox extensively uses the AOP approach internally to monitor the Java code. This gives the benefit of not making any changes to source code or build-process and hence can work with any legacy web application/jar file as well. Technologies Glassbox should work on any application server that supports Servlet 2.3 or later. The servers where Glassbox is tested and installation process is automated are Apache Tomcat, weblogic, websphere, Resin, Oracle OC4J, websphere, Resin, Jetty & GlassFish. Overhead Having Glassbox application running on same container would generate a performance overhead. Typically this would affect the response time and memory overhead. Hence it is recommended to start the Glassbox application only when it’s required for performance monitoring. Licensing Glassbox is an open source project, it is free to download and run. Glassbox uses the GNU Lesser General Public License to distribute software and documentation. Demo Application Development & Deployment to Tomcat To test the capabilities of Glassbox, a sample application is developed which has a TestServlet class. This servlet calls DelayGenerator class’s generateDelay() method. This method calls Thread class’s sleep() method which suspends the execution of servlet. A counter is being initialized in DelayGenerator class which determines the time interval till which servlet is needed to be suspended. TestServlet.java /** * File: TestServlet.java * @author Viral Thakkar */ package com.infosys.star.glassbox; import java.io.IOException; import java.io.PrintWriter; import javax.servlet.ServletException; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; public class TestServlet extends HttpServlet { protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { DelayGenerator delayObj = new DelayGenerator(); int delay = delayObj.generateDelay(); response.setContentType("text/html"); PrintWriter out = response.getWriter(); out.println(""); out.println(" Hello World from Test Servlet : "+delay+" milliseconds "); out.println(""); out.flush(); } } DelayGenerator.java /** * File: DelayGenerator.java * @author Viral Thakkar */ package com.infosys.star.glassbox; public class DelayGenerator { private static int counter = 1; public int generateDelay() { try { Thread.sleep(counter * 100); counter++; } catch (InterruptedException e) { e.printStackTrace(); } return counter*100; } } Glassbox Installation & Integration to Apache Tomcat 6.0 Glassbox installation is very straightforward for non-clustered environment for the server where it’s automated. Simply drop the glassbox.war file at the appropriate folder inside server folder or perform the server specific steps/configuration to deploy the war file. Browse to server url with context root as glassbox – http://<>:/glassbox. Follow the instructions available on this page. According to specific server, this page would suggest the configuration changes for a server. Please refer to Glassbox User Guide document for details on how to install Glassbox for clustered application server environment. For Apache Tomcat 6.0- Add following command line arguments to Tomcat’s Java options: -Dglassbox.install.dir=C:\Tomcat6.0\lib\glassbox -Djava.rmi.server.useCodebaseOnly=true -javaagent:C:\Tomcat6.0\lib\aspectjweaver.jar Monitoring & Technical Analysis Glassbox web client (URL- http://<>:<>/glassbox ) shows the summary and detailed view of all the requests/operations that container/JVM has executed. Summary Section View Different attributes (columns) which gets displayed in this table are as below - Attribute / Column Name Comments Status This indicates whether operation/request is performing OK, SLOW or FAILING Analysis For SLOW/FAILING status, this value provides the small summary of the cause of the problem. Operation This is name of the operation/request of an application Server Name of the server where monitoring is being done. In a clustered environment, this allows to distinguish operations on different servers. Executions This value indicates how many times this operation has run since the application server was started or Glassbox’s statistics were last reset. Click the request in above summary table to view its detailed analysis in below detailed section. Detailed Section View The details area provides information relating to operations selected in the summary table. Different sub-sections which gets displayed in this view are as below - Sub-section Name Comments Executive Summary High level summary view of the selected operation gets displayed in a table format. This is neat view to senior stake holders who are not interested in technical details. Technical Summary This section contains more technical details in paragraph and table representation formats to provide insight into root cause of the problem if any, like which operation, query is slow and statistics of same. Details like stack trace, thread lock name are provided to find and fix the problem. “Common solutions” sub section shows pointers to resolve the identified problem/s. “Glassbox has ruled out other potential problems” sub section saves time to know what problems have already been ruled out. Executive Summary View Technical Summary -> Technical Details Views Above two snapshots are parts of the Technical Details section and provide minute details at code level with line number so as to pinpoint where the problem is. Here cause is identified at Class com.infosys.star.glassbox.DelayGenerator inside Method generateDelay at line number 12 where Thread.sleep is invoked. Perform Load Testing Using JMeter and Monitor Using Glassbox Apache JMeter is used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. It can be used to make a graphical analysis of performance or to test server/script/object behavior under heavy concurrent load. Using JMeter, create a test plan that simulates 10 users requesting for 1 page 5 times. i.e. 10 x 1 x 5 = 50 HTTP requests. First step is to add a Thread Group element. The Thread Group tells JMeter the number of users to simulate, how often the users should send requests, and the how many requests they should send. Next step is to add HTTP Request element to added Thread Group. In parallel, have the Glassbox up and running to monitor response time statistics of the load generated by JMeter application. Below is the Executive summary view of above test in Glassbox web UI interface. Section “Monitoring & Technical Analysis” contains the details to understand the Glassbox generated analysis. Conclusion Glassbox is not the replacement for performance testing tool like load runner. Glassbox aids in the project to various stakeholders in finding, conveying and fixing the performance problems at all phases starting build (development) to post deployment. Glassbox application to be started/installed only during monitoring time so as to avoid the performance overhead for other applications due to CPU & memory footprint occupied by Glassbox application on the container. During load testing of the application, Glassbox turns out to be good option to figure out the root causes inside an application code. References Glassbox web site - http://www.glassbox.com/glassbox/Home.html Glassbox User Guide - http://nchc.dl.sourceforge.net/sourceforge/glassbox/Glassboxv2.0UserGuide.pdf Apache JMeter - http://jakarta.apache.org/jmeter/ Download & Support Glassbox Download Link - http://www.glassbox.com/glassbox/Downloads.html Glassbox forum Link - http://sourceforge.net/forum/forum.php?forum_id=575670 About Author Viral Thakkar is a Technical Architect with the Banking and Capital Markets vertical at Infosys. He has 9.5 years of technology consulting experience mainly on Java/JEE technologies and frameworks with large banks and financial institutions across the globe. He has been part of many small and large-scale initiatives related to application development, architecture creation and strategy definition. From http://viralpatel.net/blogs

Earlier today I read a post by Michael Feathers Called "10 Papers Every Developer Should Read (At Least Twice). I knew some of the articles mentioned there and learnt about few interesting ones.I liked it so much, I thought I'd compile a similar list for software architects - based on stuff I read over the years. The Byzantine Generals Problem (1982) by Leslie Lamport, Robert Shostak and Marshall Pease - The problem with distributed consensus Go To statements considered harmfull (1968) - by Edsger W. Dijkstra - Didn't you always want to know why ? :) A Note on Distributed Computing (1994) - by Samuel C. Kendall, Jim Waldo, Ann Wollrath and Geoff Wyant - Also on Michael's list but it is one of the foundation papers on distributed computing Big Ball of Mud (1999) - Brian Foote and Joseph Yoder - patterns or anti-patterns? No Silver Bullet Essence and Accidents of Software Engineering (1987) - Frederick P. Brooks - On the limitations of Technology and Technological innovations. The Open Closed Principle (1996) - Robert C. Martin (Uncle Bob) - The first in a series of articles on Object Oriented Principles (you remember the debate on SOLID...) IEEE1471-2000 A recommended practice for architectural description of software intensive systems (2000) various- It is a standard and not a paper but it is the best foundation for describing a software architecture I know. Harvest, Yield, and Scalable Tolerant Systems (1999) Armando Fox, Eric A. Brewer - That's where the CAP theorem was first defined An Introduction to Software Architecture (1993) - David Garlan and Mary Shaw - one of the foundation articles of software architecture field (although based on earlier work by the two) Who Needs an Architect? (2003) Martin Fowler - Do we or don't we? I could come up with quite a few more articles not to mention books that aren't in this list. However these are definitely some of the most influential papers I read

This article is going to show you how to build a RichFaces application with Spring.

The deployment descriptor, web.xml is the most important Java EE configuration piece of Java EE Web applications.

Until a few days ago, I've always needed to work with the rather cumbersome Office Bean and UNO Runtime when integrating OpenOffice into a Java application. I also had to configure a whole bunch of things to force OpenOffice to play nicely with the Java integration. Two days ago, however, I found out about ODF Toolkit. It seems to be a relatively new project, independent since last year some time, though I could be wrong. What's especially interesting is the ODFDOM: ''ODFDOM is an OpenDocument (ODF) framework. It's purpose is to provide an easy common way to create, access and manipulate ODF files, without requiring detailed knowledge of the ODF specification. It is designed to provide the ODF developer community an easy lightwork programming API, portable to any object-oriented language.'' Here's a snippet of it in action: public static void main(String[] args) { try { OdfDocument odfDoc = OdfDocument.loadDocument(new File("/home/geertjan/test.ods")); OdfFileDom odfContent = odfDoc.getContentDom(); XPath xpath = odfDoc.getXPath(); DTMNodeList nodeList = (DTMNodeList) xpath.evaluate("//table:table-row/table:table-cell[1]", odfContent, XPathConstants.NODESET); for (int i = 0; i < nodeList.getLength(); i++) { Node cell = nodeList.item(i); if (!cell.getTextContent().isEmpty()) { System.out.println(cell.getTextContent()); } } } catch (Exception ex) { //Handle... } } Let's assume that the 'test.ods' file above has this content: From the above, the code listing would print the following: Cuthbert Algernon Wilbert And, as a second example, here's me reading the first paragraph of an OpenOffice Text document: public static void main(String[] args) { try { OdfDocument odfDoc = OdfDocument.loadDocument(new File("/home/geertjan/chapter2.odt")); OdfFileDom odfContent = odfDoc.getContentDom(); XPath xpath = odfDoc.getXPath(); OdfParagraphElement para = (OdfParagraphElement) xpath.evaluate("//text:p[1]", odfContent, XPathConstants.NODE); System.out.println(para.getFirstChild().getNodeValue()); } catch (Exception ex) { //Handle... } } On my classpath I have "odfdom.jar" and "xerces-2.8.0.jar". I don't necessarily have OpenOffice installed, which means I can very easily process a whole bunch of spreadsheets (or other OpenOffice output) without (a) installing OpenOffice and (b) faster than I would otherwise do, since OpenOffice doesn't need to be started up, via the Office Bean or otherwise. In fact, Aljoscha Rittner from Sepix, who told me about this project and who is using it in his commercial applications, reports that his processing has sped up to a fraction of the original, also because he doesn't need to handle the situation where OpenOffice would crash randomly in the middle of long running processes, such as during the night when there's no human interaction for restarting it.

A reader of my blog sent me an email asking the following question: “I have an XSD that I am required to use to export my company's data. How can I use that XSD and return data to them in a web method? I should be able to return a data set with the information formatted the way the XSD defines but I have no idea how to do that. Any ideas would save me a ton of time and grief!” Turns out this is a really good question, and I think one a lot of developers struggle with primarily because the majority of developers are scared of XSD schemas. Hopefully I can change that. An XML Schema is really simple and it is your friend. What is an XSD or Xml Schema Definition? An XML Schema describes the structure of an XML document. That’s it, see, it isn’t that hard after all! As a developer think of an XSD the same way you would when creating business rules to validate objects before storing them into a database. Some items in objects cannot be null, some can. Some need to have their data formatted (email address for example) and others need to not exceed certain lengths, etc. When dealing with an XML document, we need to apply the same type of rules and this is where the XSD comes in. Once we have an XSD to describe an XML document we can guarantee any XML data we receive conforms to these rules. Ok, enough intro, let’s get coding. The Schema In the case of the reader’s question, an XSD schema already existed. To save time, I’ve taken then general idea of the schema I was sent and slimmed it down for simplicity sakes. The schema represents an XML document that will contain job listings. Here is a view of the schema from the schema explorer in Visual Studio as well as the schema itself (again slimmed down). Once we have a schema defined like this we can generate code to be used for a service. We have a tool at our disposal that is available within the Visual Studio Command Prompt called XSD.exe. This executable does a lot of things but one thing it can do is generate code from an XML Schema Definition. When generating from the XSD file I was sent there was a problem with it. Here is the walk through of that story so you can follow along. When I first tried this on the reader’s original XSD I got an error: Warning: cannot generate classes because no top-level elements with complex type were found. What does this mean? Well, look at the screen shot of the XML Schema Explorer above. Do you see the <> green icons? Well those represent an XML Element. An element is a fancy name for an XML tag like: Bramar Let me roll this up and take another screen shot so you can start to see the problem (i have to hit my magic number of 12 screen shots anyway :) ). See the problem? There isn’t an element in this schema (no green <> thingy). What this means is we have a schema which just lists complex types of elements and enumerations. There isn’t any “xml” in this schema. The fix is simple though. Add an element to the schema. Now we have a green thingy icon which means this XSD contains at least one element. Basically think of the JobListing element as the root element. Now that we have an element we can generate a C# class file from this XSD: The code generated is about 350 lines so I’m not going to include it in the article. There are some nice things happening for us when the C# classes are generated using xsd.exe. For starters the enumerations in the XSD are turned into C# enumerations. The classes are marked serializable and they have the appropriate XML attributes on the properties to serialize these objects into XML. This is a good thing since it means we didn’t have to create it. Code generation is your friend. Now that we have C# objects we can serialize these to XML easily. The Service The first thing we need to do is create the WCF service. For this example I chose to create a WCF Service Application in Visual Studio 2008. After the project is initialized, the first thing we need to do is define the contract for the service. In other words, what is coming in and going back out. Since we have generated our code using the XSD utility we are going to use the XmlSerializer with our service. The reason is this will make sure our XML formatted the way we intended. While we are at it, I’ve made the service a RESTful type of service. Here is the contract. [ServiceContract] public interface IJobService { [OperationContract] [XmlSerializerFormat] [System.ServiceModel.Web.WebGet(UriTemplate="/jobs/{type}", RequestFormat=WebMessageFormat.Xml)] List GetJobListings(string type); } The contract above has one method GetJobListings(). This method has two additional attributes. One, the attribute to mark the method to serialize using the XmlSerializer and two the attribute to turn the method into a RESTful service. In other words our service can be accessed like this: http://localhost/service.svc/jobs/full-time Now that the contract is setup, we just need a class to implement this contract on. Here’s a quick and dirty implementation. public class Service1 : IJobService { #region IJobService Members public List GetJobListings(string type) { return GetJobs(); } private static List GetJobs() { var jobs = new List(); jobs.Add(new Job { JobType= JobType.parttime, Category = JobCategory.Banking, Benefits= "You are on your own.", Description="I did something" }); jobs.Add(new Job { JobType= JobType.fulltime, Category = JobCategory.Banking, Benefits= "You get something." }); jobs.Add(new Job { JobType= JobType.contractor, Category = JobCategory.Banking, Benefits= "Times are tuff, deal with it." }); jobs.Add(new Job { JobType= JobType.fulltime, Category = JobCategory.Banking, Benefits= "How does $700 billion sound?" }); return jobs; } #endregion } Now all we have left is to get this working is to configure WCF to support our RESTful implementation. In order to do this we are going to use the webHttpBinding. Here is the WCF configuration to implement our RESTful service. That’s it, we are all setup. Now all we have to do is launch the service and browse to the URL we outlined earlier to get our results. This may seem like a lot of steps but it really isn’t. I encourage you to take this example and play with it. You can download the solution below. Download Solution WcfService.zip

Continuous Integration commonly known as CI is a process that consists of continuously compiling, testing, inspecting, and deploying source code. In any typical CI environment, this means running a new build every time code changes within a version control repository. Martin Fowler describes CI as: A software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. While CI is actually a process, the term Continuous Integration often is associated with three important tools in particular. As shown in the image the three pillars of CI are: 1. A version control repository like Subversion, or CVS. 2. A CI Server such as Hudson, or Cruise Control 3. An automated build process like Ant or Nant So, let’s look at each of these in detail: Version Control Repository: Version control repositories also known as SCM (source code management) play a crucial role in any software development environment. They also play a very important role for a successful CI process. The SCM is a central place for the team to store every needed artifact for the project. It is mandatory for the teams to put everything needed for a successful build into this repository. This includes the build scripts, property files, database scripts, all the libraries required to build the software and so on. The CI Server: For CI to function properly, we also need to have an automated process that monitors a version control repository and runs a build when any changes are detected. There are several CI servers available, both open source and commercial. Most of them are similar in their basic configuration and monitor a particular version control repository and run builds when any changes are detected. Some of the most commonly used open source CI servers are; Cruise Control, Continuum, and Hudson. Hudson is particularly interesting because of its ease of configuration and compelling plug-ins, which makes integration with test and static analysis tools much easier. Automated Build: The process of CI is about building software often, which is accomplished through the use of a build. A sturdy build strategy is by far the most important aspect of a successful CI process. In the absence of a solid build that does more than compile your code, CI withers. With automated builds, teams can reliably perform (in an automated fashion) otherwise manual tasks like compilation, testing, and even more interesting things like software inspection and deployment. Now that we have seen the important tools in our CI process, let’s see how a typical CI scenario looks like for a developer: CI server is configured to poll the version control repository continuously for changes. Developer commits code to the repository. CI server detects this change, and retrieves the latest code from the repository. This causes the CI server to invoke the build script with the given targets and options. If configured, CI Server will send out an e-mail to the specified recipients when a certain important event occurs. The CI server continues to poll for changes. Why is CI Important? This is one of the most frequently asked questions, and here are a few points to note about this powerful technique: Building software often greatly increases the likelihood that you will spot defects early, when they still are relatively manageable. Extends defect visibility. CI ensures that you have production ready software at every change. CI also ensures that you have reduced the risk of integration issues by building software at every change. CI server can also be configured to run continuous inspection which can assist the development team in finding potential bugs, bad programming practice, automatically check coding standards, and also provide valuable feedback on the quality of code being written. Over the past several months, I have assisted several companies in implementing CI. There was a little bit of resistance from the developers in the early stages when we implemented continuous feedback. But, never heard a single negative comment about this approach. If you already have a version control repository and automated builds, you are very close to the CI process. Download one of the open source CI servers, configure and setup a simple project. It should take less than an hour if you have automated build scripts. Start adding additional features like code inspections, generating reports, metrics, documentation and so on. Most important, send continuous feedback to your team. Give this process a try, you sure will be surprised to see how effective it is. And, as always share your thoughts, concerns or questions.

#check partition sizes df -h #check directory size du -s -h /var/log/ #check every directory and file sizes under a dir. du -s -h /var/log/* #check individual size size du -s -h /var/log/lastlog

When doing performance analyzes you often would want to see 95 percentile, 99 percentile and similar values. The "average" is the evil of performance optimization and often as helpful as "average patient temperature in the hospital". Lets set you have 10000 page views or queries and have average response time of 1 second. What does it mean ? Really nothing - may be one page view was 10000 seconds and the rest was in low milliseconds or may be you had every single page view taking 1 second, which are completely different. You also do not really care about average performance - the goal of good user experience is majority of users to have good experience and average is not a good fit here. Defining your response time goal in 95 or 99 percentile is much better. Say you say 99 percentile response time should be one second, this means only 1 percent of queries/page views are allowed to take more than that. For larger systems defining (increasing) response times for 99.9 or even 99.99 percentile numbers often make sense. It also often makes sense to define response time goals separately for different transactions - the AJAX widget response time requirements may be very different from the slow search page. So you have defined your response time in terms of 95/99 percentile and get your logs in the table, so how to get the data if MySQL only provides you the avg: mysql> SELECT count(*),avg(wtime) FROM performance_log_081128 WHERE page_type='search'; +----------+-----------------+ | count(*) | avg(wtime) +----------+-----------------+ | 106859 | 1.4469140766532 +----------+-----------------+ 1 row IN SET (2.08 sec) The average response time here is for example; the real data what we need is number of rows which matches for given query type. Dividing the count by 100 we get our 1% of values and dividing by 20 5% of values, now we can get the response time we concerned about simply by running following order-by queries: mysql> SELECT wtime FROM performance_log_081128 WHERE page_type='search' ORDER BY wtime DESC LIMIT 1068,1; +---------+ | wtime +---------+ | 10.1007 +---------+ 1 row IN SET (2.06 sec) mysql> SELECT wtime FROM performance_log_081128 WHERE page_type='search' ORDER BY wtime DESC LIMIT 5342,1; +---------+ | wtime +---------+ | 5.09297 +---------+ 1 row IN SET (2.06 sec) So for this system the 95 percentile is just over 5 sec (some 3 times more than the average) and 99% percentile is just a bit over 10 seconds (6 times more than average). The both numbers are horrible and system surely needs to be fixed. These numbers are to illustrate - the percentile numbers can be quite different from average numbers (it is not rare to see 99 percentile to be order of magnitude different from the average) and this is what you really need to focus on. Looking at the numbers from the business standpoint try to understand what these really are. In some cases I see rather bad percentile on the backend which are not really the problem for the business because there is a cache up front anyway. If 99% of requests are coming from the cache and you observe certain 99 percentile response time on the backend it is often 99.99 percentile response time which is a lot different - you often can afford 1/10000 requests to stall for few seconds, because things outside of your control (like packet loss at client side) would be responsible for larger amount of delays. Be careful though - the "random" delays, for example if system was busy and delayed servicing request is one thing, "systematic" delays, when response time is always bad in given conditions can be much worse problems. You do not want your best client to suffer for example, even if he is the only one.

Eclipse has many ways of doing things. Finding the way that suits you can be tricky. :-) The standard find dialog triggered by Crtl+F is, well, standard. But if you are a keyboard junkie then it kind of gets in the way. The other way of searching with in the current editor is Incremental Search. This is triggered by Crtl+J and has no GUI! Just press Ctrl+J and type the search text. Eclipse finds and highlights the next match as you type. You can use the Up or Down arrow key to jump to the next or previous match. I like this way... From http://www.vasanth.in/

Learn how to properly configure logs in JBoss.

This article will explain Business Process Management (BPM) in terms of 7 distinct use cases for JBoss jBPM. By giving more insight in those use cases, you'll get a better understanding of the different forms of BPM and workflow and when a BPM engine like jBPM makes sense in your project. We'll also highlight the specific jPDL process language features related to those use cases. The term BPM is highly overloaded and used for many different things resulting in a lot of confusion. These use cases give concrete descriptions for the different interpretations of the term BPM. The individual nature of these use cases is important. BPM software vendors often take a mix of different aspects and concerns into account when developing their products. That often results into BPM products that are suitable only for a specific purpose in a specific environment. This is in my opinion the reason why there are so many different BPM products which only serve a small niche market. It also explains why new products and standards in this space keep appearing, don't get enough momentum and then get pushed aside by yet another new product or standard. When evaluating a BPM products, it should be done with specific use cases in mind. What is JBoss jBPM JBoss jBPM is a flexible and powerful BPM engine. In essence, BPM systems allow for execution flows to be specified graphically. As an example, here is a process diagram for a business trip: [img_assist|nid=5583|title=|desc=|link=none|align=undefined|width=623|height=487] A key capability of BPM systems is that processes steps can be wait states. For example in the business trip process above, nodes 'manager evaluation' and 'ticket purchase' are human tasks. When the execution of the process arrives in those nodes, the system executing the process should wait till the assigned user completes the task. From a software technical point of view that capability is a big deal. As the alternative is a bunch of methods that are linked by HTTP requests, Message Driven Beans (MDB), database triggers, task forms, etc. Even when using the most applicable architectural components available in Java today, it is still very easy to end up in a bunch of unmaintainable hooks and eyes. Using an overall business process makes it a lot easier to see and maintain the overall execution flow, even from a software technical perspective. JBoss jBPM does exactly that and it differentiates itself from other BPM projects in the following topics: Easily embeddable into a Java project. Traditional BPM Systems typically require a separate server to be installed which makes it hard to integrate into the Java software development cycle. One of the deployments that JBoss jBPM supports is just adding the jBPM library to the classpath. The jBPM tables can be hosted in any database next to the application's tables. Using JBoss jBPM really fits with the normal way of developing Java software. Support for multiple process languages. The view on what BPM actually is has not yet been stabilized. There are currently many different interpretations of what BPM is, resulting in big fragmentation in the market. In fact, the body of this article tries to identify 6 distinct concepts that all are associated to BPM. Very flexible transaction management. If your application just uses a JDBC connection in a standard Java environment, then jBPM can use that very JDBC connection to perform its work. If your application uses hibernate in a standard environment, then jBPM can use the same hibernate session factory. If your application runs in an enterprise environment, then jBPM can bind to the surrounding JTA transaction. Readable jPDL process language. For developers it is very convenient if the process language is compact and readable. Not all developers want to keep using the graphical editor. Most hard core programmers start to hand code the processes after a while. Then jBPM's jPDL process language is the most readable, compact and complete. Standards BPEL and BPMN Todays most known standards in the space of BPM are BPEL and BPMN. Those two standards have a completely different background. That is a clear indication of the diverse type of problems that is currently associated to BPM. BPEL is defined with an Enterprise Service Bus (ESB) in mind. It's all based on WSDL, which binds naturally to web services. The result of deploying a BPEL process is that a new service is being published. A BPEL process can hence be seen as a scripting language for services on the bus. A more elaborate conceptual explanation of BPEL can be found in Process Component Models: The Next Generation In Workflow ? So BPEL is an executable process language, which implies that a it is human to computer communication, just like a programming language. On the other hand, the first target of BPMN is modeling process diagrams by non-technical business analysts. It defines the shapes, types and meaning of the boxes and arrows. This type of modeling should be seen in the context of 'BPM as a discipline' (see below). By definition, non technical business analyst do not think in terms of web services or other technology aspects. BPMN processes are primarily intended for communication between humans. So far, so good. But now comes the hard and confusing reality: BPEL has been presented as a solution for 'BPM as a discipline' and BPMN 2.0 plans to add concrete executable semantics. These days, most IT people already realized that BPEL is not a convenient solution for 'BPM as a discipline'. On the other hand adding concrete executable semantics to a business analyst modeling notation means that non-technical analysts will be implicitly writing a piece of executable software. Would you put software into production written by a non-technical person ? Use Case 1 : BPM as a discipline BPM as a discipline refers to the analysis, documentation and improvement of the procedures that describe how people and systems work together in an organization. Realize that most BPM is done in this way, not even resulting into any form of software or IT support. MS Visio is one the most used tools to document business processes. Of course, once the business process is documented, it might make sense to develop software support for it. Suppose that a business analyst modeled and documented a 'business trip' process, then one usage might be to find optimizations in that process. And another usage might be the development of software support for business trips. Please be aware that an automatic translation between pure analysis models and executable process models are not feasible in general. Initially BPM products tried to make this translation automagically transparent. In the context of BPM as a discipline, we believe that a process model from a non technical business analyst can never be translated into an executable process model by just adding technical details to it. The next attempt was to acknowledge existence of an analysis model and an executable process model and then try to keep them in sync. Especially because of the big differences between BPMN and BPEL this approach got a lot of attention. But then the problem of maintaining the links between analysis and executable process model appears. Who is going to spend the time to link the analysis blocks to the executable blocks and then keep that mapping up to date. This is illustrated in the following picture: [img_assist|nid=5584|title=|desc=|link=none|align=undefined|width=539|height=295] For mainstream software development resulting from 'BPM as a discipline', we believe that the original analysis diagram should serve as input with the rest of the software requirements. Developers should then construct an executable process that looks as close as possible to the original analysis diagram. After that, the analysis diagram is discarded and replaced by the executable process diagram. That way, the developer is in full control of the software technical aspects, while the analyst will still recognize the diagram. jBPM's jPDL language is designed to facilitate this approach. First of all, it's based on free graph modeling. Secondly, it has so called actions, which can be seen as event listeners. These event listeners are pieces of Java code that are called when process execution produces the event, but they are hidden from the process diagram. This allows the developer to add programming logic without changing the diagram structure. Also super-states are often used in the context of creating better communication between business analyst and developer. Use Case 2 : Combining template based and ad hoc task management Orchestrating human tasks can be found in most process engines in one form or another. But what is often overlooked is that template based task orchestration only suits for a limited number of scenarios. First the process must be relatively stable. And secondly, enough executions of this process have to happen so that the gain that can be achieved with software support is worth the development effort. A lot of work being done in organizations does not meet those two requirements. For example, organizing a one-time team building event, a restyling of the office space or cleaning up after the sprinkler system went off unexpectedly. For that type of work, people in charge will invent the process on the spot and it will only be executed once. Human Interaction Management (HIM) focuses on this type of work. A story in HIM reflects a task that person creates for which an ad-hoc process will be created. People can get involved in different roles. There are big benefits of tracing such work with a task management system. First, people that get involved with such a task after a while will get instant visibility into the history of the whole story. And secondly, an audit trail is logged automatically. In jBPM 4, the task management component will support this ad hoc human tasks. The combination will be awsome. Human tasks in processes can be specified at a course grained level. When such a process task is created for a person, the assignee can involve other people by creating subtasks and assigning people in different roles to these tasks. Only when the owner decides that the overall task is finished, then the process will continue. Use case 3 : Transactional asynchronous architectures Gregor Hohpe's Entperprise Integration Patterns describes building blocks for asynchronous architectures. These patterns are based on the notion of Message Oriented Middleware (MOM) aka message brokers. MOM's are good at asynchronous transactional communication between two systems and JMS is the Java standard to work with it. Of course point-to-point communication can be set up easily with existing Java infrastructure like Message Driven Beans. But in many cases many of these point-to-point communications are related. For example, to process one order, a message might come from a client into the order processing system. As part of that transaction, a notification might be sent to the warehouse and stock planning team. Eventually after many more steps, a message might be sent back to the client as a confirmation of the order. If all these transactional communications are build linearly, it will be very hard to manage the overall state of order processing. BPEL also focuses on asynchronous architectures, but then in a (web) services environment, rather then a Java environment. In BPEL only (web) services can be invoked and XML based technologies like XPath are integrated. So for more complex calculations, a piece of programming logic needs to be included. To include programming logic in BPEL, it needs t be wrapped and exposed as a service. That can be quite cumbersome in certain situations. In contrast, jPDL is embedded in a Java environment. Such a central dispatching functionality is really suited to be implemented by a process language like jPDL. An incoming message can start a new jPDL process execution or provide a trigger for an existing execution. jPDL allows to include programming logic straight into the process transaction. Cause jPDL is based on a Java architecture, that results into much more flexibility and convenience. Using the central dispatching approach creates immediately more insight into the overall state of a process. In terms of our order processing example, it will be very easy to keep track of the state of each order. Further more, process engines like jPDL collect the history information. This is an extra feature that you get for free when using a process engine in this way. From the history information it is very easy to collect valuable statistical information like the average time in each step of the process. The difference with the use case 'BPM as a discipline' is that in this case, the goal is software technical in nature. The process is looked at from an implementation perspective. There doesn't have to be a relation to a business level analysis process. Use case 4 : Service orchestration Service orchestration is actually a variant of the previous use case 'transactional asynchronous architectures'. Interactions between services on an Enterprise Service Bus (ESB) are usually asynchronous. This time, the communication is not done through sending messages over a MOM, but instead by calling services over an ESB. BPEL is a process language specifically designed for this use case. Apart from a construct to invoke WSDL services, it has a whole infrastructure for supporting conversations. Conversations are so called long running processes. The clue here is that a BPEL process specifies a number of service operations that are being published when a BPEL process is deployed. A BPEL process can then contain receive activities, which mean that the process execution will wait until a service operation invocation is received. In jBPM we have implemented the BPEL process language as on of the languages that we support on top of our Process Virtual Machine. Use Case 5 : Visual programming With visual programming, we will target developers that do not yet have the full skillset to develop in Java. They can graphically specify the activities and draw transitions to indicate the control flow. Instead of typing Java code, just put blocks like 'perform hql query', 'generate pdf', 'send email', 'read file', 'parse xml', 'send msg to esb'. Instead of writing Java, they will be composing software programs by linking activity boxes in the diagram and filling in the configuration details. This will, of course, not have the same kind of flexibility as Java programming itself. Visual programming as we describe it here will not replace programming as we know it today. Today's programmers can be considered the power users and that kind will always be needed. But visual programming can lower the treshold to build applications for developers that have no or limited Java knowledge. The Process Virtual Machine (PVM) is the foundation of jBPM. In the PVM infrastructure it is really easy to add a new activity type. Think of it as a kind of command pattern, where the commands are configurable. So it will be very easy for us to develop new activity types. This way, we can expose the bulk part of the Java programming functionalities as activity types in the jPDL process language. In jPDL, these visual programs will be executable transactionally or without persisting the state of the execution. Use Case 6 : Thread Control Language This is a specific aspect of visual programming that also can be used by experienced developers. Coding multi-threaded Java programs is not easy. In fact, it is at least tedious and mostly pretty hard to code. Starting new threads, passing data into them, joining and making them stop properly can be a challenge. We'll develop a Thread Control Language which lets you specify a multithreaded Java concurrency by drawing forks, joins and method activities. A method activity invokes a methods on your Java object. And the concurrency control is handled by the forks and join in the process. These processes will be executed without any persistence. Use Case 7 : Easy creation of DSLs General purpose process languages are different from domain specific process languages. For example, think of a process language to specify approvals in an Enterprise Content Management (ECM) system. This gives a scoped functionality and a fixed environment. In such cases, a specific process language could be developed for this purpose. One of the advantages of such dedicated languages is that they can be made simple enough so that non technical business users can actually create fully executable processes. Another example of a domain specific process language is SEAM's Pageflow. It allows developers of a JSF based web site to specify the pages and navigations between the pages graphically. There is even an easier way. Instead of creating a full process language for a specific purpose, it is also possible to leverage jPDL's capabilities and just add new node types to it. That is already possible in jBPM 3 and will be peanuts in jBPM 4, even adding graphical support in the process designer for these new node types will be made simple. Conclusion The typical understanding of BPM is that non technical business people create diagrams that then automagically get executed on a BPM system. The first use case 'BPM as a discipline' describes that point of view. The value lies in the fact that non technical business people can communicate with the developers around a diagram. That diagram facilitates the communication between business analysts and developers. But even in that use case, the traditional understanding needs fine tuning. An analysis diagram at some point has to be converted into an executable process. At that moment, the responsibility over the process is transferred from the analyst to the developer. Many vendors have created the tools and illusion that this can be done automagically. But in practice, this black box approach creates more problems then it solves. The description of the 7 individual use cases shows distinct aspects of BPM. Knowing the difference between service orchestration and 'BPM as a discipline' will be key for organizations to select the most appropriate technology. All of the use cases have at least a technical side. That is understandable as all the use cases here target some form of software automation at some point. And other use cases target solely technical aspects. That is why the embeddability of jBPM is so important. Monolithic BPM engines have a high treshold to be incorporated into a typical software development project. jBPM has a big focus on delivering solutions for these distinct use cases to developers in a way that is easy for them to consume and integrate into their own software development project. Tom Baeyens is the founder and lead of JBoss jBPM, the leading open source BPM system. Tom mission is to bring the power of BPM technology into the hands of the developers. He's a frequent speaker at international conferences and maintains a blog at http://processdevelopments.blogspot.com

You had that nice looking editors and views. You loved the button color and the geeky font. But your boss didn't. Solution? Put a preference so that the user can choose which ever he likes ;-) Ok. That may not be the real rationale, but for a valid reason, you wanted to have Color and Font preferences in the PreferencePage. The obvious answer for us (the plugin develoepers) would be to add ColorFieldEditor & FontFieldEditor in our existing PreferencePage. But its not obvious for the user to look into your plugins Preference Page for changing them. Eclipse already provides a PreferencePage for these customizations: Colors and Fonts page ( General->Appearance->Colors and Fonts ) Users might be looking into this place to change the colors & fonts. As you can see in the above image it categorizes the available preferences; gives a filter to search; provides space for description and preview; and a tool tip for the current value of the preference. This is much more than the *FieldEditor right? In this tip let us see how to add our preferences into that and access them. You need to extend the org.eclipse.ui.themes extension point for adding your content into this page. I would like to provide one category with one color preference and one font preference. The extension is: An example theme category Your description goes here Description for this font Guess all the elements are self explanatory - except for the value element. For colorDefinition the value element will be the COLOR_* constants defined in the SWT class. You can specify your custom colors thru comma separated RGB values like 125,245,96. For fontDefinition it would be fontname-style-height. Now our UI will look like: To get the values stored in the preferences, you need to use the IThemeManager: IThemeManager themeManager = PlatformUI.getWorkbench().getThemeManager(); ITheme currentTheme = themeManager.getCurrentTheme(); ColorRegistry colorRegistry = currentTheme.getColorRegistry(); Color color = colorRegistry.get("com.eclipse-tips.blog.preferences.colorDefinition"); FontRegistry fontRegistry = currentTheme.getFontRegistry(); Font font = fontRegistry.get("com.eclipse-tips.blog.preferences.fontDefinition"); Once you get the current theme, you can get the preferences stored by giving the appropriate ids defined in the plugin.xml What you have seen is just the tip of the iceberg. The themes extension point has much more functionalities (The Preview is one such example). I'll explain them later, but you can see the Extension Point description here. This is an extract from http://blog.eclipse-tips.com/2008/08/adding-color-and-font-preferences.html