Execute the slow integration tests separately from unit tests and show as much information about them as possible in Sonar.

The logging in Jersey, the reference JAX-RS implementation, is little sub-optimal. For example if it cannot find a method producing the expected MIME type then it will return “Unsupported mime type” to the client but won’t log anything (which mime type was requested, which mime types are actually available, …). Debugging it isn’t exactly easy either, so what to do? Well, I don’t know the ultimate solution but want to share few tips. Enable Tracing of Request Matching Jersey since version 1.1.5 supports request matching tracing, provided somehow detailed information about the matching process in the response headers. To enable it for the Jersey Test framework you’d do something like this in your test class: public class MyJerseyTest extends JerseyTest { public MyJerseyTest() { super(new WebAppDescriptor .Builder("my.package.with.jaxrs.resources") .contextPath("myCtxPath") .servletPath("/myServletPath") .initParam("com.sun.jersey.config.feature.Trace", "true") .build()); } // Your test methods here ...; you can get the trace headers via ClientResponse#getHeaders() } To enable it for the server itself (though might be not such a good idea to enable this in production), you would set it in web.xml: Jersey REST Service for value codes com.sun.jersey.spi.container.servlet.ServletContainer ... com.sun.jersey.config.feature.Trace true ... The headers, which you can obtain via e.g. curl -i or via ClientResponse#getHeaders(), might look like this: {X-Jersey-Trace-008=[mapped exception to response: javax.ws.rs.WebApplicationException@56f9659d -> 415 (Unsupported Media Type)], X-Jersey-Trace-002=[accept right hand path java.util.regex.Matcher[pattern=/myResource/([-0-9a-zA-Z_]+)(/.*)? region=0,17 lastmatch=/myResource/23/mySubresources]: "/myResource/23/mySubresources" -> "/myResource/23" : "/mySubresources"], X-Jersey-Trace-003=[accept resource: "myResource/23" -> @Path("/myResource/{item: [-0-9a-zA-Z_]+}") com.example.MyExampleResource@41babddb], X-Jersey-Trace-000=[accept root resource classes: "/myResource/23/mySubresources"], X-Jersey-Trace-001=[match path "/myResource/23/mySubresources" -> "/application\.wadl(/.*)?", "/myResource/([-0-9a-zA-Z_]+)(/.*)?", "/myResource(/.*)?", "/mySubresources/([-0-9a-zA-Z_]+)(/.*)?"], X-Jersey-Trace-006=[accept sub-resource methods: "myResource/23" : "/mySubresources", GET -> com.example.MyExampleResource@41babddb], X-Jersey-Trace-007=[accept termination (matching failure): "/mySubresources"], X-Jersey-Trace-004=[match path "/mySubresources" -> "/mySubresources(/)?", ""], X-Jersey-Trace-005=[accept right hand path java.util.regex.Matcher[pattern=/mySubresources(/)? region=0,6 lastmatch=/mySubresources]: "/mySubresources" -> "/mySubresources" : ""] , Transfer-Encoding=[chunked], Date=[Tue, 31 Jan 2012 14:48:26 GMT], server=[grizzly/2.1.2], Content-Type=[text/html; charset=iso-8859-1]} provided that you have the class com.example.MyExampleResource annotated with @Path("/myResource/{item: [-0-9a-zA-Z_]+}") and a method annotated with @GET @Path("mySubresources") (and the class field @PathParam("item") Long item). As you can see, there is still no info regarding accepted/supported MIME types. Get Detailed Logging Into a File Jersey uses Java logging, which is know for being difficult to configure. Here is a dirty trick to get detailed Jersey logs into a file: public class MyJerseyTest extends JerseyTest { @BeforeClass public static void setupJerseyLog() throws Exception { Handler fh = new FileHandler("/tmp/jersey_test.log"); Logger.getLogger("").addHandler(fh); Logger.getLogger("com.sun.jersey").setLevel(Level.FINEST); } // Your test methods here ... } Notice that even thoug the log level is ALL, the logs still might be quite useless to troubleshoot some problems (such s the unsupported MIME type). Configure Request/Response Logging Filters Jersey provides a LoggingFilter that can be used to log request/response entities and it can be installed both into the server and the client. The com.sun.jersey.api.container.filter.LoggingFilter may be installed into the server via init-params, the com.sun.jersey.api.client.filter.LoggingFilter may be installed into the client via client.addFilter. JerseyTest automatically installs the LoggingFilter into the client it uses if the system property “enableLogging” is set (to whatever). From http://theholyjava.wordpress.com/2012/01/31/troubleshooting-jersey-rest-server-and-client/

WADL (Web Application Description Language) is to REST what WSDL is to SOAP. The mere existence of this language causes a lot of controversy (see: Do we need WADL? and To WADL or not to WADL). I can think of few legitimate use cases for using WADL, but if you are here already, you are probably not seeking for yet another discussion. So let us move forward to the WADL itself. In principle WADL is similar to WSDL, but the structure of the language is much different. Whilst WSDL defines a flat list of messages and operations either consuming or producing some of them, WADL emphasizes the hierarchical nature of RESTful web services. In REST, the primary artifact is the resource. Each resource (noun) is represented as an URI. Every resource can define both CRUD operations (verbs, implemented as HTTP methods) and nested resources. The nested resource has a strong relationship with a parent resource, typically representing an ownership. A simple example would be http://example.com/api/books resource representing a list of books. You can (HTTP) GET this resource, meaning to retrieve the whole list. You can also GET the http://example.com/api/books/7 resource, fetching the details of 7th book inside books resource. Or you can even PUT new version or DELETE the resource altogether using the same URI. You are not limited to a single level of nesting: GETting http://example.com/api/books/7/reviews?page=2&size=10 will retrieve the second page (up to 10 items) of reviews of 7th book. Obviously you can also place other resources next to books, like http://example.com/api/readers The requirement arose to formally and precisely describe every available resource, method, request and response, just like WSDL guys were able to do. WADL is one of the options to describe “available URIs", although some believe that well-written REST service should be self-descriptive (see HATEOAS). Nevertheless here is a simple, empty WADL document: Nothing fancy here. Note that the tag defines base API address. All named resources, which we are just about to add, are relative to this address. Also you can define several tags to describe more than one APIs. So, let's add a simple resource: This defines resource under http://example.com/api/books with two methods possible: GET to retrieve the whole list and POST to create (add) new item. Depending on your requirements you might want to allow DELETE method as well (to delete all items), and it is the responsibility of WADL to document what is allowed. Remember our example at the beginning: /books/7? Obviously 7 is just an example and we won't declare every possible book id in WADL. Instead there is a handy placeholder syntax:There are two important aspects you should note: first, The {bookId} place-holder was used in place of nested resource. Secondly, to make it clear, we are documenting this place-holder using tag. We will see soon how it can be used in combination with methods. Just to make sure you are still with me, the document above describes GET /books and GET /books/some_id resources. The web service is getting complex, however it describes quite a lot of operations. First of all GET /books/42/reviews is a valid operation. But the interesting part is the nested tag. As you can see we can describe parameters of each method independently. In our case optional query parameters (as opposed to template parameters used previously for URI place-holders) were defined. This gives the client additional knowledge about acceptable page and size query parameters. This means that /books/7/reviews?page=2&size=10 is a valid resource identifier. And did I mention that every resource, method and parameter can have documentation attached as per the WADL specification? We will stop here and only mention about remaining pieces of WADL. First of all, as you have probably guessed so far, there is also a child tag possible for each . Both request and response can define exact grammar (e.g. in XML Schema) that either the request or the response must follow. The response can also document possible HTTP response codes. But since we will be using the knowledge you have gained so far in a code-first application, I intentionally left the definition. WADL is agile and it allows you to define as little (or as much) information as you need. So we know the basics of WADL, now we would like to use it, maybe as a consumer or as a producer in a Java-based application. Fortunately there is a wadl.xsd XML Schema description of the language itself, which we can use to generate JAXB-annotated POJOs to work with (using xjc tool in the JDK): $ wget http://www.w3.org/Submission/wadl/wadl.xsd $ xjc wadl.xsd And there it... hangs! The life of a software developer is full of challenges and non-trivial problems. And sometimes it is just an annoying network filter that makes suspicious packets (together with half hour of your life) disappear. It is not hard to spot the problem, once you recall that article written around 2008: W3C’s Excessive DTD Traffic: Accessing xml.xsd from the browser returns an HTML page instantly, but xjc tool waits forever. Downloading this file locally and correcting the schemaLocation attribute in wadl.xsd helped. It's always the little things... $ xjc wadl.xsd parsing a schema... compiling a schema... net/java/dev/wadl/_2009/_02/Application.java net/java/dev/wadl/_2009/_02/Doc.java net/java/dev/wadl/_2009/_02/Grammars.java net/java/dev/wadl/_2009/_02/HTTPMethods.java net/java/dev/wadl/_2009/_02/Include.java net/java/dev/wadl/_2009/_02/Link.java net/java/dev/wadl/_2009/_02/Method.java net/java/dev/wadl/_2009/_02/ObjectFactory.java net/java/dev/wadl/_2009/_02/Option.java net/java/dev/wadl/_2009/_02/Param.java net/java/dev/wadl/_2009/_02/ParamStyle.java net/java/dev/wadl/_2009/_02/Representation.java net/java/dev/wadl/_2009/_02/Request.java net/java/dev/wadl/_2009/_02/Resource.java net/java/dev/wadl/_2009/_02/ResourceType.java net/java/dev/wadl/_2009/_02/Resources.java net/java/dev/wadl/_2009/_02/Response.java net/java/dev/wadl/_2009/_02/package-info.java Since we'll be using these classes in a maven based project (and I hate committing generated classes to source repository), let's move xjc execution to maven lifecycle: org.codehaus.mojo jaxb2-maven-plugin 1.3 net.java.dev.jaxb2-commons jaxb-fluent-api 2.0.1 com.sun.xml jaxb-xjc xjc -Xfluent-api bindings.xjb net.java.dev.wadl Well, pom.xml isn't the most concise format ever... Never mind, this will generate WADL XML classes during every build, before the source code is compiled. I also love the fluent-api plugin that adds with*() methods along with ordinary setters, returning this to allow chaining. Pretty convenient. Finally we define more pleasant package name for generated artifacts (if you find net.java.dev.wadl._2009._02 package name pleasant enough, you can skip this step) and add Wadl prefix to all generated classes bindings.xjb file: We are now ready to produce and consume WADL in XML format using JAXB and POJO classes. Equipped with that knowledge and the foundation we are ready to develop some interesting library – which will be the subject of the next article. From http://nurkiewicz.blogspot.com/2012/01/gentle-introduction-to-wadl-in-java.html

Data exchanges between companies are increasing a lot. The number of applications that must be integrated is increasing, too. The interfaces use different technologies, protocols and data formats. Nevertheless, the integration of these applications must be modeled in a standardized way, realized efficiently and supported by automatic tests. Three integration frameworks are available in the JVM environment, which fulfil these requirements: Spring Integration, Mule ESB and Apache Camel. They implement the well-known Enteprise Integration Patterns (EIP, http://www.eaipatterns.com) and therefore offer a standardized, domain-specific language to integrate applications. These integration frameworks can be used in almost every integration project within the JVM environment – no matter which technologies, transport protocols or data formats are used. All integration projects can be realized in a consistent way without redundant boilerplate code. This article compares all three alternatives and discusses their pros and cons. If you want to know, when to use a more powerful Enterprise Service Bus (ESB) instead of one of these lightweight integration frameworks, then you should read this blog post: http://www.kai-waehner.de/blog/2011/06/02/when-to-use-apache-camel/ (it explains when to use Apache Camel, but the title could also be „When to use a lightweight integration framework“). Comparison Criteria Several criteria can be used to compare these three integration frameworks: Open source Basic concepts / architecture Testability Deployment Popularity Commercial support IDE-Support Errorhandling Monitoring Enterprise readiness Domain specific language (DSL) Number of components for interfaces, technologies and protocols Expandability Similarities All three frameworks have many similarities. Therefore, many of the above comparison criteria are even! All implement the EIPs and offer a consistent model and messaging architecture to integrate several technologies. No matter which technologies you have to use, you always do it the same way, i.e. same syntax, same API, same automatic tests. The only difference is the the configuration of each endpoint (e.g. JMS needs a queue name while JDBC needs a database connection url). IMO, this is the most significant feature. Each framework uses different names, but the idea is the same. For instance, „Camel routes“ are equivalent to „Mule flows“, „Camel components“ are called „adapters“ in Spring Integration. Besides, several other similarities exists, which differ from heavyweight ESBs. You just have to add some libraries to your classpath. Therefore, you can use each framework everywhere in the JVM environment. No matter if your project is a Java SE standalone application, or if you want to deploy it to a web container (e.g. Tomcat), JEE application server (e.g. Glassfish), OSGi container or even to the cloud. Just add the libraries, do some simple configuration, and you are done. Then you can start implementing your integration stuff (routing, transformation, and so on). All three frameworks are open source and offer familiar, public features such as source code, forums, mailing lists, issue tracking and voting for new features. Good communities write documentation, blogs and tutorials (IMO Apache Camel has the most noticeable community). Only the number of released books could be better for all three. Commercial support is available via different vendors: Spring Integration: SpringSource (http://www.springsource.com) Mule ESB: MuleSoft (http://www.mulesoft.org) Apache Camel: FuseSource (http://fusesource.com) and Talend (http://www.talend.com) IDE support is very good, even visual designers are available for all three alternatives to model integration problems (and let them generate the code). Each of the frameworks is enterprise ready, because all offer required features such as error handling, automatic testing, transactions, multithreading, scalability and monitoring. Differences If you know one of these frameworks, you can learn the others very easily due to their same concepts and many other similarities. Next, let’s discuss their differences to be able to decide when to use which one. The two most important differences are the number of supported technologies and the used DSL(s). Thus, I will concentrate especially on these two criteria in the following. I will use code snippets implementing the well-known EIP „Content-based Router“ in all examples. Judge for yourself, which one you prefer. Spring Integration Spring Integration is based on the well-known Spring project and extends the programming model with integration support. You can use Spring features such as dependency injection, transactions or security as you do in other Spring projects. Spring Integration is awesome, if you already have got a Spring project and need to add some integration stuff. It is almost no effort to learn Spring Integration if you know Spring itself. Nevertheless, Spring Integration only offers very rudimenary support for technologies – just „basic stuff“ such as File, FTP, JMS, TCP, HTTP or Web Services. Mule and Apache Camel offer many, many further components! Integrations are implemented by writing a lot of XML code (without a real DSL), as you can see in the following code snippet: You can also use Java code and annotations for some stuff, but in the end, you need a lot of XML. Honestly, I do not like too much XML declaration. It is fine for configuration (such as JMS connection factories), but not for complex integration logic. At least, it should be a DSL with better readability, but more complex Spring Integration examples are really tough to read. Besides, the visual designer for Eclipse (called integration graph) is ok, but not as good and intuitive as its competitors. Therefore, I would only use Spring Integration if I already have got an existing Spring project and must just add some integration logic requiring only „basic technologies“ such as File, FTP, JMS or JDBC. Mule ESB Mule ESB is – as the name suggests – a full ESB including several additional features instead of just an integration framework (you can compare it to Apache ServiceMix which is an ESB based on Apache Camel). Nevertheless, Mule can be use as lightweight integration framework, too – by just not adding and using any additional features besides the EIP integration stuff. As Spring Integration, Mule only offers a XML DSL. At least, it is much easier to read than Spring Integration, in my opinion. Mule Studio offers a very good and intuitive visual designer. Compare the following code snippet to the Spring integration code from above. It is more like a DSL than Spring Integration. This matters if the integration logic is more complex. The major advantage of Mule is some very interesting connectors to important proprietary interfaces such as SAP, Tibco Rendevous, Oracle Siebel CRM, Paypal or IBM’s CICS Transaction Gateway. If your integration project requires some of these connectors, then I would probably choose Mule! A disadvantage for some projects might be that Mule says no to OSGi: http://blogs.mulesoft.org/osgi-no-thanks/ Apache Camel Apache Camel is almost identical to Mule. It offers many, many components (even more than Mule) for almost every technology you could think of. If there is no component available, you can create your own component very easily starting with a Maven archetype! If you are a Spring guy: Camel has awesome Spring integration, too. As the other two, it offers a XML DSL: ${in.header.type} is ‘com.kw.DvdOrder’ ${in.header.type} is ‘com.kw.VideogameOrder’ Readability is better than Spring Integration and almost identical to Mule. Besides, a very good (but commercial) visual designer called Fuse IDE is available by FuseSource – generating XML DSL code. Nevertheless, it is a lot of XML, no matter if you use a visual designer or just your xml editor. Personally, I do not like this. Therefore, let’s show you another awesome feature: Apache Camel also offers DSLs for Java, Groovy and Scala. You do not have to write so much ugly XML. Personally, I prefer using one of these fluent DSLs instead XML for integration logic. I only do configuration stuff such as JMS connection factories or JDBC properties using XML. Here you can see the same example using a Java DSL code snippet: from(“file:incomingOrders “) .choice() .when(body().isInstanceOf(com.kw.DvdOrder.class)) .to(“file:incoming/dvdOrders”) .when(body().isInstanceOf(com.kw.VideogameOrder.class)) .to(“jms:videogameOrdersQueue “) .otherwise() .to(“mock:OtherOrders “); The fluent programming DSLs are very easy to read (even in more complex examples). Besides, these programming DSLs have better IDE support than XML (code completion, refactoring, etc.). Due to these awesome fluent DSLs, I would always use Apache Camel, if I do not need some of Mule’s excellent connectors to proprietary products. Due to its very good integration to Spring, I would even prefer Apache Camel to Spring Integration in most use cases. By the way: Talend offers a visual designer generating Java DSL code, but it generates a lot of boilerplate code and does not allow vice-versa editing (i.e. you cannot edit the generated code). This is a no-go criteria and has to be fixed soon (hopefully)! And the winner is… … all three integration frameworks, because they are all lightweight and easy to use – even for complex integration projects. It is awesome to integrate several different technologies by always using the same syntax and concepts – including very good testing support. My personal favorite is Apache Camel due to its awesome Java, Groovy and Scala DSLs, combined with many supported technologies. I would only use Mule if I need some of its unique connectors to proprietary products. I would only use Spring Integration in an existing Spring project and if I only need to integrate „basic technologies“ such as FTP or JMS. Nevertheless: No matter which of these lightweight integration frameworks you choose, you will have much fun realizing complex integration projects easily with low efforts. Remember: Often, a fat ESB has too much functionality, and therefore too much, unnecessary complexity and efforts. Use the right tool for the right job! Best regards, Kai Wähner (Twitter: @KaiWaehner) http://www.kai-waehner.de/blog/2012/01/10/spoilt-for-choice-which-integration-framework-to-use-spring-integration-mule-esb-or-apache-camel/

One of the methods of exchanging data between processes with the multiprocessing module is directly shared memory via multiprocessing.Value. As any method that’s very general, it can sometimes be tricky to use. I’ve seen a variation of this question asked a couple of times on StackOverflow: I have some processes that do work, and I want them to increment some shared counter because [... some irrelevant reason ...] – how can this be done? The wrong way And surprisingly enough, some answers given to this question are wrong, since they use multiprocessing.Value incorrectly, as follows: import time from multiprocessing import Process, Value def func(val): for i in range(50): time.sleep(0.01) val.value += 1 if __name__ == '__main__': v = Value('i', 0) procs = [Process(target=func, args=(v,)) for i in range(10)] for p in procs: p.start() for p in procs: p.join() print v.value This code is a demonstration of the problem, distilling only the usage of the shared counter. A "pool" of 10 processes is created to run the func function. All processes share a Value and increment it 50 times. You would expect this code to eventually print 500, but in all likeness it won’t. Here’s some output taken from 10 runs of that code: > for i in {1..10}; do python sync_nolock_wrong.py; done 435 464 484 448 491 481 490 471 497 494 Why does this happen? I must admit that the documentation of multiprocessing.Value can be a bit confusing here, especially for beginners. It states that by default, a lock is created to synchronize access to the value, so one may be falsely led to believe that it would be OK to modify this value in any way imaginable from multiple processes. But it’s not. Explanation – the default locking done by Value This section is advanced and isn’t strictly required for the overall flow of the post. If you just want to understand how to synchronize the counter correctly, feel free to skip it. The locking done by multiprocessing.Value is very fine-grained. Value is a wrapper around a ctypes object, which has an underlying value attribute representing the actual object in memory. All Value does is ensure that only a single process or thread may read or write this value attribute simultaneously. This is important, since (for some types, on some architectures) writes and reads may not be atomic. I.e. to actually fill up the object’s memory, the CPU may need several instructions, and another process reading the same (shared) memory at the same time could see some intermediate, invalid state. The built-in lock of Value prevents this from happening. However, when we do this: val.value +=1 What Python actually performs is the following (disassembled bytecode with the dis module). I’ve annotated the locking done by Value in #<-- comments: 0 LOAD_FAST 0 (val) 3 DUP_TOP #<--- Value lock acquired 4 LOAD_ATTR 0 (value) #<--- Value lock released 7 LOAD_CONST 1 (1) 10 INPLACE_ADD 11 ROT_TWO #<--- Value lock acquired 12 STORE_ATTR 0 (value) #<--- Value lock released So it’s obvious that while process #1 is now at instruction 7 (LOAD_CONST), nothing prevents process #2 from also loading the (old) value attribute and be on instruction 7 too. Both processes will proceed incrementing their private copy and writing it back. The result: the actual value got incremented only once, not twice. The right way Fortunately, this problem is very easy to fix. A separate Lock is needed to guarantee the atomicity of modifications to the Value: import time from multiprocessing import Process, Value, Lock def func(val, lock): for i in range(50): time.sleep(0.01) with lock: val.value += 1 if __name__ == '__main__': v = Value('i', 0) lock = Lock() procs = [Process(target=func, args=(v, lock)) for i in range(10)] for p in procs: p.start() for p in procs: p.join() print v.value Now we get the expected result: > for i in {1..10}; do python sync_lock_right.py; done 500 500 500 500 500 500 500 500 500 500 A value and a lock may appear like too much baggage to carry around at all times. So, we can create a simple "synchronized shared counter" object to encapsulate this functionality: import time from multiprocessing import Process, Value, Lock class Counter(object): def __init__(self, initval=0): self.val = Value('i', initval) self.lock = Lock() def increment(self): with self.lock: self.val.value += 1 def value(self): with self.lock: return self.val.value def func(counter): for i in range(50): time.sleep(0.01) counter.increment() if __name__ == '__main__': counter = Counter(0) procs = [Process(target=func, args=(counter,)) for i in range(10)] for p in procs: p.start() for p in procs: p.join() print counter.value() Bonus: since we’ve now placed a more coarse-grained lock on the modification of the value, we may throw away Value with its fine-grained lock altogether, and just use multiprocessing.RawValue, that simply wraps a shared object without any locking. Source: http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing/

SOAP uses XML to marshal data that is transported to a software application. Since SOAP’s introduction, three SOAP encoding styles have become popular and are reliably implemented across software vendors and technology providers: SOAP Remote Procedure Call (RPC) encoding, also known as Section 5 encoding, which is defined by the SOAP 1.1 specification SOAP Remote Procedure Call Literal encoding (SOAP RPC-literal), which uses RPC methods to make calls but uses an XML do-it-yourself method for marshalling the data SOAP document-style encoding, which is also known as message-style or document-literal encoding. There are other encoding styles, but software developers have not widely adopted them, mostly because their promoters disagree on a standard. For example, Microsoft is promoting Direct Internet Message Exchange (DIME) to encode binary file data, while the rest of the world is promoting SOAP with Attachments. SOAP RPC encoding, RPC-literal, and document-style SOAP encoding have emerged as the encoding styles that a software developer can count on. SOAP RPC is the encoding style that offers you the most simplicity. You make a call to a remote object, passing along any necessary parameters. The SOAP stack serializes the parameters into XML, moves the data to the destination using transports such as HTTP and SMTP, receives the response, deserializes the response back into objects, and returns the results to the calling method. Whew! SOAP RPC handles all the encoding and decoding, even for very complex data types, and binds to the remote object automatically. Now, imagine that you have some data already in XML format. SOAP RPC also allows literal encoding of the XML data as a single field that is serialized and sent to the Web service host. This is what’s referred to as RPC-literal encoding. Since there is only one parameter — the XML tree — the SOAP stack only needs to serialize one value. The SOAP stack still deals with the transport issues to get the request to the remote object. The stack binds the request to the remote object and handles the response. Lastly, in a SOAP document-style call, the SOAP stack sends an entire XML document to a server without even requiring a return value. The message can contain any sort of XML data that is appropriate to the remote service. In SOAP document-style encoding, the developer handles everything, including determining the transport (e.g., HTTP, MQ, SMTP), marshaling and unmarshaling the body of the SOAP envelope, and parsing the XML in the request and response to find the needed data. The three encoding systems are compared here: SOAP RPC encoding is easiest for the software developer; however, all that ease comes with a scalability and performance penalty. In SOAP RPC-literal encoding, you are more involved with handling XML parsing, but it requires there to be overhead for the SOAP stack to deal with. SOAP document-literal encoding is most difficult for the software developer, but consequently requires little SOAP overhead. Why is SOAP RPC easier? With this encoding style, you only need to define the public object method in your code once; the SOAP stack unmarshals the request parameters into objects and passes them directly into the method call of your object. Otherwise, you are stuck with the task of parsing through the XML tree to find the data elements you need before you get to make the call to the public method. There is an argument for parsing the XML data yourself: since you know the data in the XML tree best, your code will parse that data more efficiently than generalized SOAP stack code. You will find this when measuring scalability and performance in SOAP encoding styles. References: 1. Discover SOAP encoding’s impact on Web service performance (http://www.ibm.com/developerworks/webservices/library/ws-soapenc/) From http://singztechmusings.wordpress.com/2011/12/20/different-soap-encoding-styles-rpc-rpc-literal-and-document-literal/

So you're still interested in unit testing JavaScript (good). This post is an extension of my much more indepth first posting on how to unit test JavaScript using JS Test Driver. Please check it out here. Recap Last Posting In the last posting we successfully unit tested JavaScript using Maven and JsTest Driver. This allowes us to test JavaScript when on an environment that has a modern browser installed and can be run. Problem with typical CI environments So what happens when the test are passing on your local box, but you go to check in your code and the Continuous Integration (CI) server pukes on the new tests becasue there is no "screen" to run chrome or firefox? As of this posting, none of the top-tier browsers have a "headless" or an in-memory only browser window. There are alternatives to running JavaScript in a browser, such as rhino.js, env.js or HtmlUnit, however, these are just ports of browsers and the JavaScript and DOM representation are not 100% accurate which can lead to problems with your code when rendered in a client's browser. Approach What we need to do is to run JSTestDriver's browser in a Virtual X Framebuffer (Xvfb) which is possible on nearly all Linux based systems. The example below uses a Solaris version of Linux, however, Debian and RedHat linux distrubutions come with the simplified bash script to easily run an appliation in a virtual framebuffer. This solution was derived from one posted solution on the JS Test Driver wiki. The given example is also a full working example that is in use at my current client. Here is the quick list of what we will accomplish. Note, several of these steps are discussed in depth in the previous post and are not covered in depth here. Create a profile to run Js Unit-Tests Copy JsTestDriver library to a known location for Maven to use Copy JavaScript main and test files to known locations Use ANT to start JsTestDriver and pipe the screen into xvfb Here is a sample profile to use. You will need to adjust the properties at the top of the profile to match your system. ci-jstests /opt/swf/bin/firefox 1.3.2 /opt/X11R6/xvfb-run org.apache.maven.plugins maven-dependency-plugin 2.1 copy generate-resources copy com.google.jstestdriver jstestdriver ${js-test-driver.version} jar true jsTestDriver.jar ${project.build.directory}/jstestdriver false true maven-resources-plugin 2.4.3 copy-main-files generate-test-resources copy-resources ${project.build.directory}/test-classes/main-js src/main/webapp/scripts false copy-test-files generate-test-resources copy-resources ${project.build.directory}/test-classes/test-js src/test/webapp/scripts false org.apache.maven.plugins maven-antrun-plugin 1.6 test run Possible problems Although I cannot predict or fix all problems, I can share the one major problem I ran into with Solaris and the script used to fix that. In Solaris (and could happen to other distros) the xvfb-run script was not available and several of the other libraries did not exist. I first had to download the latest X libraries and place them in their appropriate locations on the CI server. Next, I had to re-engineer the xvfb-run script. Here is a copy of my script (NOTE: This is the solution for my server and this may not work for you) I created a script that contains: /usr/openwin/bin/Xvfb :1 screen 0 1280x1024x8 pixdepths 8 24 fbdir /tmp/.X11-vbf & From http://www.ensor.cc/2011/08/maven-javascript-unit-test-running-in.html

This is the sixth of a series of articles about setting up a secure RESTful Web Service using Spring 3.1 and Spring Security 3.1. A previous article introduced security in the context of a RESTful service, using form-based authentication. This article will focus on configuration of Basic and Digest authentication and on configuring both protocols for the same URI mapping of the API, using Spring Security 3.1. The REST with Spring series: Part 1 – Bootstrapping a web application with Spring 3.1 and Java based Configuration Part 2 – Building a RESTful Web Service with Spring 3.1 and Java based Configuration Part 3 – Securing a RESTful Web Service with Spring Security 3.1 Part 4 – RESTful Web Service Discoverability Part 5 – REST Service Discoverability with Spring Configuration of Basic Authentication In part 3 of the series, the Spring Security configuration was done using form based authentication, which is not really ideal for a RESTful service. To start setting up basic authentication, first we remove the old custom entry point and filter from the main security element: Note how support for basic authentication has been added with a single configuration line – – which handles the creation and wiring of both the BasicAuthenticationFilter and the BasicAuthenticationEntryPoint. Satisfying the stateless constraint – getting rid of sessions One of the main constraints of the RESTful architectural style is that the client-server communication is fully stateless, as the original dissertation reads: 5.1.3 Stateless We next add a constraint to the client-server interaction: communication must be stateless in nature, as in the client-stateless-server (CSS) style of Section 3.4.3 (Figure 5-3), such that each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client. The concept of Session on the server is one with a long history in Spring Security, and removing it entirely has been difficult until now, especially when configuration was done by using the namespace. However, Spring Security 3.1 augments the namespace configuration with a new stateless option for session creation, which effectively guarantees that no session will be created or used by Spring. What this new option does is completely removes all session related filters from the security filter chain, ensuring that authentication is performed for each request. Configuration of Digest Authentication Starting with the previous configuration, the filter and entry point necessary to set up digest authentication will be defined as beans. Then, the digest entry point will override the one created by behind the scenes. Finally, the custom digest filter will be introduced in the security filter chain using the after semantics of the security namespace to position it directly after the basic authentication filter. Unfortunately there is no support in the security namespace to automatically configure the digest authentication the way basic authentication can be configured with . Because of that, the necessary beans had to be defined and wired manually into the security configuration. Supporting both authentication protocols in the same RESTful service Basic or Digest authentication alone can be easily implemented in Spring Security 3.x; it is supporting both of them for the same RESTful web service, on the same URI mappings that introduces a new level of complexity into the configuration and testing of the service. Anonymous request With both basic and digest filters in the security chain, the way a anonymous request – a request containing no authentication credentials (Authorization HTTP header) – is processed by Spring Security is – the two authentication filters will find no credentials and will continue execution of the filter chain. Then, seeing how the request wasn’t authenticated, an AccessDeniedException is thrown and caught in the ExceptionTranslationFilter, which commences the digest entry point, prompting the client for credentials. The responsibilities of both the basic and digest filters are very narrow – they will continue to execute the security filter chain if they are unable to identify the type of authentication credentials in the request. It is because of this that Spring Security can have the flexibility to be configured with support for multiple authentication protocols on the same URI. When a request is made containing the correct authentication credentials – either basic or digest – that protocol will be rightly used. However, for an anonymous request, the client will get prompted only for digest authentication credentials. This is because the digest entry point is configured as the main and single entry point of the Spring Security chain; as such digest authentication can be considered the default. Request with authentication credentials A request with credentials for Basic authentication will be identified by the Authorization header starting with the prefix “Basic”. When processing such a request, the credentials will be decoded in the basic authentication filter and the request will be authorized. Similarly, a request with credentials for Digest authentication will use the prefix “Digest” for it’s Authorization header. Testing both scenarios The tests will consume the REST service by creating a new resource after authenticating with either basic or digest: @Test public void givenAuthenticatedByBasicAuth_whenAResourceIsCreated_then201IsReceived(){ // Given // When Response response = given() .auth().preemptive().basic( ADMIN_USERNAME, ADMIN_PASSWORD ) .contentType( HttpConstants.MIME_JSON ).body( new Foo( randomAlphabetic( 6 ) ) ) .post( this.paths.getFooURL() ); // Then assertThat( response.getStatusCode(), is( 201 ) ); } @Test public void givenAuthenticatedByDigestAuth_whenAResourceIsCreated_then201IsReceived(){ // Given // When Response response = given() .auth().digest( ADMIN_USERNAME, ADMIN_PASSWORD ) .contentType( HttpConstants.MIME_JSON ).body( new Foo( randomAlphabetic( 6 ) ) ) .post( this.paths.getFooURL() ); // Then assertThat( response.getStatusCode(), is( 201 ) ); } Note that the test using basic authentication adds credentials to the request preemptively, regardless if the server has challenged for authentication or not. This is to ensure that the server doesn’t need to challenge the client for credentials, because if it did, the challenge would be for Digest credentials, since that is the default. Conclusion This article covered the configuration and implementation of both Basic and Digest authentication for a RESTful service, using mostly Spring Security 3.0 namespace support as well as some new features added by Spring Security 3.1. In the next articles I will focus on OAuth authentication. In the meantime, check out the github project. From the REST with Spring series.

Develop your own API service for your PHP projects.

1. Overview This is the third of a series of articles about setting up a secure RESTful Web Service using Spring 3.1 and Spring Security 3.1 with Java based configuration. This article will focus on the security configuration using Spring Security 3.1, assuming some understanding of Spring Security basics and focusing on the specifics of securing the RESTful web service. The REST with Spring series: Part 1 – Bootstrapping a web application with Spring 3.1 and Java based Configuration Part 2 – Building a RESTful Web Service with Spring 3.1 and Java based Configuration Part 4 – RESTful Web Service Discoverability Part 5 – REST Service Discoverability with Spring Part 6 – Basic and Digest authentication for a RESTful Service with Spring Security 3.1 Part 7 – REST Pagination in Spring Part 8 – Authentication against a RESTful Service with Spring Security Part 9 – ETags for REST with Spring 2. Introducing Spring Security in the web.xml The architecture of Spring Security is based entirely on servlet filters and, as such, comes before Spring MVC in regards to the processing of HTTP requests. Keeping this in mind, to begin with, a filter needs to be declared in the web.xml of the application: springSecurityFilterChain org.springframework.web.filter.DelegatingFilterProxy springSecurityFilterChain /* The filter must necessarily be named ‘springSecurityFilterChain’ to match the default bean created by Spring Security in the container. Note that the defined filter is not the actual class implementing the security logic but a DelegatingFilterProxy with the purpose of delegating the Filter’s methods to an internal bean. This is done so that the target bean can still benefit from the Spring context lifecycle and flexibility. The URL pattern used to configure the Filter is /* even though the entire web service is mapped to /api/* so that the security configuration has the option to secure other possible mappings as well, if required. 3. The security configuration Most of the configuration is done using the security namespace – for this to be enabled, the schema locations must be defined and pointed to the new 3.1 versions. The namespace is designed so that it expresses the common uses of Spring Security while still providing hooks to the underlying beans. 3.1. The basics The element is the main container element for HTTP security configuration. In the current implementation, it only secured a single mapping: /api/admin/**. Note that the mapping is relative to the root context of the web application, not to the rest servlet; this is because the entire security configuration lives in the root Spring context and not in the child context of the servlet. 3.2. The entry point In a standard web application, the authentication process may be automatically triggered when the client tries to access a secured resource without being authenticated – this is usually done by redirecting to a login page so that the user can enter credentials. However, for a RESTful Web Service this behavior doesn’t make much sense – authentication should only be done by a request to the correct URI and all other requests should simply fail with a 401 UNAUTHORIZED status code if the user is not authenticated. Spring Security handles this automatic triggering of the authentication process with the concept of an entry point; the entry point is a required part of the configuration, and can be injected via the entry-point-ref attribute of the element. Keeping in mind that this functionality doesn’t make sense in the context of the RESTful web service, the new custom entry point is defined: @Component( "restAuthenticationEntryPoint" ) public final class RestAuthenticationEntryPoint implements AuthenticationEntryPoint{ @Override public final void commence ( HttpServletRequest request, HttpServletResponse response, AuthenticationException authException ) throws IOException{ response.sendError( HttpServletResponse.SC_UNAUTHORIZED, "Unauthorized" ); } } 3.3. The login There are multiple ways to do authentication for a RESTful Web Service – one of the default Spring Security provides is form login – which uses an authentication processing filter – org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter. Since the element doesn’t automatically create this particular filter by default, it needs to be explicitly specified in the configuration, using the element at the position FORM_LOGIN_FILTER; the only required dependency for this bean is the authentication manager. Note that for a standard web application, the auto-configattribute of the element is shorthand syntax for some useful security configuration. While this may be appropriate for some very simple configurations, it doesn’t fit and should not be used for a REST API. 3.4. Authentication should return 200 instead of 301 By default, form login will answer a successful authentication request with a 301 MOVED PERMANENTLY status code; this makes sense in the context of an actual login form which needs to redirect after login. For a RESTful web service however, the desired response for a successful authentication should be 200 OK. This is done by injecting a custom authentication success handler in the form login filter, to replace the default one. The new handler implements the exact same login as the default org.springframework.security.web.authentication.SavedRequestAwareAuthenticationSuccessHandler with one notable difference – the redirect logic is removed: // Use the DefaultSavedRequest URL // final String targetUrl = savedRequest.getRedirectUrl(); // this.logger.debug( "Redirecting to DefaultSavedRequest Url: " + targetUrl ); // this.getRedirectStrategy().sendRedirect( request, response, targetUrl ); 3.5. The authentication manager and provider The authentication process uses an in-memory provider to perform authentication – this is meant to simplify the configuration as a production implementation of these artifacts is outside the scope of this post. 4. Maven and other trouble In addition to the pom.xml from the first post, as well as the one from the second post, the Spring Security maven dependencies need to be added: org.springframework.security spring-security-web ${spring-security.version} org.springframework.security spring-security-config ${spring-security.version} org.springframework spring-tx ${spring.version} org.springframework spring-aop ${spring.version} 3.2.2.RELEASE 3.1.3.RELEASE Notice that the version of the security dependencies is no longer 3.1.0.BUILD-SNAPSHOT as for the standard Spring framework, but 3.1.0.CI-SNAPSHOT. What is more, the security artifacts define dependencies to the 3.0.x versions of Spring, more specifically spring-security-web depends on spring-aop and on spring-tx version 3.0.x instead of the expected 3.1.x. To understand why this is a problem, we need to understand how the Maven conflict resolution algorithm works – in case of conflict, Maven will chose which jar to include based on the distance between the particular dependency and the root of the tree. In our case, the conflicts are the spring-aop and spring-tx jars, appearing once with version 3.0.6 and once with 3.1.0. In the case of spring-aop, it appears once as a level 1 dependency of both spring-security-web and spring-security-config with version 3.0.6 , and once as a level 2 dependency of spring-webmvc with the version 3.1.0; since the 3.0.6 versioned jar is closer to the root, it will be the one chosen by the conflict resolution mechanism. Now that we understand why it is that Maven will deploy the 3.0.6 version of the jars with the application and not the intended 3.1.0 version, we need to address the issue. The solution is to add the two dependencies, with the intended 3.1.0 versions, directly into the pom – this will shorten the distance between them and the root to 0 and will force Maven to use them first. 5. Conclusion This post covered the basic security configuration and implementation for a RESTful service using Spring Security 3.1, discussing the web.xml, the security configuration, the HTTP status codes for the authentication process and the Maven resolution of the security artifacts. In the next articles I will focus on a Java based configuration for Spring Security, integration testing of the secure API using the rest-assured library and HTTP basic authentication. In the meantime, check out the github project. If you read this far, you should follow me on twitter here. Original at Securing a RESTful Web Service with Spring Security 3.1 from the REST with Spring series.

1. Overview This is the second of a series of posts about setting up a RESTful web service using Spring 3.1 with Java based configuration. The first post of the series focused on bootstrapping the web application; this post will focus on setting up REST in Spring, the Controller and HTTP response codes, configuration of payload marshalling and content negotiation. 2. Understanding REST in Spring The Spring framework supports 2 ways of creating RESTful services: using MVC with ModelAndView using HTTP message converters The ModelAndView approach is older and much better documented, but also more verbose and configuration heavy. It tries to shoehorn the REST paradigm into the old model, which is not without problems. The Spring team understood this and provided first-class REST support starting with Spring 3.0. The new approach, based on HttpMessageConverter and annotations, is much more lightweight and easy to implement. Configuration is minimal and it provides sensible defaults for what you would expect from a RESTful service. It is however newer and a a bit on the light side concerning documentation; what’s more, the Spring reference doesn’t go out of it’s way to make the distinction and the tradeoffs between the two approaches as clear as they should be. Nevertheless, this is the way RESTful services should be build after Spring 3.0. 3. The Java configuration @Configuration @EnableWebMvc public class WebConfig{ // } The new @EnableWebMvc annotation does a number of useful things – specifically, in the case of REST, it detect the existence of Jackson and JAXB 2 on the classpath and automatically creates and registers default JSON and XML converters. The functionality of the annotation is equivalent to the XML version: This is a shortcut, and though it may be useful in many situations, it’s not perfect. When more complex configuration is needed, remove the annotation and extend WebMvcConfigurationSupport directly. 4. Testing the Spring context Starting with Spring 3.1, we get first-class testing support for @Configuration classes: @RunWith( SpringJUnit4ClassRunner.class ) @ContextConfiguration( classes = { ApplicationConfig.class, PersistenceConfig.class },loader = AnnotationConfigContextLoader.class ) public class SpringTest{ @Test public void whenSpringContextIsInstantiated_thenNoExceptions(){ // When } } The Java configuration classes are simply specified with the @ContextConfiguration annotation and the new AnnotationConfigContextLoader loads the bean definitions from the @Configuration classes. Notice that the WebConfig configuration class was not included in the test because it needs to run in a servlet context, which is not provided. 5. The Controller The @Controller is the central artifact in the entire Web Tier of the RESTful API. For the purpose of this post, the controller is modeling a simple REST resource – Foo: @Controller class FooController{ @Autowired IFooService service; @RequestMapping( value = "foo",method = RequestMethod.GET ) @ResponseBody public List< Foo > getAll(){ return this.service.getAll(); } @RequestMapping( value = "foo/{id}",method = RequestMethod.GET ) @ResponseBody public Foo get( @PathVariable( "id" ) Long id ){ return RestPreconditions.checkNotNull( this.service.getById( id ) ); } @RequestMapping( value = "foo",method = RequestMethod.POST ) @ResponseStatus( HttpStatus.CREATED ) @ResponseBody public Long create( @RequestBody Foo entity ){ RestPreconditions.checkNotNullFromRequest( entity ); return this.service.create( entity ); } @RequestMapping( value = "foo",method = RequestMethod.PUT ) @ResponseStatus( HttpStatus.OK ) public void update( @RequestBody Foo entity ){ RestPreconditions.checkNotNullFromRequest( entity ); RestPreconditions.checkNotNull( this.service.getById( entity.getId() ) ); this.service.update( entity ); } @RequestMapping( value = "foo/{id}",method = RequestMethod.DELETE ) @ResponseStatus( HttpStatus.OK ) public void delete( @PathVariable( "id" ) Long id ){ this.service.deleteById( id ); } } The Controller implementation is non-public – this is because there is no need for it to be. Usually the controller is the last in the chain of dependencies – it receives HTTP requests from the Spring front controller (the DispathcerServlet) and simply delegate them forward to a service layer. If there is no use case where the controller has to be injected or manipulated through a direct reference, then I prefer not to declare it as public. The request mappings are straightforward – as with any Spring controller, the actual value of the mapping as well as the HTTP method are used to determine the target method for the request. @RequestBody will bind the parameters of the method to the body of the HTTP request, whereas @ResponseBody does the same for the response and return type. They also ensure that the resource will be marshalled and unmarshalled using the correct HTTP converter. Content negotiation will take place to choose which one of the active converters will be used, based mostly on the Accept header, although other HTTP headers may be used to determine the representation as well. 6. Mapping the HTTP response codes The status codes of the HTTP response are one of the most important parts of the REST service, and the subject can quickly become very complex. Getting these right can be what makes or breaks the service. 6.1. Unmapped requests If Spring MVC receives a request which doesn’t have a mapping, it considers the request not to be allowed and returns a 405 METHOD NOT ALLOWED back to the client. It is also good practice to include the Allow HTTP header when returning a 405 to the client, in order to specify which operations are allowed. This is the standard behavior of Spring MVC and does not require any additional configuration. 6.2. Valid, mapped requests For any request that does have a mapping, Spring MVC considers the request valid and responds with 200 OK if no other status code is specified otherwise. It is because of this that controller declares different @ResponseStatus for the create, update and delete actions but not for get, which should indeed return the default 200 OK. 6.3. Client error In case of a client error, custom exceptions are defined and mapped to the appropriate error codes. Simply throwing these exceptions from any of the layers of the web tier will ensure Spring maps the corresponding status code on the HTTP response. @ResponseStatus( value = HttpStatus.BAD_REQUEST ) public class BadRequestException extends RuntimeException{ // } @ResponseStatus( value = HttpStatus.NOT_FOUND ) public class ResourceNotFoundException extends RuntimeException{ // } These exceptions are part of the REST API and, as such, should only be used in the appropriate layers corresponding to REST; if for instance a DAO/DAL layer exist, it should not use the exceptions directly. Note also that these are not checked exceptions but runtime exceptions – in line with Spring practices and idioms. 6.4. Using @ExceptionHandler Another option to map custom exceptions on specific status codes is to use the @ExceptionHandler annotation in the controller. The problem with that approach is that the annotation only applies to the controller in which it is defined, not to the entire Spring Container, which means that it needs to be declared in each controller individually. This quickly becomes cumbersome, especially in more complex applications which many controllers. There are a few JIRA issues opened with Spring at this time to handle this and other related limitations: SPR-8124, SPR-7278, SPR-8406. 7. Additional Maven dependencies In addition to the pom.xml from the first post, two dependencies need to be added: org.codehaus.jackson jackson-mapper-asl ${jackson-mapper-asl.version} runtime javax.xml.bind jaxb-api ${jaxb-api.version} runtime 1.9.12 2.2.4 These are the libraries used to convert the representation of the REST resource to either JSON or XML. 8. Conclusion This post covered the configuration and implementation of a RESTful service using Spring 3.1 and Java based configuration, discussing HTTP response codes, basic content negotiation and marshaling. In the next articles of the series I will focus on discoverability of the API, advanced content negotiation and working with additional representations of a resource. In the meantime, check out the github project.

Version 2.3 of the Java servlet spec introduced the concept of filters. According to the documentation from Oracle’s site: “A filter dynamically intercepts requests and responses to transform or use the information contained in the requests or responses”. Today I’ll show you how to build a simple filter to intercept the response HTTP response code using annotations introduced in the Servlet 3.0 specification. With NetBeans IDE 7 create a new Maven Java Web Application called: Intercept Delete the index.jsp file under the Web Pages folder. Right-click on the project and add a new servlet called: MainServlet Since we are using the new Servlet 3 annotations we don’t need to set a whole lot of properties. Maven generates a decent MainServlet.java file for us, I just removed the comments for the output. My file looks like this: package com.giantflyingsaucer.intercept; import java.io.IOException; import java.io.PrintWriter; import javax.servlet.ServletException; import javax.servlet.annotation.WebServlet; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; @WebServlet(name = "MainServlet", urlPatterns = {"/"}) public class MainServlet extends HttpServlet { protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setContentType("text/html;charset=UTF-8"); PrintWriter out = response.getWriter(); try { out.println(""); out.println(""); out.println(""); out.println(""); out.println(""); out.println("Servlet MainServlet"); out.println(""); out.println(""); } finally { out.close(); } } // /** * Handles the HTTP GET method. * @param request servlet request * @param response servlet response * @throws ServletException if a servlet-specific error occurs * @throws IOException if an I/O error occurs */ @Override protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { processRequest(request, response); } /** * Handles the HTTP POST method. * @param request servlet request * @param response servlet response * @throws ServletException if a servlet-specific error occurs * @throws IOException if an I/O error occurs */ @Override protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { processRequest(request, response); } /** * Returns a short description of the servlet. * @return a String containing servlet description */ @Override public String getServletInfo() { return "Short description"; }// } Right-click on the project and add a Filter called: InterceptFilter We will add the following two lines to the doFilter method. HttpServletResponse hsr = (HttpServletResponse) response; System.out.println("HTTP Status: " + hsr.getStatus()); My doFilter method looks like this: @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { if (debug) { log("InterceptFilter:doFilter()"); } doBeforeProcessing(request, response); HttpServletResponse hsr = (HttpServletResponse) response; System.out.println("HTTP Status: " + hsr.getStatus()); Throwable problem = null; try { chain.doFilter(request, response); } catch (Throwable t) { problem = t; t.printStackTrace(); } doAfterProcessing(request, response); if (problem != null) { if (problem instanceof ServletException) { throw (ServletException) problem; } if (problem instanceof IOException) { throw (IOException) problem; } sendProcessingError(problem, response); } } Clean and Build the project and deploy it to Apache Tomcat. Access the URL with a browser and take a look at your catalina.out file and you should see the HTTP response code. Note: You shouldn’t need to do any changes to the web.xml file for this project to work. From http://www.giantflyingsaucer.com/blog/?p=3279

OpenStreetMap is a global project with an aim of collaboratively collecting map data, and today Ken Guest has submitted his PHP package for communitcating with the OSM API to the public and the PEAR PEPr review process: So over the last while, I’ve been working on a PHP package imaginatively named Services_Openstreetmap, for interacting with the openstreetmap API. I initially needed it so I could search for certain POIs and tabulate the results; it’s now also capable of adding data to the openstreetmap database – nodes and other elements can be created, updated and so on. It will even access the details of the user that is being used to modify that data, which is one difference between it and the other single purpose OSM frameworks. --Ken Guest You can view the submission here, and you should definitely take a look at openstreetmap.org if you haven't already. Good news for PHP developers looking to use this project more heavily in their applications.

i guess, i’m writing this post after a long time. this time, i’m writing about apache tika api that a friend of mine and i tried out to extract/retrieve metadata information from audio files supported by it – .mp3, .aiff, .au, .midi, .wav. to make it clear, here’s a screenshot of the information shown by windows vista about an audio file: we wanted to extract this using java and with googling, found that apache tika would help. we needed this metadata to index audio files for it to be searchable in a search application that we’re building using apache lucene . here’s a sample java program that extracts metadata from an mp3 file: package singz.samples.search.audio.metadata; import java.io.file; import java.io.fileinputstream; import java.io.filenotfoundexception; import java.io.ioexception; import java.io.inputstream; import org.apache.tika.exception.tikaexception; import org.apache.tika.metadata.metadata; import org.apache.tika.parser.parsecontext; import org.apache.tika.parser.parser; import org.apache.tika.parser.mp3.mp3parser; import org.xml.sax.contenthandler; import org.xml.sax.saxexception; import org.xml.sax.helpers.defaulthandler; /** * @author singaram subramanian * extract metadata of an audio file using apache tika api * */ public class audiometadataextractordemo { public static void main(string[] args) { // this audio file has metadata embedded in xmp (extensible metadata platform) standard // created by adobe systems inc. xmp standardizes the definition, creation, and // processing of extensible metadata. string audiofileloc = "c:\\pop\\backstreetboys_showmethemeaningofbeinglonely.mp3"; try { inputstream input = new fileinputstream(new file(audiofileloc)); contenthandler handler = new defaulthandler(); metadata metadata = new metadata(); parser parser = new mp3parser(); parsecontext parsectx = new parsecontext(); parser.parse(input, handler, metadata, parsectx); input.close(); // list all metadata string[] metadatanames = metadata.names(); for(string name : metadatanames){ system.out.println(name + ": " + metadata.get(name)); } // retrieve the necessary info from metadata // names - title, xmpdm:artist etc. - mentioned below may differ based // on the standard used for processing and storing standardized and/or // proprietary information relating to the contents of a file. system.out.println("title: " + metadata.get("title")); system.out.println("artists: " + metadata.get("xmpdm:artist")); system.out.println("genre: " + metadata.get("xmpdm:genre")); } catch (filenotfoundexception e) { e.printstacktrace(); } catch (ioexception e) { e.printstacktrace(); } catch (saxexception e) { e.printstacktrace(); } catch (tikaexception e) { e.printstacktrace(); } } } maven pom xml 4.0.0 singz.samples.search.audio audiometadataextractor 0.0.1 jar audiometadataextractor http://maven.apache.org utf-8 org.apache.tika tika-core 0.10 org.apache.tika tika-parsers 0.10 output xmpdm:releasedate: 2001 xmpdm:audiochanneltype: stereo xmpdm:album: top 100 pop author: backstreet boys xmpdm:artist: backstreet boys channels: 2 xmpdm:audiosamplerate: 44100 xmpdm:logcomment: eng xmpdm:tracknumber: 04 version: mpeg 3 layer iii version 1 xmpdm:composer: null xmpdm:audiocompressor: mp3 title: show me the meaning of being lonely samplerate: 44100 xmpdm:genre: pop content-type: audio/mpeg title: show me the meaning of being lonely artists: backstreet boys genre: pop about apache tika http://tika.apache.org/index.html “the apache tika™ toolkit detects and extracts metadata and structured text content from various documents using existing parser libraries.” http://www.lucidimagination.com/devzone/technical-articles/content-extraction-tika#article.tika “apache tika is a content type detection and content extraction framework. tika provides a general application programming interface that can be used to detect the content type of a document and also parse textual content and metadata from several document formats. tika does not try to understand the full variety of different document formats by itself but instead delegates the real work to various existing parser libraries such as apache poi for microsoft formats, pdfbox for adobe pdf, neko html for html etc. the grand idea behind tika is that it offers a generic interface for parsing multiple formats. the tika api hides the technical differences of the various parser implementations. this means that you don’t have to learn and consume one api for every format you use but can instead use a single api – the tika api. internally tika usually delegates the parsing work to existing parsing libraries and adapts the parse result so that client applications can easily manage variety of formats. tika aims to be efficient in using available resources (mainly ram) while parsing. the tika api is stream oriented so that the parsed source document does not need to be loaded into memory all at once but only as it is needed. ultimately, however, the amount of resources consumed is mandated by the parser libraries that tika uses. at the time of writing this, tika supports directly around 30 document formats. see list of supported document formats . the list of supported document formats is not limited by tika in any way. in the simplest case you can add support for new document formats by implementing a thin adapter that that implements the parser interface for the new document format.” about xmp standard http://en.wikipedia.org/wiki/extensible_metadata_platform “the adobe extensible metadata platform ( xmp ) is a standard, created by adobe systems inc. , for processing and storing standardized and proprietary information relating to the contents of a file. xmp standardizes the definition, creation, and processing of extensible metadata . serialized xmp can be embedded into a significant number of popular file formats, without breaking their readability by non-xmp-aware applications. embedding metadata avoids many problems that occur when metadata is stored separately. xmp is used in pdf , photography and photo editing applications. xmp can be used in several file formats such as pdf , jpeg , jpeg 2000 , jpeg xr , gif , png , html , tiff , adobe illustrator , psd , mp3 , mp4 , audio video interleave , wav , rf64 , audio interchange file format , postscript , encapsulated postscript , and proposed for djvu . in a typical edited jpeg file, xmp information is typically included alongside exif and iptc information interchange model data.” from http://singztechmusings.wordpress.com/2011/10/17/how-to-retrieveextract-metadata-information-from-audio-files-using-java-and-apache-tika-api/

f you want to create any of these projects listed below using Eclipse IDE, OSGi Application Project OSGi Bundle Project OSGi Composite Bundle Project OSGi Fragment Project Blueprint File you need to have IBM Rational Development Tools for OSGi Applications installed. Why do we need these tools? Create and edit OSGi bundles, composite bundles, bundle fragments, and applications. Import and export OSGi bundles, composite bundles, bundle fragments, and applications. Convert existing Java Enterprise Edition (Java EE) web, Java Persistence Application (JPA), plug-in, or simple Java projects into OSGi bundles. Edit OSGi bundle, application, and composite bundle manifest files. Create and edit OSGi blueprint configuration files. Edit OSGi blueprint binding configuration files. Diagnose and fix problems in your bundles and applications using validation and quick fixes. Eclipse Plugin Installation Before you install the tools, you must have the Eclipse Helios v3.6.2 package, Eclipse IDE for Java EE Developers installed. 1. Click on Help > Install New Software 2. Point to this update site – http://public.dhe.ibm.com/ibmdl/export/pub/software/rational/OSGiAppTools – and click on Add. 3. You’ll see a list of tools – OSGi Application Development Tools, OSGi Application Development Tools UI, OSGi context-sensitive help, OSGi Help documentation, Rational Development Tools for OSGi Applications help documentation. Select all those are listed (you can ignore help stuff) and go ahead with the installation. As of writing this, the development tools’ version is 1.0.3. Maven Integration If you want to manage any of these OSGi projects using Maven, you can right-click on it and select Maven > Enable Dependency Management. You need to have Maven Integration for Eclipse(m2e) installed for this which you can find in Eclipse Marketplace. If you use Maven, you can try using the plugins provided by Apache Felix project for bundling (building an OSGi bundle). More on this plugin @ http://felix.apache.org/site/apache-felix-maven-bundle-plugin-bnd.html, http://felix.apache.org/site/apache-felix-maven-osgi-plugin.html References: http://www.ibm.com/developerworks/rational/downloads/10/rationaldevtoolsforosgiapplications.html#download From http://singztechmusings.wordpress.com/2011/09/12/creating-osgi-projects-using-eclipse-ide-and-maven/

This blog comes as an answer to a question from a reader. Now, as a rule, I don’t do request blogs, but I thought that this sounded like an interesting topic. The question refers to my blog from July 11 on accessing request parameters using Spring 3 MVC and was “Do you know how to encode the uuid so it is not visible to the user for security concerns?”. The answer is “Yes”, but that would make for a very short blog. In this case I’m assuming that the question is talking about using RESTful URLs to maintain session state information between pages. REST is a subject that’s had many words written about it, but putting it in very simplistic terms I’m defining it as not storing session or state information on your web server between browser calls. Even more simply put, it means not using the HttpSession interface implementations. Not maintaining state on the server gives you at least two obvious benefits; the first being improved loading. This is because your server is not using memory for storing state data which may, or may not be needed. The second benefit is scaling. If you’re using a domain with multiple servers, then tying session state to one server is problematic as you can’t usually guarantee that subsequent sequential calls to the domain will be directed to the same physical server. There are various techniques use to solve this problem, including making everything form based and storing state information in hidden fields, using cookies, and manipulating URLs. This blog demonstrates the third technique of encrypting state data and tagging it onto the end of a URL, so that it’s passed back and forth between the browser and the server as a request parameter: http://www.mywebsite.com/thepageIwant?session=ThisIsEncodedData An example scenario for this technique would an e-commerce site. In this case, you’d select a couple of items and add them to your basket. You’d then login and review your basket and then perhaps modify the delivery address before proceeding to pay. In doing all this, you conceptually remain logged in to the server but, in the RESTful world, the server doesn’t remember you. This example uses three screens to demonstrate this idea. The first screen logs you in, the second contains a URL that includes some session data, and the third displays the session data for review. The screen shot above shows a simple login screen. Note that I’ve used a clear text password for demonstration purposes only. The second screen in this scenario is the one that contains the encrypted session information; namely the user name and password, which has both been displayed and glued on to the end of the href of an anchor tag. The last screen shows the decoded session information demonstrating that we’ve still got it and that it’s correct. Moving on to the actual Java code for this demo... The simple bean below I’ve called Session and it holds all the state data we want encoding. public class Session { private static final String SEP = "=/="; private String name; private String password; public Session() { // Blank } public Session(String combined) { String[] split = combined.split(SEP); name = split[0]; password = split[1]; } public String getName() { return name; } public void setName(String username) { this.name = username; } public String getPassword() { return password; } public void setPassword(String password) { this.password = password; } @Override public String toString() { return name + SEP + password; } } If you look at the code above, you’ll see that it contains two features that give it context in the application. The toString(...) returns a string that the application can encode and the single argument constructor can create a full bean from its argument. This works cohesively with the controller code, which is shown below... @Controller public class RewriteController { private static final String ENCODING_KEY = "ThisWillBeTheKey"; /** * Create the initial blank form */ @RequestMapping(value = "/loginform", method = RequestMethod.GET) public String getCreateForm(Model model) { model.addAttribute("Session", new Session()); return "login.page"; } /** * This is where you login... */ @RequestMapping(value = "/login", method = RequestMethod.POST) public String login(Session userDetails, Model model) throws UnsupportedEncodingException { byte[] encodedBytes = encodeURL(userDetails); String encodedURL = new String(Base64.encodeBase64(encodedBytes)); model.addAttribute("encodedURL", encodedURL); return "display.encoded.url"; } private byte[] encodeURL(Session userDetails) throws UnsupportedEncodingException { RC4Lite rc4 = getRC4Lite(); byte[] in = userDetails.toString().getBytes("UTF8"); byte[] out = new byte[in.length]; rc4.encrypt(in, 0, out, 0, in.length); return out; } private RC4Lite getRC4Lite() throws UnsupportedEncodingException { RC4Lite rc4 = new RC4Lite(); rc4.setKey(ENCODING_KEY.getBytes("UTF8")); return rc4; } /** * This is where you're still in the REST session and re-validate the user */ @RequestMapping(value = "/decode", method = RequestMethod.GET) public String someSessionMethod( @RequestParam(value = "session") String sessionParam, Model model) throws UnsupportedEncodingException { byte[] encodedBytes = Base64 .decodeBase64(sessionParam.getBytes("UTF8")); String decodedString = decodeURL(encodedBytes); Session session = new Session(decodedString); model.addAttribute(session); return "display.decoded.url"; } private String decodeURL(byte[] encodedBytes) throws UnsupportedEncodingException { RC4Lite rc4 = getRC4Lite(); byte[] out = new byte[encodedBytes.length]; rc4.decrypt(encodedBytes, 0, out, 0, encodedBytes.length); return new String(out); } } The controller class above contains three handler methods. The first handler method, getCreateForm(...), is fairly straight forward and simply adds a Session bean the model before displaying the login in form. The second handler method, login(...), is where all the action takes place. There are two parts to encoding the session data (the name and password). Part one is to encrypt the data and for that I’ve simply borrowed my RC4Lite class from my RC4 Encryption blog1. Once encrypted, the RC4 bytes will need converting into ASCII and for that I’ve used Apache’s Base64 class as described in my base64 blog. Converting to ASCII ensures that the characters are printable and don’t cause any problems. The string is then added to the model where it’s attached to the anchor tag as shown in the following JSP snippet: The final handler method, someSessionMethod(...) receives the input from the above link, decodes it using a reverse process to encoding It then creates a new Session object and puts that into the model, which is then displayed on the final page. Remember that this is only a sample, it just demonstrates the idea of encrypting session information and adding to a URL as a parameter. There are improvements you could make here, for example: using a interceptor to authenticate with the server upon each request before doing the business logic in the controller, or using a better encryption class, but in essence the principle applies to any webapp. There are also other techniques you can use for achieving secure web communications, so more on those later perhaps... 1 There are better ways of doing encryption than RC4, it's used here only for convenience in this demonstration. From http://www.captaindebug.com/2011/09/using-restful-urls-on-your-spring-3-mvc.html

Here’s how to do it in CXF proprietary way: // Create a list of SOAP headers List headersList = new ArrayList(); Header testHeader = new Header(new QName("uri:singz.ws.sample", "test"), "A custom header", new JAXBDataBinding(String.class)); headersList.add(testHeader); ((BindingProvider)proxy).getRequestContext().put(Header.HEADER_LIST, headersList); The headers in the list are streamed at the appropriate time to the wire according to the databinding object found in the Header object. This doesn’t require changes to WSDL or method signatures. It’s much faster as it doesn’t break streaming and the memory overhead is less. More on this @ http://cxf.apache.org/faq.html#FAQ-HowcanIaddsoapheaderstotherequest%2Fresponse%3F From http://singztechmusings.wordpress.com/2011/09/08/apache-cxf-how-to-add-custom-soap-headers-to-the-web-service-request/

Are you building a Java web service based on EJB3? Do you need to return a more significant message to your web service clients other that just the exception message or even worst the recurring javax.transaction.TransactionRolledbackException? Well if the answer is YES to the above questions then keep reading... The code in this article has been tested with JBoss 5.1.0 but it should (!?) work on other EJB containers as well Create a base application exception that will be extended by all the other exception, I will refer to it as MyApplicationBaseException . This exception contains a list of UserMessage, again a class I created with some messages and locale information You need to create a javax.xml.ws.handler.soap.SOAPHandler < SOAPMessageContext > implementation. Mine looks like this import java.util.Set; import javax.xml.bind.JAXBContext; import javax.xml.bind.JAXBException; import javax.xml.bind.Marshaller; import javax.xml.namespace.QName; import javax.xml.soap.SOAPException; import javax.xml.soap.SOAPFault; import javax.xml.soap.SOAPMessage; import javax.xml.ws.handler.MessageContext; import javax.xml.ws.handler.soap.SOAPHandler; import javax.xml.ws.handler.soap.SOAPMessageContext; import org.apache.commons.lang.exception.ExceptionUtils; public class SoapExceptionHandler implements SOAPHandler { private transient Logger logger = ServiceLogFactory.getLogger(SoapExceptionHandler.class); @Override public void close(MessageContext context) { } @Override public boolean handleFault(SOAPMessageContext context) { try { boolean outbound = (Boolean) context.get(MessageContext.MESSAGE_OUTBOUND_PROPERTY); if (outbound) { logger.info("Processing " + context + " for exceptions"); SOAPMessage msg = ((SOAPMessageContext) context).getMessage(); SOAPFault fault = msg.getSOAPBody().getFault(); // Retrives the exception from the context Exception ex = (Exception) context.get("exception"); if (ex != null) { // Add a fault to the body if not there already if (fault == null) { fault = msg.getSOAPBody().addFault(); } // Get my exception int indexOfType = ExceptionUtils.indexOfType(ex, MyApplicationBaseException.class); if (indexOfType != -1) { ex = (MyApplicationBaseException)ExceptionUtils.getThrowableList(ex).get(indexOfType); MyApplicationBaseException myEx = (AmsException) ex; fault.setFaultString(myEx.getMessage()); try { JAXBContext jaxContext = JAXBContext.newInstance(UserMessages.class); Marshaller marshaller = jaxContext.createMarshaller(); //Add the UserMessage xml as a fault detail. Detail interface extends Node marshaller.marshal(amsEx.getUserMessages(), fault.addDetail()); } catch (JAXBException e) { throw new RuntimeException("Can't marshall the user message ", e); } }else { logger.info("This is not an AmsException"); } }else { logger.warn("No exception found in the webServiceContext"); } } } catch (SOAPException e) { logger.warn("Error when trying to access the soap message", e); } return true; } @Override public boolean handleMessage(SOAPMessageContext context) { return true; } @Override public Set getHeaders() { return null; } } Now that you have the exception handler you need to register this SoapHandler with the EJB. To do that you'll need to create an Xml file in your class path and add an annotation to the EJB implementation class. The xml file : ExceptionHandler com.mycompany.utilities.ExceptionHandler and the EJB with annotation will be import javax.jws.HandlerChain; @Local(MyService.class) @Stateless @HandlerChain(file = "soapHandler.xml") @Interceptors( { MyApplicationInterceptor.class }) @SOAPBinding(style = SOAPBinding.Style.RPC) @WebService(endpointInterface = "com.mycompany.services.myservice", targetNamespace = "http://myservice.services.mycompany.com") public final class MyServiceImpl implements MyService { // service implementation } To make sure all my exceptions have proper messages and that the exception is set in the SOAPMessageContext I use an Interceptor to wrap all the service methods and transform any exception to an instance of MyApplicationException The interceptor has a single method @AroundInvoke private Object setException(InvocationContext ic) throws Exception { Object toReturn = null; try { toReturn = ic.proceed(); } catch (Exception e) { logger.error("Exception during the request processing.", e); //converts any exception to MyApplicationException e = MyApplicationExceptionHandler.getMyApplicationException(e); if (context != null && context.getMessageContext() != null) { context.getMessageContext().put("exception", e); } throw e; } return toReturn; } That's it! You're done. From http://www.devinprogress.info/2011/02/how-to-add-information-to-soap-fault.html

Memory is stored within the cache system in units know as cache lines. Cache lines are a power of 2 of contiguous bytes which are typically 32-256 in size. The most common cache line size is 64 bytes. False sharing is a term which applies when threads unwittingly impact the performance of each other while modifying independent variables sharing the same cache line. Write contention on cache lines is the single most limiting factor on achieving scalability for parallel threads of execution in an SMP system. I’ve heard false sharing described as the silent performance killer because it is far from obvious when looking at code. To achieve linear scalability with number of threads, we must ensure no two threads write to the same variable or cache line. Two threads writing to the same variable can be tracked down at a code level. To be able to know if independent variables share the same cache line we need to know the memory layout, or we can get a tool to tell us. Intel VTune is such a profiling tool. In this article I’ll explain how memory is laid out for Java objects and how we can pad out our cache lines to avoid false sharing. Figure 1. Figure 1. above illustrates the issue of false sharing. A thread running on core 1 wants to update variable X while a thread on core 2 wants to update variable Y. Unfortunately these two hot variables reside in the same cache line. Each thread will race for ownership of the cache line so they can update it. If core 1 gets ownership then the cache sub-system will need to invalidate the corresponding cache line for core 2. When Core 2 gets ownership and performs its update, then core 1 will be told to invalidate its copy of the cache line. This will ping pong back and forth via the L3 cache greatly impacting performance. The issue would be further exacerbated if competing cores are on different sockets and additionally have to cross the socket interconnect. Java Memory Layout For the Hotspot JVM, all objects have a 2-word header. First is the “mark” word which is made up of 24-bits for the hash code and 8-bits for flags such as the lock state, or it can be swapped for lock objects. The second is a reference to the class of the object. Arrays have an additional word for the size of the array. Every object is aligned to an 8-byte granularity boundary for performance. Therefore to be efficient when packing, the object fields are re-ordered from declaration order to the following order based on size in bytes: doubles (8) and longs (8) ints (4) and floats (4) shorts (2) and chars (2) booleans (1) and bytes (1) references (4/8) With this knowledge we can pad a cache line between any fields with 7 longs. Within the Disruptor we pad cache lines around the RingBuffer cursor and BatchEventProcessor sequences. To show the performance impact let’s take a few threads each updating their own independent counters. These counters will be volatile longs so the world can see their progress. public final class FalseSharing implements Runnable { public final static int NUM_THREADS = 4; // change public final static long ITERATIONS = 500L * 1000L * 1000L; private final int arrayIndex; private static VolatileLong[] longs = new VolatileLong[NUM_THREADS]; static { for (int i = 0; i < longs.length; i++) { longs[i] = new VolatileLong(); } } public FalseSharing(final int arrayIndex) { this.arrayIndex = arrayIndex; } public static void main(final String[] args) throws Exception { final long start = System.nanoTime(); runTest(); System.out.println("duration = " + (System.nanoTime() - start)); } private static void runTest() throws InterruptedException { Thread[] threads = new Thread[NUM_THREADS]; for (int i = 0; i < threads.length; i++) { threads[i] = new Thread(new FalseSharing(i)); } for (Thread t : threads) { t.start(); } for (Thread t : threads) { t.join(); } } public void run() { long i = ITERATIONS + 1; while (0 != --i) { longs[arrayIndex].value = i; } } public final static class VolatileLong { public volatile long value = 0L; public long p1, p2, p3, p4, p5, p6; // comment out } } Results Running the above code while ramping the number of threads and adding/removing the cache line padding, I get the results depicted in Figure 2. below. This is measuring the duration of test runs on my 4-core Nehalem at home. Figure 2. The impact of false sharing can clearly be seen by the increased execution time required to complete the test. Without the cache line contention we achieve near linear scale up with threads. This is not a perfect test because we cannot be sure where the VolatileLongs will be laid out in memory. They are independent objects. However experience shows that objects allocated at the same time tend to be co-located. So there you have it. False sharing can be a silent performance killer. From http://mechanical-sympathy.blogspot.com/2011/07/false-sharing.html

The integration framework Apache Camel already supports several important cloud services (see my overview article at http://www.kai-waehner.de/blog/2011/07/09/cloud-computing-heterogeneity-will-require-cloud-integration-apache-camel-is-already-prepared for more details). This article describes the combination of Apache Camel and the Amazon Web Services (AWS) interfaces of Simple Storage Service (S3), Simple Queue Service (SQS) and Simple Notification Service (SNS). Thus, The concept of Infrastructure as a Service (IaaS) is used to access messaging systems and data storage without any need for configuration. Registration to AWS and Setup of Camel First, you have to register to the Amazon Web Services (for free). Most AWS services include a free monthly quota, which is absolutely sufficient to play around and develop some simple applications. As its name states, AWS uses technology-independent web services. Besides, APIs for several different programming languages are available to ease development. By the way, Camel uses the AWS SDK for Java (http://aws.amazon.com/sdkforjava), of course. The documentation is detailed and easy to understand, including tutorials, screenshots and code examples . Hint 1: You should read the introductions to S3, SQS and SNS (go to http://aws.amazon.com and click on „products“) and play around with the AWS Management Console (http://aws.amazon.com/console) before you continue. This step is very easy and takes less than one hour. Then, you will have a much better understanding about AWS and where Camel can help you! Hint 2: It really helps to look at the source code of the camel-aws component, It helps you to understand how Camel uses the AWS Java API internally. If you want to write tests, you can do it the same way. In the past, I was afraid of looking at „complex“ source code of open source frameworks. But there is no need to be scared! The camel-aws component (and most other camel components) contain only of a few classes. Everything is easy to understand. It helps you to understand Camel internals, the AWS API, and to spot and solve errors due to exceptions in your code. In the meanwhile, the current Camel version 2.8 supports three AWS services: S3, SQS and SNS. All of them use similar concepts. Therefore, they are included in one single camel component: „camel-aws“. You have to add the libraries to your existing Camel project. As always, the simplest way is to use Maven and add the following dependency to the pom.xml: org.apache.camel camel-aws ${camel-version} Configuration of the Camel Endpoint The implementation and configuration of all three services is very similar. The URI looks like this (the code shows the SQS service): aws-sqs://queue-name[?options] There are two alternatives to configure your endpoint. Using Parameters The easy way is to use two paramters in the URI of your endpoint: „accessKey“ and „secretKey“ (you receive both after your AWS registration). “aws-sqs://unique-queue-name?accessKey=“INSERT_ME“&secretKey=INSERT_ME” Be aware of the following problem, which can result in a strange, non-speaking exception (thanks to Brendan Long): You’ll need to URL encode any +’s in your secret key (otherwise, they’ll be treated as spaces). + = %2B, so if your secretkey was “my+secret\key”, your Camel URL should have “secretKey=my%2Bsecret\key”. “Within the query string, the plus sign is reserved as shorthand notation for a space. Therefore, real plus signs must be encoded. This method was used to make query URIs easier to pass in systems which did not allow spaces.” Source: WC3 URI Recommendations Adding a configured AmazonClient to the Registry If you need to do more configuration (e.g. because your system is behind a firewall), you have to add an AmazonClient object to your registry. The following code shows an example using SQS, but SNS and S3 use exactly the same concept. @Override protected JndiRegistry createRegistry() throws Exception { JndiRegistry registry = super.createRegistry(); AWSCredentials awsCredentials = new BasicAWSCredentials(“INSERT_ME”, “INSERT_ME”); ClientConfiguration clientConfiguration = new ClientConfiguration(); clientConfiguration.setProxyHost(“http://myProxyHost”); clientConfiguration.setProxyPort(8080); AmazonSQSClient client = new AmazonSQSClient(awsCredentials, clientConfiguration); registry.bind(“amazonSQSClient”, client); return registry; } This example overwrites the createRegistry() method of a JUnit test (extending CamelTestSupport). You can also add this information to your runtime Camel application, of course. Apache Camel and the Simple Storage Service (S3) Simple Storage Service (S3) is a key-value-store. You can store small to very large data. The usage is very easy. You create buckets and put key-value data into these buckets. You can also create folders within buckets to organize your data. That’s it. You can monitor your buckets using the AWS Management Console – an intuitive GUI supporting most AWS services. The following example shows both alternatives for accessing the Amazon services (as described above): Paramenters and the AmazonClient. // Transfer data from your file inbox to the AWS S3 service from(“file:files/inbox”) // This is the key of your key-value data .setHeader(S3Constants.KEY, simple(“This is a static key”)) // Using parameters for accessing the AWS service .to(“aws-s3://camel-integration-bucket-mwea-kw?accessKey=INSERT_ME&secretKey=INSERT_ME&region=eu-west-1″); // Transfer data from the AWS S3 service to your file outbox from(“aws-s3://camel-integration-bucket-mwea-kw?amazonS3Client=#amazonS3Client&region=eu-wes”) .to(“file:files/outbox”); There are some additional parameters, for instance you can submit the desired AWS region or delete data after receiving it (see http://camel.apache.org/aws-s3.html and the corresponding SQS and SNS sites for more details about parameters and message headers). As you see in the code, you can use the AWS-S3 endpoint for producing and for consuming messages. Each bucket must be unique, thus you have to add some specific information such as your company to its name. Hint: If a bucket does not exist, Camel is creating it automatically (as the AWS API does). This concept is also used for SQS queues and SNS topics. Apache Camel and the Simple Queue Service (SQS) The Simple Queue Service (SQS) is similar to a JMS provider such as WebSphere MQ or ActiveMQ (but with some differences). You create queues and send messages to them. Consumers receive the messages. Contrary to most other AWS services, you cannot monitor queues by using the AWS management console directly. You have to use the service „Cloudwatch“ (http://aws.amazon.com/cloudwatch) and start an EC2 instance to monitor queues and its content. As you can see in the following code example, the syntax and concepts are almost the same as for the S3 service: from(“file:inbox”) .to(“aws-sqs://camel-integration-queue-mwea-kw?accessKey=INSERT_ME&secretKey=INSERT_ME”); from(“aws-sqs://camel-integration-queue-mwea-kw?amazonSQSClient=#amazonSQSClient”) .to(“file:outbox?fileName=sqs-${date:now:yyyy.MM.dd-hh:mm:ss:SS}”); Again, you can use the AWS-SQS endpoint for producing and for consuming messages. Each queue name must be unique. There exist two important differences to JMS (copy & paste from the AWS documentation): Q: How many times will I receive each message? Amazon SQS is engineered to provide “at least once” delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies. Q: Why are there separate ReceiveMessage and DeleteMessage operations? When Amazon SQS returns a message to you, that message stays in the queue, whether or not you actually received the message. You are responsible for deleting the message; the delete request acknowledges that you’re done processing the message. If you don’t delete the message, Amazon SQS will deliver it again on another receive request. Apache Camel and the Simple Notification Service (SNS) The Simple Notification Service (SNS) acts like JMS topics. You create a topic, consumers subscribe to the topic and then receive notifications. Several transport protocols are supported: HTTP(S), Email and SQS. Further interfaces will be added in the future, e.g. the Short Message Service (SMS) for mobile phones. Contrary to S3 and SQS, Camel only offers a producer endpoint for this AWS service. You can only create topics and send messages via Camel. The reason is simple: Camel already offers endpoints for consuming these messages: HTTP, Email and SQS are already available. There is one tradeoff: A consumer cannot subscribe to topics using Camel – at the moment. The AWS Management Console has to be used. A very interesting discussion can be read on the Camel JIRA issue regarding the following questions: Should Camel be able to subscribe to topics? Should the producer contain this feature or should there be a consumer? In my opinion, there should be a consumer which is able to subscribe to topics, otherwise Camel is missing a key part of the AWS SNS service! Please read the discussion and contribute your opinion: https://issues.apache.org/jira/browse/CAMEL-3476. Apache Camel is already ready for the Cloud Computing Era AWS offers many more services for the cloud. Probably, it does not make sense to integrate everyone into Camel, but more AWS services will be supported in the future. For instance, SimpleDB and the Relational Database Service (RDS) are already planned and make sende, too: http://camel.apache.org/aws.html. The conclusion is easy: Apache Camel is already ready for the cloud computing era. Several important cloud services are already supported. Cloud integration will become very important in the future. Thus, Camel is on a very good way. Hopefully, we will see more cloud components, soon. I will continue to write articles about other Camel cloud components (and new AWS addons, ouf course). For instance, a component for the Platform as a Service (PaaS) product Google App Engine (GAE) is already available. If you have any additional important information, questions or other feedback, please write a comment. Thank you in advance… Best regards, Kai Wähner (Twitter: @KaiWaehner) [Content from my Blog: Cloud Integration with Apache Camel and Amazon Web Services (AWS): S3, SQS and SNS]