So you're still interested in unit testing JavaScript (good). This post is an extension of my much more indepth first posting on how to unit test JavaScript using JS Test Driver. Please check it out here. Recap Last Posting In the last posting we successfully unit tested JavaScript using Maven and JsTest Driver. This allowes us to test JavaScript when on an environment that has a modern browser installed and can be run. Problem with typical CI environments So what happens when the test are passing on your local box, but you go to check in your code and the Continuous Integration (CI) server pukes on the new tests becasue there is no "screen" to run chrome or firefox? As of this posting, none of the top-tier browsers have a "headless" or an in-memory only browser window. There are alternatives to running JavaScript in a browser, such as rhino.js, env.js or HtmlUnit, however, these are just ports of browsers and the JavaScript and DOM representation are not 100% accurate which can lead to problems with your code when rendered in a client's browser. Approach What we need to do is to run JSTestDriver's browser in a Virtual X Framebuffer (Xvfb) which is possible on nearly all Linux based systems. The example below uses a Solaris version of Linux, however, Debian and RedHat linux distrubutions come with the simplified bash script to easily run an appliation in a virtual framebuffer. This solution was derived from one posted solution on the JS Test Driver wiki. The given example is also a full working example that is in use at my current client. Here is the quick list of what we will accomplish. Note, several of these steps are discussed in depth in the previous post and are not covered in depth here. Create a profile to run Js Unit-Tests Copy JsTestDriver library to a known location for Maven to use Copy JavaScript main and test files to known locations Use ANT to start JsTestDriver and pipe the screen into xvfb Here is a sample profile to use. You will need to adjust the properties at the top of the profile to match your system. ci-jstests /opt/swf/bin/firefox 1.3.2 /opt/X11R6/xvfb-run org.apache.maven.plugins maven-dependency-plugin 2.1 copy generate-resources copy com.google.jstestdriver jstestdriver ${js-test-driver.version} jar true jsTestDriver.jar ${project.build.directory}/jstestdriver false true maven-resources-plugin 2.4.3 copy-main-files generate-test-resources copy-resources ${project.build.directory}/test-classes/main-js src/main/webapp/scripts false copy-test-files generate-test-resources copy-resources ${project.build.directory}/test-classes/test-js src/test/webapp/scripts false org.apache.maven.plugins maven-antrun-plugin 1.6 test run Possible problems Although I cannot predict or fix all problems, I can share the one major problem I ran into with Solaris and the script used to fix that. In Solaris (and could happen to other distros) the xvfb-run script was not available and several of the other libraries did not exist. I first had to download the latest X libraries and place them in their appropriate locations on the CI server. Next, I had to re-engineer the xvfb-run script. Here is a copy of my script (NOTE: This is the solution for my server and this may not work for you) I created a script that contains: /usr/openwin/bin/Xvfb :1 screen 0 1280x1024x8 pixdepths 8 24 fbdir /tmp/.X11-vbf & From http://www.ensor.cc/2011/08/maven-javascript-unit-test-running-in.html

This is the sixth of a series of articles about setting up a secure RESTful Web Service using Spring 3.1 and Spring Security 3.1. A previous article introduced security in the context of a RESTful service, using form-based authentication. This article will focus on configuration of Basic and Digest authentication and on configuring both protocols for the same URI mapping of the API, using Spring Security 3.1. The REST with Spring series: Part 1 – Bootstrapping a web application with Spring 3.1 and Java based Configuration Part 2 – Building a RESTful Web Service with Spring 3.1 and Java based Configuration Part 3 – Securing a RESTful Web Service with Spring Security 3.1 Part 4 – RESTful Web Service Discoverability Part 5 – REST Service Discoverability with Spring Configuration of Basic Authentication In part 3 of the series, the Spring Security configuration was done using form based authentication, which is not really ideal for a RESTful service. To start setting up basic authentication, first we remove the old custom entry point and filter from the main security element: Note how support for basic authentication has been added with a single configuration line – – which handles the creation and wiring of both the BasicAuthenticationFilter and the BasicAuthenticationEntryPoint. Satisfying the stateless constraint – getting rid of sessions One of the main constraints of the RESTful architectural style is that the client-server communication is fully stateless, as the original dissertation reads: 5.1.3 Stateless We next add a constraint to the client-server interaction: communication must be stateless in nature, as in the client-stateless-server (CSS) style of Section 3.4.3 (Figure 5-3), such that each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client. The concept of Session on the server is one with a long history in Spring Security, and removing it entirely has been difficult until now, especially when configuration was done by using the namespace. However, Spring Security 3.1 augments the namespace configuration with a new stateless option for session creation, which effectively guarantees that no session will be created or used by Spring. What this new option does is completely removes all session related filters from the security filter chain, ensuring that authentication is performed for each request. Configuration of Digest Authentication Starting with the previous configuration, the filter and entry point necessary to set up digest authentication will be defined as beans. Then, the digest entry point will override the one created by behind the scenes. Finally, the custom digest filter will be introduced in the security filter chain using the after semantics of the security namespace to position it directly after the basic authentication filter. Unfortunately there is no support in the security namespace to automatically configure the digest authentication the way basic authentication can be configured with . Because of that, the necessary beans had to be defined and wired manually into the security configuration. Supporting both authentication protocols in the same RESTful service Basic or Digest authentication alone can be easily implemented in Spring Security 3.x; it is supporting both of them for the same RESTful web service, on the same URI mappings that introduces a new level of complexity into the configuration and testing of the service. Anonymous request With both basic and digest filters in the security chain, the way a anonymous request – a request containing no authentication credentials (Authorization HTTP header) – is processed by Spring Security is – the two authentication filters will find no credentials and will continue execution of the filter chain. Then, seeing how the request wasn’t authenticated, an AccessDeniedException is thrown and caught in the ExceptionTranslationFilter, which commences the digest entry point, prompting the client for credentials. The responsibilities of both the basic and digest filters are very narrow – they will continue to execute the security filter chain if they are unable to identify the type of authentication credentials in the request. It is because of this that Spring Security can have the flexibility to be configured with support for multiple authentication protocols on the same URI. When a request is made containing the correct authentication credentials – either basic or digest – that protocol will be rightly used. However, for an anonymous request, the client will get prompted only for digest authentication credentials. This is because the digest entry point is configured as the main and single entry point of the Spring Security chain; as such digest authentication can be considered the default. Request with authentication credentials A request with credentials for Basic authentication will be identified by the Authorization header starting with the prefix “Basic”. When processing such a request, the credentials will be decoded in the basic authentication filter and the request will be authorized. Similarly, a request with credentials for Digest authentication will use the prefix “Digest” for it’s Authorization header. Testing both scenarios The tests will consume the REST service by creating a new resource after authenticating with either basic or digest: @Test public void givenAuthenticatedByBasicAuth_whenAResourceIsCreated_then201IsReceived(){ // Given // When Response response = given() .auth().preemptive().basic( ADMIN_USERNAME, ADMIN_PASSWORD ) .contentType( HttpConstants.MIME_JSON ).body( new Foo( randomAlphabetic( 6 ) ) ) .post( this.paths.getFooURL() ); // Then assertThat( response.getStatusCode(), is( 201 ) ); } @Test public void givenAuthenticatedByDigestAuth_whenAResourceIsCreated_then201IsReceived(){ // Given // When Response response = given() .auth().digest( ADMIN_USERNAME, ADMIN_PASSWORD ) .contentType( HttpConstants.MIME_JSON ).body( new Foo( randomAlphabetic( 6 ) ) ) .post( this.paths.getFooURL() ); // Then assertThat( response.getStatusCode(), is( 201 ) ); } Note that the test using basic authentication adds credentials to the request preemptively, regardless if the server has challenged for authentication or not. This is to ensure that the server doesn’t need to challenge the client for credentials, because if it did, the challenge would be for Digest credentials, since that is the default. Conclusion This article covered the configuration and implementation of both Basic and Digest authentication for a RESTful service, using mostly Spring Security 3.0 namespace support as well as some new features added by Spring Security 3.1. In the next articles I will focus on OAuth authentication. In the meantime, check out the github project. From the REST with Spring series.

Develop your own API service for your PHP projects.

1. Overview This is the third of a series of articles about setting up a secure RESTful Web Service using Spring 3.1 and Spring Security 3.1 with Java based configuration. This article will focus on the security configuration using Spring Security 3.1, assuming some understanding of Spring Security basics and focusing on the specifics of securing the RESTful web service. The REST with Spring series: Part 1 – Bootstrapping a web application with Spring 3.1 and Java based Configuration Part 2 – Building a RESTful Web Service with Spring 3.1 and Java based Configuration Part 4 – RESTful Web Service Discoverability Part 5 – REST Service Discoverability with Spring Part 6 – Basic and Digest authentication for a RESTful Service with Spring Security 3.1 Part 7 – REST Pagination in Spring Part 8 – Authentication against a RESTful Service with Spring Security Part 9 – ETags for REST with Spring 2. Introducing Spring Security in the web.xml The architecture of Spring Security is based entirely on servlet filters and, as such, comes before Spring MVC in regards to the processing of HTTP requests. Keeping this in mind, to begin with, a filter needs to be declared in the web.xml of the application: springSecurityFilterChain org.springframework.web.filter.DelegatingFilterProxy springSecurityFilterChain /* The filter must necessarily be named ‘springSecurityFilterChain’ to match the default bean created by Spring Security in the container. Note that the defined filter is not the actual class implementing the security logic but a DelegatingFilterProxy with the purpose of delegating the Filter’s methods to an internal bean. This is done so that the target bean can still benefit from the Spring context lifecycle and flexibility. The URL pattern used to configure the Filter is /* even though the entire web service is mapped to /api/* so that the security configuration has the option to secure other possible mappings as well, if required. 3. The security configuration Most of the configuration is done using the security namespace – for this to be enabled, the schema locations must be defined and pointed to the new 3.1 versions. The namespace is designed so that it expresses the common uses of Spring Security while still providing hooks to the underlying beans. 3.1. The basics The element is the main container element for HTTP security configuration. In the current implementation, it only secured a single mapping: /api/admin/**. Note that the mapping is relative to the root context of the web application, not to the rest servlet; this is because the entire security configuration lives in the root Spring context and not in the child context of the servlet. 3.2. The entry point In a standard web application, the authentication process may be automatically triggered when the client tries to access a secured resource without being authenticated – this is usually done by redirecting to a login page so that the user can enter credentials. However, for a RESTful Web Service this behavior doesn’t make much sense – authentication should only be done by a request to the correct URI and all other requests should simply fail with a 401 UNAUTHORIZED status code if the user is not authenticated. Spring Security handles this automatic triggering of the authentication process with the concept of an entry point; the entry point is a required part of the configuration, and can be injected via the entry-point-ref attribute of the element. Keeping in mind that this functionality doesn’t make sense in the context of the RESTful web service, the new custom entry point is defined: @Component( "restAuthenticationEntryPoint" ) public final class RestAuthenticationEntryPoint implements AuthenticationEntryPoint{ @Override public final void commence ( HttpServletRequest request, HttpServletResponse response, AuthenticationException authException ) throws IOException{ response.sendError( HttpServletResponse.SC_UNAUTHORIZED, "Unauthorized" ); } } 3.3. The login There are multiple ways to do authentication for a RESTful Web Service – one of the default Spring Security provides is form login – which uses an authentication processing filter – org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter. Since the element doesn’t automatically create this particular filter by default, it needs to be explicitly specified in the configuration, using the element at the position FORM_LOGIN_FILTER; the only required dependency for this bean is the authentication manager. Note that for a standard web application, the auto-configattribute of the element is shorthand syntax for some useful security configuration. While this may be appropriate for some very simple configurations, it doesn’t fit and should not be used for a REST API. 3.4. Authentication should return 200 instead of 301 By default, form login will answer a successful authentication request with a 301 MOVED PERMANENTLY status code; this makes sense in the context of an actual login form which needs to redirect after login. For a RESTful web service however, the desired response for a successful authentication should be 200 OK. This is done by injecting a custom authentication success handler in the form login filter, to replace the default one. The new handler implements the exact same login as the default org.springframework.security.web.authentication.SavedRequestAwareAuthenticationSuccessHandler with one notable difference – the redirect logic is removed: // Use the DefaultSavedRequest URL // final String targetUrl = savedRequest.getRedirectUrl(); // this.logger.debug( "Redirecting to DefaultSavedRequest Url: " + targetUrl ); // this.getRedirectStrategy().sendRedirect( request, response, targetUrl ); 3.5. The authentication manager and provider The authentication process uses an in-memory provider to perform authentication – this is meant to simplify the configuration as a production implementation of these artifacts is outside the scope of this post. 4. Maven and other trouble In addition to the pom.xml from the first post, as well as the one from the second post, the Spring Security maven dependencies need to be added: org.springframework.security spring-security-web ${spring-security.version} org.springframework.security spring-security-config ${spring-security.version} org.springframework spring-tx ${spring.version} org.springframework spring-aop ${spring.version} 3.2.2.RELEASE 3.1.3.RELEASE Notice that the version of the security dependencies is no longer 3.1.0.BUILD-SNAPSHOT as for the standard Spring framework, but 3.1.0.CI-SNAPSHOT. What is more, the security artifacts define dependencies to the 3.0.x versions of Spring, more specifically spring-security-web depends on spring-aop and on spring-tx version 3.0.x instead of the expected 3.1.x. To understand why this is a problem, we need to understand how the Maven conflict resolution algorithm works – in case of conflict, Maven will chose which jar to include based on the distance between the particular dependency and the root of the tree. In our case, the conflicts are the spring-aop and spring-tx jars, appearing once with version 3.0.6 and once with 3.1.0. In the case of spring-aop, it appears once as a level 1 dependency of both spring-security-web and spring-security-config with version 3.0.6 , and once as a level 2 dependency of spring-webmvc with the version 3.1.0; since the 3.0.6 versioned jar is closer to the root, it will be the one chosen by the conflict resolution mechanism. Now that we understand why it is that Maven will deploy the 3.0.6 version of the jars with the application and not the intended 3.1.0 version, we need to address the issue. The solution is to add the two dependencies, with the intended 3.1.0 versions, directly into the pom – this will shorten the distance between them and the root to 0 and will force Maven to use them first. 5. Conclusion This post covered the basic security configuration and implementation for a RESTful service using Spring Security 3.1, discussing the web.xml, the security configuration, the HTTP status codes for the authentication process and the Maven resolution of the security artifacts. In the next articles I will focus on a Java based configuration for Spring Security, integration testing of the secure API using the rest-assured library and HTTP basic authentication. In the meantime, check out the github project. If you read this far, you should follow me on twitter here. Original at Securing a RESTful Web Service with Spring Security 3.1 from the REST with Spring series.

1. Overview This is the second of a series of posts about setting up a RESTful web service using Spring 3.1 with Java based configuration. The first post of the series focused on bootstrapping the web application; this post will focus on setting up REST in Spring, the Controller and HTTP response codes, configuration of payload marshalling and content negotiation. 2. Understanding REST in Spring The Spring framework supports 2 ways of creating RESTful services: using MVC with ModelAndView using HTTP message converters The ModelAndView approach is older and much better documented, but also more verbose and configuration heavy. It tries to shoehorn the REST paradigm into the old model, which is not without problems. The Spring team understood this and provided first-class REST support starting with Spring 3.0. The new approach, based on HttpMessageConverter and annotations, is much more lightweight and easy to implement. Configuration is minimal and it provides sensible defaults for what you would expect from a RESTful service. It is however newer and a a bit on the light side concerning documentation; what’s more, the Spring reference doesn’t go out of it’s way to make the distinction and the tradeoffs between the two approaches as clear as they should be. Nevertheless, this is the way RESTful services should be build after Spring 3.0. 3. The Java configuration @Configuration @EnableWebMvc public class WebConfig{ // } The new @EnableWebMvc annotation does a number of useful things – specifically, in the case of REST, it detect the existence of Jackson and JAXB 2 on the classpath and automatically creates and registers default JSON and XML converters. The functionality of the annotation is equivalent to the XML version: This is a shortcut, and though it may be useful in many situations, it’s not perfect. When more complex configuration is needed, remove the annotation and extend WebMvcConfigurationSupport directly. 4. Testing the Spring context Starting with Spring 3.1, we get first-class testing support for @Configuration classes: @RunWith( SpringJUnit4ClassRunner.class ) @ContextConfiguration( classes = { ApplicationConfig.class, PersistenceConfig.class },loader = AnnotationConfigContextLoader.class ) public class SpringTest{ @Test public void whenSpringContextIsInstantiated_thenNoExceptions(){ // When } } The Java configuration classes are simply specified with the @ContextConfiguration annotation and the new AnnotationConfigContextLoader loads the bean definitions from the @Configuration classes. Notice that the WebConfig configuration class was not included in the test because it needs to run in a servlet context, which is not provided. 5. The Controller The @Controller is the central artifact in the entire Web Tier of the RESTful API. For the purpose of this post, the controller is modeling a simple REST resource – Foo: @Controller class FooController{ @Autowired IFooService service; @RequestMapping( value = "foo",method = RequestMethod.GET ) @ResponseBody public List< Foo > getAll(){ return this.service.getAll(); } @RequestMapping( value = "foo/{id}",method = RequestMethod.GET ) @ResponseBody public Foo get( @PathVariable( "id" ) Long id ){ return RestPreconditions.checkNotNull( this.service.getById( id ) ); } @RequestMapping( value = "foo",method = RequestMethod.POST ) @ResponseStatus( HttpStatus.CREATED ) @ResponseBody public Long create( @RequestBody Foo entity ){ RestPreconditions.checkNotNullFromRequest( entity ); return this.service.create( entity ); } @RequestMapping( value = "foo",method = RequestMethod.PUT ) @ResponseStatus( HttpStatus.OK ) public void update( @RequestBody Foo entity ){ RestPreconditions.checkNotNullFromRequest( entity ); RestPreconditions.checkNotNull( this.service.getById( entity.getId() ) ); this.service.update( entity ); } @RequestMapping( value = "foo/{id}",method = RequestMethod.DELETE ) @ResponseStatus( HttpStatus.OK ) public void delete( @PathVariable( "id" ) Long id ){ this.service.deleteById( id ); } } The Controller implementation is non-public – this is because there is no need for it to be. Usually the controller is the last in the chain of dependencies – it receives HTTP requests from the Spring front controller (the DispathcerServlet) and simply delegate them forward to a service layer. If there is no use case where the controller has to be injected or manipulated through a direct reference, then I prefer not to declare it as public. The request mappings are straightforward – as with any Spring controller, the actual value of the mapping as well as the HTTP method are used to determine the target method for the request. @RequestBody will bind the parameters of the method to the body of the HTTP request, whereas @ResponseBody does the same for the response and return type. They also ensure that the resource will be marshalled and unmarshalled using the correct HTTP converter. Content negotiation will take place to choose which one of the active converters will be used, based mostly on the Accept header, although other HTTP headers may be used to determine the representation as well. 6. Mapping the HTTP response codes The status codes of the HTTP response are one of the most important parts of the REST service, and the subject can quickly become very complex. Getting these right can be what makes or breaks the service. 6.1. Unmapped requests If Spring MVC receives a request which doesn’t have a mapping, it considers the request not to be allowed and returns a 405 METHOD NOT ALLOWED back to the client. It is also good practice to include the Allow HTTP header when returning a 405 to the client, in order to specify which operations are allowed. This is the standard behavior of Spring MVC and does not require any additional configuration. 6.2. Valid, mapped requests For any request that does have a mapping, Spring MVC considers the request valid and responds with 200 OK if no other status code is specified otherwise. It is because of this that controller declares different @ResponseStatus for the create, update and delete actions but not for get, which should indeed return the default 200 OK. 6.3. Client error In case of a client error, custom exceptions are defined and mapped to the appropriate error codes. Simply throwing these exceptions from any of the layers of the web tier will ensure Spring maps the corresponding status code on the HTTP response. @ResponseStatus( value = HttpStatus.BAD_REQUEST ) public class BadRequestException extends RuntimeException{ // } @ResponseStatus( value = HttpStatus.NOT_FOUND ) public class ResourceNotFoundException extends RuntimeException{ // } These exceptions are part of the REST API and, as such, should only be used in the appropriate layers corresponding to REST; if for instance a DAO/DAL layer exist, it should not use the exceptions directly. Note also that these are not checked exceptions but runtime exceptions – in line with Spring practices and idioms. 6.4. Using @ExceptionHandler Another option to map custom exceptions on specific status codes is to use the @ExceptionHandler annotation in the controller. The problem with that approach is that the annotation only applies to the controller in which it is defined, not to the entire Spring Container, which means that it needs to be declared in each controller individually. This quickly becomes cumbersome, especially in more complex applications which many controllers. There are a few JIRA issues opened with Spring at this time to handle this and other related limitations: SPR-8124, SPR-7278, SPR-8406. 7. Additional Maven dependencies In addition to the pom.xml from the first post, two dependencies need to be added: org.codehaus.jackson jackson-mapper-asl ${jackson-mapper-asl.version} runtime javax.xml.bind jaxb-api ${jaxb-api.version} runtime 1.9.12 2.2.4 These are the libraries used to convert the representation of the REST resource to either JSON or XML. 8. Conclusion This post covered the configuration and implementation of a RESTful service using Spring 3.1 and Java based configuration, discussing HTTP response codes, basic content negotiation and marshaling. In the next articles of the series I will focus on discoverability of the API, advanced content negotiation and working with additional representations of a resource. In the meantime, check out the github project.

Version 2.3 of the Java servlet spec introduced the concept of filters. According to the documentation from Oracle’s site: “A filter dynamically intercepts requests and responses to transform or use the information contained in the requests or responses”. Today I’ll show you how to build a simple filter to intercept the response HTTP response code using annotations introduced in the Servlet 3.0 specification. With NetBeans IDE 7 create a new Maven Java Web Application called: Intercept Delete the index.jsp file under the Web Pages folder. Right-click on the project and add a new servlet called: MainServlet Since we are using the new Servlet 3 annotations we don’t need to set a whole lot of properties. Maven generates a decent MainServlet.java file for us, I just removed the comments for the output. My file looks like this: package com.giantflyingsaucer.intercept; import java.io.IOException; import java.io.PrintWriter; import javax.servlet.ServletException; import javax.servlet.annotation.WebServlet; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; @WebServlet(name = "MainServlet", urlPatterns = {"/"}) public class MainServlet extends HttpServlet { protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setContentType("text/html;charset=UTF-8"); PrintWriter out = response.getWriter(); try { out.println(""); out.println(""); out.println(""); out.println(""); out.println(""); out.println("Servlet MainServlet"); out.println(""); out.println(""); } finally { out.close(); } } // /** * Handles the HTTP GET method. * @param request servlet request * @param response servlet response * @throws ServletException if a servlet-specific error occurs * @throws IOException if an I/O error occurs */ @Override protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { processRequest(request, response); } /** * Handles the HTTP POST method. * @param request servlet request * @param response servlet response * @throws ServletException if a servlet-specific error occurs * @throws IOException if an I/O error occurs */ @Override protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { processRequest(request, response); } /** * Returns a short description of the servlet. * @return a String containing servlet description */ @Override public String getServletInfo() { return "Short description"; }// } Right-click on the project and add a Filter called: InterceptFilter We will add the following two lines to the doFilter method. HttpServletResponse hsr = (HttpServletResponse) response; System.out.println("HTTP Status: " + hsr.getStatus()); My doFilter method looks like this: @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { if (debug) { log("InterceptFilter:doFilter()"); } doBeforeProcessing(request, response); HttpServletResponse hsr = (HttpServletResponse) response; System.out.println("HTTP Status: " + hsr.getStatus()); Throwable problem = null; try { chain.doFilter(request, response); } catch (Throwable t) { problem = t; t.printStackTrace(); } doAfterProcessing(request, response); if (problem != null) { if (problem instanceof ServletException) { throw (ServletException) problem; } if (problem instanceof IOException) { throw (IOException) problem; } sendProcessingError(problem, response); } } Clean and Build the project and deploy it to Apache Tomcat. Access the URL with a browser and take a look at your catalina.out file and you should see the HTTP response code. Note: You shouldn’t need to do any changes to the web.xml file for this project to work. From http://www.giantflyingsaucer.com/blog/?p=3279

OpenStreetMap is a global project with an aim of collaboratively collecting map data, and today Ken Guest has submitted his PHP package for communitcating with the OSM API to the public and the PEAR PEPr review process: So over the last while, I’ve been working on a PHP package imaginatively named Services_Openstreetmap, for interacting with the openstreetmap API. I initially needed it so I could search for certain POIs and tabulate the results; it’s now also capable of adding data to the openstreetmap database – nodes and other elements can be created, updated and so on. It will even access the details of the user that is being used to modify that data, which is one difference between it and the other single purpose OSM frameworks. --Ken Guest You can view the submission here, and you should definitely take a look at openstreetmap.org if you haven't already. Good news for PHP developers looking to use this project more heavily in their applications.

i guess, i’m writing this post after a long time. this time, i’m writing about apache tika api that a friend of mine and i tried out to extract/retrieve metadata information from audio files supported by it – .mp3, .aiff, .au, .midi, .wav. to make it clear, here’s a screenshot of the information shown by windows vista about an audio file: we wanted to extract this using java and with googling, found that apache tika would help. we needed this metadata to index audio files for it to be searchable in a search application that we’re building using apache lucene . here’s a sample java program that extracts metadata from an mp3 file: package singz.samples.search.audio.metadata; import java.io.file; import java.io.fileinputstream; import java.io.filenotfoundexception; import java.io.ioexception; import java.io.inputstream; import org.apache.tika.exception.tikaexception; import org.apache.tika.metadata.metadata; import org.apache.tika.parser.parsecontext; import org.apache.tika.parser.parser; import org.apache.tika.parser.mp3.mp3parser; import org.xml.sax.contenthandler; import org.xml.sax.saxexception; import org.xml.sax.helpers.defaulthandler; /** * @author singaram subramanian * extract metadata of an audio file using apache tika api * */ public class audiometadataextractordemo { public static void main(string[] args) { // this audio file has metadata embedded in xmp (extensible metadata platform) standard // created by adobe systems inc. xmp standardizes the definition, creation, and // processing of extensible metadata. string audiofileloc = "c:\\pop\\backstreetboys_showmethemeaningofbeinglonely.mp3"; try { inputstream input = new fileinputstream(new file(audiofileloc)); contenthandler handler = new defaulthandler(); metadata metadata = new metadata(); parser parser = new mp3parser(); parsecontext parsectx = new parsecontext(); parser.parse(input, handler, metadata, parsectx); input.close(); // list all metadata string[] metadatanames = metadata.names(); for(string name : metadatanames){ system.out.println(name + ": " + metadata.get(name)); } // retrieve the necessary info from metadata // names - title, xmpdm:artist etc. - mentioned below may differ based // on the standard used for processing and storing standardized and/or // proprietary information relating to the contents of a file. system.out.println("title: " + metadata.get("title")); system.out.println("artists: " + metadata.get("xmpdm:artist")); system.out.println("genre: " + metadata.get("xmpdm:genre")); } catch (filenotfoundexception e) { e.printstacktrace(); } catch (ioexception e) { e.printstacktrace(); } catch (saxexception e) { e.printstacktrace(); } catch (tikaexception e) { e.printstacktrace(); } } } maven pom xml 4.0.0 singz.samples.search.audio audiometadataextractor 0.0.1 jar audiometadataextractor http://maven.apache.org utf-8 org.apache.tika tika-core 0.10 org.apache.tika tika-parsers 0.10 output xmpdm:releasedate: 2001 xmpdm:audiochanneltype: stereo xmpdm:album: top 100 pop author: backstreet boys xmpdm:artist: backstreet boys channels: 2 xmpdm:audiosamplerate: 44100 xmpdm:logcomment: eng xmpdm:tracknumber: 04 version: mpeg 3 layer iii version 1 xmpdm:composer: null xmpdm:audiocompressor: mp3 title: show me the meaning of being lonely samplerate: 44100 xmpdm:genre: pop content-type: audio/mpeg title: show me the meaning of being lonely artists: backstreet boys genre: pop about apache tika http://tika.apache.org/index.html “the apache tika™ toolkit detects and extracts metadata and structured text content from various documents using existing parser libraries.” http://www.lucidimagination.com/devzone/technical-articles/content-extraction-tika#article.tika “apache tika is a content type detection and content extraction framework. tika provides a general application programming interface that can be used to detect the content type of a document and also parse textual content and metadata from several document formats. tika does not try to understand the full variety of different document formats by itself but instead delegates the real work to various existing parser libraries such as apache poi for microsoft formats, pdfbox for adobe pdf, neko html for html etc. the grand idea behind tika is that it offers a generic interface for parsing multiple formats. the tika api hides the technical differences of the various parser implementations. this means that you don’t have to learn and consume one api for every format you use but can instead use a single api – the tika api. internally tika usually delegates the parsing work to existing parsing libraries and adapts the parse result so that client applications can easily manage variety of formats. tika aims to be efficient in using available resources (mainly ram) while parsing. the tika api is stream oriented so that the parsed source document does not need to be loaded into memory all at once but only as it is needed. ultimately, however, the amount of resources consumed is mandated by the parser libraries that tika uses. at the time of writing this, tika supports directly around 30 document formats. see list of supported document formats . the list of supported document formats is not limited by tika in any way. in the simplest case you can add support for new document formats by implementing a thin adapter that that implements the parser interface for the new document format.” about xmp standard http://en.wikipedia.org/wiki/extensible_metadata_platform “the adobe extensible metadata platform ( xmp ) is a standard, created by adobe systems inc. , for processing and storing standardized and proprietary information relating to the contents of a file. xmp standardizes the definition, creation, and processing of extensible metadata . serialized xmp can be embedded into a significant number of popular file formats, without breaking their readability by non-xmp-aware applications. embedding metadata avoids many problems that occur when metadata is stored separately. xmp is used in pdf , photography and photo editing applications. xmp can be used in several file formats such as pdf , jpeg , jpeg 2000 , jpeg xr , gif , png , html , tiff , adobe illustrator , psd , mp3 , mp4 , audio video interleave , wav , rf64 , audio interchange file format , postscript , encapsulated postscript , and proposed for djvu . in a typical edited jpeg file, xmp information is typically included alongside exif and iptc information interchange model data.” from http://singztechmusings.wordpress.com/2011/10/17/how-to-retrieveextract-metadata-information-from-audio-files-using-java-and-apache-tika-api/

f you want to create any of these projects listed below using Eclipse IDE, OSGi Application Project OSGi Bundle Project OSGi Composite Bundle Project OSGi Fragment Project Blueprint File you need to have IBM Rational Development Tools for OSGi Applications installed. Why do we need these tools? Create and edit OSGi bundles, composite bundles, bundle fragments, and applications. Import and export OSGi bundles, composite bundles, bundle fragments, and applications. Convert existing Java Enterprise Edition (Java EE) web, Java Persistence Application (JPA), plug-in, or simple Java projects into OSGi bundles. Edit OSGi bundle, application, and composite bundle manifest files. Create and edit OSGi blueprint configuration files. Edit OSGi blueprint binding configuration files. Diagnose and fix problems in your bundles and applications using validation and quick fixes. Eclipse Plugin Installation Before you install the tools, you must have the Eclipse Helios v3.6.2 package, Eclipse IDE for Java EE Developers installed. 1. Click on Help > Install New Software 2. Point to this update site – http://public.dhe.ibm.com/ibmdl/export/pub/software/rational/OSGiAppTools – and click on Add. 3. You’ll see a list of tools – OSGi Application Development Tools, OSGi Application Development Tools UI, OSGi context-sensitive help, OSGi Help documentation, Rational Development Tools for OSGi Applications help documentation. Select all those are listed (you can ignore help stuff) and go ahead with the installation. As of writing this, the development tools’ version is 1.0.3. Maven Integration If you want to manage any of these OSGi projects using Maven, you can right-click on it and select Maven > Enable Dependency Management. You need to have Maven Integration for Eclipse(m2e) installed for this which you can find in Eclipse Marketplace. If you use Maven, you can try using the plugins provided by Apache Felix project for bundling (building an OSGi bundle). More on this plugin @ http://felix.apache.org/site/apache-felix-maven-bundle-plugin-bnd.html, http://felix.apache.org/site/apache-felix-maven-osgi-plugin.html References: http://www.ibm.com/developerworks/rational/downloads/10/rationaldevtoolsforosgiapplications.html#download From http://singztechmusings.wordpress.com/2011/09/12/creating-osgi-projects-using-eclipse-ide-and-maven/

This blog comes as an answer to a question from a reader. Now, as a rule, I don’t do request blogs, but I thought that this sounded like an interesting topic. The question refers to my blog from July 11 on accessing request parameters using Spring 3 MVC and was “Do you know how to encode the uuid so it is not visible to the user for security concerns?”. The answer is “Yes”, but that would make for a very short blog. In this case I’m assuming that the question is talking about using RESTful URLs to maintain session state information between pages. REST is a subject that’s had many words written about it, but putting it in very simplistic terms I’m defining it as not storing session or state information on your web server between browser calls. Even more simply put, it means not using the HttpSession interface implementations. Not maintaining state on the server gives you at least two obvious benefits; the first being improved loading. This is because your server is not using memory for storing state data which may, or may not be needed. The second benefit is scaling. If you’re using a domain with multiple servers, then tying session state to one server is problematic as you can’t usually guarantee that subsequent sequential calls to the domain will be directed to the same physical server. There are various techniques use to solve this problem, including making everything form based and storing state information in hidden fields, using cookies, and manipulating URLs. This blog demonstrates the third technique of encrypting state data and tagging it onto the end of a URL, so that it’s passed back and forth between the browser and the server as a request parameter: http://www.mywebsite.com/thepageIwant?session=ThisIsEncodedData An example scenario for this technique would an e-commerce site. In this case, you’d select a couple of items and add them to your basket. You’d then login and review your basket and then perhaps modify the delivery address before proceeding to pay. In doing all this, you conceptually remain logged in to the server but, in the RESTful world, the server doesn’t remember you. This example uses three screens to demonstrate this idea. The first screen logs you in, the second contains a URL that includes some session data, and the third displays the session data for review. The screen shot above shows a simple login screen. Note that I’ve used a clear text password for demonstration purposes only. The second screen in this scenario is the one that contains the encrypted session information; namely the user name and password, which has both been displayed and glued on to the end of the href of an anchor tag. The last screen shows the decoded session information demonstrating that we’ve still got it and that it’s correct. Moving on to the actual Java code for this demo... The simple bean below I’ve called Session and it holds all the state data we want encoding. public class Session { private static final String SEP = "=/="; private String name; private String password; public Session() { // Blank } public Session(String combined) { String[] split = combined.split(SEP); name = split[0]; password = split[1]; } public String getName() { return name; } public void setName(String username) { this.name = username; } public String getPassword() { return password; } public void setPassword(String password) { this.password = password; } @Override public String toString() { return name + SEP + password; } } If you look at the code above, you’ll see that it contains two features that give it context in the application. The toString(...) returns a string that the application can encode and the single argument constructor can create a full bean from its argument. This works cohesively with the controller code, which is shown below... @Controller public class RewriteController { private static final String ENCODING_KEY = "ThisWillBeTheKey"; /** * Create the initial blank form */ @RequestMapping(value = "/loginform", method = RequestMethod.GET) public String getCreateForm(Model model) { model.addAttribute("Session", new Session()); return "login.page"; } /** * This is where you login... */ @RequestMapping(value = "/login", method = RequestMethod.POST) public String login(Session userDetails, Model model) throws UnsupportedEncodingException { byte[] encodedBytes = encodeURL(userDetails); String encodedURL = new String(Base64.encodeBase64(encodedBytes)); model.addAttribute("encodedURL", encodedURL); return "display.encoded.url"; } private byte[] encodeURL(Session userDetails) throws UnsupportedEncodingException { RC4Lite rc4 = getRC4Lite(); byte[] in = userDetails.toString().getBytes("UTF8"); byte[] out = new byte[in.length]; rc4.encrypt(in, 0, out, 0, in.length); return out; } private RC4Lite getRC4Lite() throws UnsupportedEncodingException { RC4Lite rc4 = new RC4Lite(); rc4.setKey(ENCODING_KEY.getBytes("UTF8")); return rc4; } /** * This is where you're still in the REST session and re-validate the user */ @RequestMapping(value = "/decode", method = RequestMethod.GET) public String someSessionMethod( @RequestParam(value = "session") String sessionParam, Model model) throws UnsupportedEncodingException { byte[] encodedBytes = Base64 .decodeBase64(sessionParam.getBytes("UTF8")); String decodedString = decodeURL(encodedBytes); Session session = new Session(decodedString); model.addAttribute(session); return "display.decoded.url"; } private String decodeURL(byte[] encodedBytes) throws UnsupportedEncodingException { RC4Lite rc4 = getRC4Lite(); byte[] out = new byte[encodedBytes.length]; rc4.decrypt(encodedBytes, 0, out, 0, encodedBytes.length); return new String(out); } } The controller class above contains three handler methods. The first handler method, getCreateForm(...), is fairly straight forward and simply adds a Session bean the model before displaying the login in form. The second handler method, login(...), is where all the action takes place. There are two parts to encoding the session data (the name and password). Part one is to encrypt the data and for that I’ve simply borrowed my RC4Lite class from my RC4 Encryption blog1. Once encrypted, the RC4 bytes will need converting into ASCII and for that I’ve used Apache’s Base64 class as described in my base64 blog. Converting to ASCII ensures that the characters are printable and don’t cause any problems. The string is then added to the model where it’s attached to the anchor tag as shown in the following JSP snippet: The final handler method, someSessionMethod(...) receives the input from the above link, decodes it using a reverse process to encoding It then creates a new Session object and puts that into the model, which is then displayed on the final page. Remember that this is only a sample, it just demonstrates the idea of encrypting session information and adding to a URL as a parameter. There are improvements you could make here, for example: using a interceptor to authenticate with the server upon each request before doing the business logic in the controller, or using a better encryption class, but in essence the principle applies to any webapp. There are also other techniques you can use for achieving secure web communications, so more on those later perhaps... 1 There are better ways of doing encryption than RC4, it's used here only for convenience in this demonstration. From http://www.captaindebug.com/2011/09/using-restful-urls-on-your-spring-3-mvc.html

Here’s how to do it in CXF proprietary way: // Create a list of SOAP headers List headersList = new ArrayList(); Header testHeader = new Header(new QName("uri:singz.ws.sample", "test"), "A custom header", new JAXBDataBinding(String.class)); headersList.add(testHeader); ((BindingProvider)proxy).getRequestContext().put(Header.HEADER_LIST, headersList); The headers in the list are streamed at the appropriate time to the wire according to the databinding object found in the Header object. This doesn’t require changes to WSDL or method signatures. It’s much faster as it doesn’t break streaming and the memory overhead is less. More on this @ http://cxf.apache.org/faq.html#FAQ-HowcanIaddsoapheaderstotherequest%2Fresponse%3F From http://singztechmusings.wordpress.com/2011/09/08/apache-cxf-how-to-add-custom-soap-headers-to-the-web-service-request/

Are you building a Java web service based on EJB3? Do you need to return a more significant message to your web service clients other that just the exception message or even worst the recurring javax.transaction.TransactionRolledbackException? Well if the answer is YES to the above questions then keep reading... The code in this article has been tested with JBoss 5.1.0 but it should (!?) work on other EJB containers as well Create a base application exception that will be extended by all the other exception, I will refer to it as MyApplicationBaseException . This exception contains a list of UserMessage, again a class I created with some messages and locale information You need to create a javax.xml.ws.handler.soap.SOAPHandler < SOAPMessageContext > implementation. Mine looks like this import java.util.Set; import javax.xml.bind.JAXBContext; import javax.xml.bind.JAXBException; import javax.xml.bind.Marshaller; import javax.xml.namespace.QName; import javax.xml.soap.SOAPException; import javax.xml.soap.SOAPFault; import javax.xml.soap.SOAPMessage; import javax.xml.ws.handler.MessageContext; import javax.xml.ws.handler.soap.SOAPHandler; import javax.xml.ws.handler.soap.SOAPMessageContext; import org.apache.commons.lang.exception.ExceptionUtils; public class SoapExceptionHandler implements SOAPHandler { private transient Logger logger = ServiceLogFactory.getLogger(SoapExceptionHandler.class); @Override public void close(MessageContext context) { } @Override public boolean handleFault(SOAPMessageContext context) { try { boolean outbound = (Boolean) context.get(MessageContext.MESSAGE_OUTBOUND_PROPERTY); if (outbound) { logger.info("Processing " + context + " for exceptions"); SOAPMessage msg = ((SOAPMessageContext) context).getMessage(); SOAPFault fault = msg.getSOAPBody().getFault(); // Retrives the exception from the context Exception ex = (Exception) context.get("exception"); if (ex != null) { // Add a fault to the body if not there already if (fault == null) { fault = msg.getSOAPBody().addFault(); } // Get my exception int indexOfType = ExceptionUtils.indexOfType(ex, MyApplicationBaseException.class); if (indexOfType != -1) { ex = (MyApplicationBaseException)ExceptionUtils.getThrowableList(ex).get(indexOfType); MyApplicationBaseException myEx = (AmsException) ex; fault.setFaultString(myEx.getMessage()); try { JAXBContext jaxContext = JAXBContext.newInstance(UserMessages.class); Marshaller marshaller = jaxContext.createMarshaller(); //Add the UserMessage xml as a fault detail. Detail interface extends Node marshaller.marshal(amsEx.getUserMessages(), fault.addDetail()); } catch (JAXBException e) { throw new RuntimeException("Can't marshall the user message ", e); } }else { logger.info("This is not an AmsException"); } }else { logger.warn("No exception found in the webServiceContext"); } } } catch (SOAPException e) { logger.warn("Error when trying to access the soap message", e); } return true; } @Override public boolean handleMessage(SOAPMessageContext context) { return true; } @Override public Set getHeaders() { return null; } } Now that you have the exception handler you need to register this SoapHandler with the EJB. To do that you'll need to create an Xml file in your class path and add an annotation to the EJB implementation class. The xml file : ExceptionHandler com.mycompany.utilities.ExceptionHandler and the EJB with annotation will be import javax.jws.HandlerChain; @Local(MyService.class) @Stateless @HandlerChain(file = "soapHandler.xml") @Interceptors( { MyApplicationInterceptor.class }) @SOAPBinding(style = SOAPBinding.Style.RPC) @WebService(endpointInterface = "com.mycompany.services.myservice", targetNamespace = "http://myservice.services.mycompany.com") public final class MyServiceImpl implements MyService { // service implementation } To make sure all my exceptions have proper messages and that the exception is set in the SOAPMessageContext I use an Interceptor to wrap all the service methods and transform any exception to an instance of MyApplicationException The interceptor has a single method @AroundInvoke private Object setException(InvocationContext ic) throws Exception { Object toReturn = null; try { toReturn = ic.proceed(); } catch (Exception e) { logger.error("Exception during the request processing.", e); //converts any exception to MyApplicationException e = MyApplicationExceptionHandler.getMyApplicationException(e); if (context != null && context.getMessageContext() != null) { context.getMessageContext().put("exception", e); } throw e; } return toReturn; } That's it! You're done. From http://www.devinprogress.info/2011/02/how-to-add-information-to-soap-fault.html

Memory is stored within the cache system in units know as cache lines. Cache lines are a power of 2 of contiguous bytes which are typically 32-256 in size. The most common cache line size is 64 bytes. False sharing is a term which applies when threads unwittingly impact the performance of each other while modifying independent variables sharing the same cache line. Write contention on cache lines is the single most limiting factor on achieving scalability for parallel threads of execution in an SMP system. I’ve heard false sharing described as the silent performance killer because it is far from obvious when looking at code. To achieve linear scalability with number of threads, we must ensure no two threads write to the same variable or cache line. Two threads writing to the same variable can be tracked down at a code level. To be able to know if independent variables share the same cache line we need to know the memory layout, or we can get a tool to tell us. Intel VTune is such a profiling tool. In this article I’ll explain how memory is laid out for Java objects and how we can pad out our cache lines to avoid false sharing. Figure 1. Figure 1. above illustrates the issue of false sharing. A thread running on core 1 wants to update variable X while a thread on core 2 wants to update variable Y. Unfortunately these two hot variables reside in the same cache line. Each thread will race for ownership of the cache line so they can update it. If core 1 gets ownership then the cache sub-system will need to invalidate the corresponding cache line for core 2. When Core 2 gets ownership and performs its update, then core 1 will be told to invalidate its copy of the cache line. This will ping pong back and forth via the L3 cache greatly impacting performance. The issue would be further exacerbated if competing cores are on different sockets and additionally have to cross the socket interconnect. Java Memory Layout For the Hotspot JVM, all objects have a 2-word header. First is the “mark” word which is made up of 24-bits for the hash code and 8-bits for flags such as the lock state, or it can be swapped for lock objects. The second is a reference to the class of the object. Arrays have an additional word for the size of the array. Every object is aligned to an 8-byte granularity boundary for performance. Therefore to be efficient when packing, the object fields are re-ordered from declaration order to the following order based on size in bytes: doubles (8) and longs (8) ints (4) and floats (4) shorts (2) and chars (2) booleans (1) and bytes (1) references (4/8) With this knowledge we can pad a cache line between any fields with 7 longs. Within the Disruptor we pad cache lines around the RingBuffer cursor and BatchEventProcessor sequences. To show the performance impact let’s take a few threads each updating their own independent counters. These counters will be volatile longs so the world can see their progress. public final class FalseSharing implements Runnable { public final static int NUM_THREADS = 4; // change public final static long ITERATIONS = 500L * 1000L * 1000L; private final int arrayIndex; private static VolatileLong[] longs = new VolatileLong[NUM_THREADS]; static { for (int i = 0; i < longs.length; i++) { longs[i] = new VolatileLong(); } } public FalseSharing(final int arrayIndex) { this.arrayIndex = arrayIndex; } public static void main(final String[] args) throws Exception { final long start = System.nanoTime(); runTest(); System.out.println("duration = " + (System.nanoTime() - start)); } private static void runTest() throws InterruptedException { Thread[] threads = new Thread[NUM_THREADS]; for (int i = 0; i < threads.length; i++) { threads[i] = new Thread(new FalseSharing(i)); } for (Thread t : threads) { t.start(); } for (Thread t : threads) { t.join(); } } public void run() { long i = ITERATIONS + 1; while (0 != --i) { longs[arrayIndex].value = i; } } public final static class VolatileLong { public volatile long value = 0L; public long p1, p2, p3, p4, p5, p6; // comment out } } Results Running the above code while ramping the number of threads and adding/removing the cache line padding, I get the results depicted in Figure 2. below. This is measuring the duration of test runs on my 4-core Nehalem at home. Figure 2. The impact of false sharing can clearly be seen by the increased execution time required to complete the test. Without the cache line contention we achieve near linear scale up with threads. This is not a perfect test because we cannot be sure where the VolatileLongs will be laid out in memory. They are independent objects. However experience shows that objects allocated at the same time tend to be co-located. So there you have it. False sharing can be a silent performance killer. From http://mechanical-sympathy.blogspot.com/2011/07/false-sharing.html

The integration framework Apache Camel already supports several important cloud services (see my overview article at http://www.kai-waehner.de/blog/2011/07/09/cloud-computing-heterogeneity-will-require-cloud-integration-apache-camel-is-already-prepared for more details). This article describes the combination of Apache Camel and the Amazon Web Services (AWS) interfaces of Simple Storage Service (S3), Simple Queue Service (SQS) and Simple Notification Service (SNS). Thus, The concept of Infrastructure as a Service (IaaS) is used to access messaging systems and data storage without any need for configuration. Registration to AWS and Setup of Camel First, you have to register to the Amazon Web Services (for free). Most AWS services include a free monthly quota, which is absolutely sufficient to play around and develop some simple applications. As its name states, AWS uses technology-independent web services. Besides, APIs for several different programming languages are available to ease development. By the way, Camel uses the AWS SDK for Java (http://aws.amazon.com/sdkforjava), of course. The documentation is detailed and easy to understand, including tutorials, screenshots and code examples . Hint 1: You should read the introductions to S3, SQS and SNS (go to http://aws.amazon.com and click on „products“) and play around with the AWS Management Console (http://aws.amazon.com/console) before you continue. This step is very easy and takes less than one hour. Then, you will have a much better understanding about AWS and where Camel can help you! Hint 2: It really helps to look at the source code of the camel-aws component, It helps you to understand how Camel uses the AWS Java API internally. If you want to write tests, you can do it the same way. In the past, I was afraid of looking at „complex“ source code of open source frameworks. But there is no need to be scared! The camel-aws component (and most other camel components) contain only of a few classes. Everything is easy to understand. It helps you to understand Camel internals, the AWS API, and to spot and solve errors due to exceptions in your code. In the meanwhile, the current Camel version 2.8 supports three AWS services: S3, SQS and SNS. All of them use similar concepts. Therefore, they are included in one single camel component: „camel-aws“. You have to add the libraries to your existing Camel project. As always, the simplest way is to use Maven and add the following dependency to the pom.xml: org.apache.camel camel-aws ${camel-version} Configuration of the Camel Endpoint The implementation and configuration of all three services is very similar. The URI looks like this (the code shows the SQS service): aws-sqs://queue-name[?options] There are two alternatives to configure your endpoint. Using Parameters The easy way is to use two paramters in the URI of your endpoint: „accessKey“ and „secretKey“ (you receive both after your AWS registration). “aws-sqs://unique-queue-name?accessKey=“INSERT_ME“&secretKey=INSERT_ME” Be aware of the following problem, which can result in a strange, non-speaking exception (thanks to Brendan Long): You’ll need to URL encode any +’s in your secret key (otherwise, they’ll be treated as spaces). + = %2B, so if your secretkey was “my+secret\key”, your Camel URL should have “secretKey=my%2Bsecret\key”. “Within the query string, the plus sign is reserved as shorthand notation for a space. Therefore, real plus signs must be encoded. This method was used to make query URIs easier to pass in systems which did not allow spaces.” Source: WC3 URI Recommendations Adding a configured AmazonClient to the Registry If you need to do more configuration (e.g. because your system is behind a firewall), you have to add an AmazonClient object to your registry. The following code shows an example using SQS, but SNS and S3 use exactly the same concept. @Override protected JndiRegistry createRegistry() throws Exception { JndiRegistry registry = super.createRegistry(); AWSCredentials awsCredentials = new BasicAWSCredentials(“INSERT_ME”, “INSERT_ME”); ClientConfiguration clientConfiguration = new ClientConfiguration(); clientConfiguration.setProxyHost(“http://myProxyHost”); clientConfiguration.setProxyPort(8080); AmazonSQSClient client = new AmazonSQSClient(awsCredentials, clientConfiguration); registry.bind(“amazonSQSClient”, client); return registry; } This example overwrites the createRegistry() method of a JUnit test (extending CamelTestSupport). You can also add this information to your runtime Camel application, of course. Apache Camel and the Simple Storage Service (S3) Simple Storage Service (S3) is a key-value-store. You can store small to very large data. The usage is very easy. You create buckets and put key-value data into these buckets. You can also create folders within buckets to organize your data. That’s it. You can monitor your buckets using the AWS Management Console – an intuitive GUI supporting most AWS services. The following example shows both alternatives for accessing the Amazon services (as described above): Paramenters and the AmazonClient. // Transfer data from your file inbox to the AWS S3 service from(“file:files/inbox”) // This is the key of your key-value data .setHeader(S3Constants.KEY, simple(“This is a static key”)) // Using parameters for accessing the AWS service .to(“aws-s3://camel-integration-bucket-mwea-kw?accessKey=INSERT_ME&secretKey=INSERT_ME&region=eu-west-1″); // Transfer data from the AWS S3 service to your file outbox from(“aws-s3://camel-integration-bucket-mwea-kw?amazonS3Client=#amazonS3Client&region=eu-wes”) .to(“file:files/outbox”); There are some additional parameters, for instance you can submit the desired AWS region or delete data after receiving it (see http://camel.apache.org/aws-s3.html and the corresponding SQS and SNS sites for more details about parameters and message headers). As you see in the code, you can use the AWS-S3 endpoint for producing and for consuming messages. Each bucket must be unique, thus you have to add some specific information such as your company to its name. Hint: If a bucket does not exist, Camel is creating it automatically (as the AWS API does). This concept is also used for SQS queues and SNS topics. Apache Camel and the Simple Queue Service (SQS) The Simple Queue Service (SQS) is similar to a JMS provider such as WebSphere MQ or ActiveMQ (but with some differences). You create queues and send messages to them. Consumers receive the messages. Contrary to most other AWS services, you cannot monitor queues by using the AWS management console directly. You have to use the service „Cloudwatch“ (http://aws.amazon.com/cloudwatch) and start an EC2 instance to monitor queues and its content. As you can see in the following code example, the syntax and concepts are almost the same as for the S3 service: from(“file:inbox”) .to(“aws-sqs://camel-integration-queue-mwea-kw?accessKey=INSERT_ME&secretKey=INSERT_ME”); from(“aws-sqs://camel-integration-queue-mwea-kw?amazonSQSClient=#amazonSQSClient”) .to(“file:outbox?fileName=sqs-${date:now:yyyy.MM.dd-hh:mm:ss:SS}”); Again, you can use the AWS-SQS endpoint for producing and for consuming messages. Each queue name must be unique. There exist two important differences to JMS (copy & paste from the AWS documentation): Q: How many times will I receive each message? Amazon SQS is engineered to provide “at least once” delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies. Q: Why are there separate ReceiveMessage and DeleteMessage operations? When Amazon SQS returns a message to you, that message stays in the queue, whether or not you actually received the message. You are responsible for deleting the message; the delete request acknowledges that you’re done processing the message. If you don’t delete the message, Amazon SQS will deliver it again on another receive request. Apache Camel and the Simple Notification Service (SNS) The Simple Notification Service (SNS) acts like JMS topics. You create a topic, consumers subscribe to the topic and then receive notifications. Several transport protocols are supported: HTTP(S), Email and SQS. Further interfaces will be added in the future, e.g. the Short Message Service (SMS) for mobile phones. Contrary to S3 and SQS, Camel only offers a producer endpoint for this AWS service. You can only create topics and send messages via Camel. The reason is simple: Camel already offers endpoints for consuming these messages: HTTP, Email and SQS are already available. There is one tradeoff: A consumer cannot subscribe to topics using Camel – at the moment. The AWS Management Console has to be used. A very interesting discussion can be read on the Camel JIRA issue regarding the following questions: Should Camel be able to subscribe to topics? Should the producer contain this feature or should there be a consumer? In my opinion, there should be a consumer which is able to subscribe to topics, otherwise Camel is missing a key part of the AWS SNS service! Please read the discussion and contribute your opinion: https://issues.apache.org/jira/browse/CAMEL-3476. Apache Camel is already ready for the Cloud Computing Era AWS offers many more services for the cloud. Probably, it does not make sense to integrate everyone into Camel, but more AWS services will be supported in the future. For instance, SimpleDB and the Relational Database Service (RDS) are already planned and make sende, too: http://camel.apache.org/aws.html. The conclusion is easy: Apache Camel is already ready for the cloud computing era. Several important cloud services are already supported. Cloud integration will become very important in the future. Thus, Camel is on a very good way. Hopefully, we will see more cloud components, soon. I will continue to write articles about other Camel cloud components (and new AWS addons, ouf course). For instance, a component for the Platform as a Service (PaaS) product Google App Engine (GAE) is already available. If you have any additional important information, questions or other feedback, please write a comment. Thank you in advance… Best regards, Kai Wähner (Twitter: @KaiWaehner) [Content from my Blog: Cloud Integration with Apache Camel and Amazon Web Services (AWS): S3, SQS and SNS]

when studying both the java nio and io api's, a question quickly pops into mind: when should i use io and when should i use nio? in this text i will try to shed some light on the differences between java nio and io, their use cases, and how they affect the design of your code. main differences of java nio and io the table below summarizes the main differences between java nio and io. i will get into more detail about each difference in the sections following the table. io nio stream oriented buffer oriented blocking io non blocking io selectors stream oriented vs. buffer oriented the first big difference between java nio and io is that io is stream oriented, where nio is buffer oriented. so, what does that mean? java io being stream oriented means that you read one or more bytes at a time, from a stream. what you do with the read bytes is up to you. they are not cached anywhere. furthermore, you cannot move forth and back in the data in a stream. if you need to move forth and back in the data read from a stream, you will need to cache it in a buffer first. java nio's buffer oriented approach is slightly different. data is read into a buffer from which it is later processed. you can move forth and back in the buffer as you need to. this gives you a bit more flexibility during processing. however, you also need to check if the buffer contains all the data you need in order to fully process it. and, you need to make sure that when reading more data into the buffer, you do not overwrite data in the buffer you have not yet processed. blocking vs. non-blocking io java io's various streams are blocking. that means, that when a thread invokes a read() or write(), that thread is blocked until there is some data to read, or the data is fully written. the thread can do nothing else in the meantime. java nio's non-blocking mode enables a thread to request reading data from a channel, and only get what is currently available, or nothing at all, if no data is currently available. rather than remain blocked until data becomes available for reading, the thread can go on with something else. the same is true for non-blocking writing. a thread can request that some data be written to a channel, but not wait for it to be fully written. the thread can then go on and do something else in the mean time. what threads spend their idle time on when not blocked in io calls, is usually performing io on other channels in the meantime. that is, a single thread can now manage multiple channels of input and output. selectors java nio's selectors allow a single thread to monitor multiple channels of input. you can register multiple channels with a selector, then use a single thread to "select" the channels that have input available for processing, or select the channels that are ready for writing. this selector mechanism makes it easy for a single thread to manage multiple channels. how nio and io influences application design whether you choose nio or io as your io toolkit may impact the following aspects of your application design: the api calls to the nio or io classes. the processing of data. the number of thread used to process the data. the api calls of course the api calls when using nio look different than when using io. this is no surprise. rather than just read the data byte for byte from e.g. an inputstream, the data must first be read into a buffer, and then be processed from there. the processing of data the processing of the data is also affected when using a pure nio design, vs. an io design. in an io design you read the data byte for byte from an inputstream or a reader. imagine you were processing a stream of line based textual data. for instance: name: anna age: 25 email: [email protected] phone: 1234567890 this stream of text lines could be processed like this: inputstream input = ... ; // get the inputstream from the client socket bufferedreader reader = new bufferedreader(new inputstreamreader(input)); string nameline = reader.readline(); string ageline = reader.readline(); string emailline = reader.readline(); string phoneline = reader.readline(); notice how the processing state is determined by how far the program has executed. in other words, once the first reader.readline() method returns, you know for sure that a full line of text has been read. the readline() blocks until a full line is read, that's why. you also know that this line contains the name. similarly, when the second readline() call returns, you know that this line contains the age etc. as you can see, the program progresses only when there is new data to read, and for each step you know what that data is. once the executing thread have progressed past reading a certain piece of data in the code, the thread is not going backwards in the data (mostly not). this principle is also illustrated in this diagram: java io: reading data from a blocking stream. a nio implementation would look different. here is a simplified example: bytebuffer buffer = bytebuffer.allocate(48); int bytesread = inchannel.read(buffer); notice the second line which reads bytes from the channel into the bytebuffer. when that method call returns you don't know if all the data you need is inside the buffer. all you know is that the buffer contains some bytes. this makes processing somewhat harder. imagine if, after the first read(buffer) call, that all what was read into the buffer was half a line. for instance, "name: an". can you process that data? not really. you need to wait until at leas a full line of data has been into the buffer, before it makes sense to process any of the data at all. so how do you know if the buffer contains enough data for it to make sense to be processed? well, you don't. the only way to find out, is to look at the data in the buffer. the result is, that you may have to inspect the data in the buffer several times before you know if all the data is inthere. this is both inefficient, and can become messy in terms of program design. for instance: bytebuffer buffer = bytebuffer.allocate(48); int bytesread = inchannel.read(buffer); while(! bufferfull(bytesread) ) { bytesread = inchannel.read(buffer); } the bufferfull() method has to keep track of how much data is read into the buffer, and return either true or false, depending on whether the buffer is full. in other words, if the buffer is ready for processing, it is considered full. the bufferfull() method scans through the buffer, but must leave the buffer in the same state as before the bufferfull() method was called. if not, the next data read into the buffer might not be read in at the correct location. this is not impossible, but it is yet another issue to watch out for. if the buffer is full, it can be processed. if it is not full, you might be able to partially process whatever data is there, if that makes sense in your particular case. in many cases it doesn't. the is-data-in-buffer-ready loop is illustrated in this diagram: java nio: reading data from a channel until all needed data is in buffer. summary nio allows you to manage multiple channels (network connections or files) using only a single (or few) threads, but the cost is that parsing the data might be somewhat more complicated than when reading data from a blocking stream. if you need to manage thousands of open connections simultanously, which each only send a little data, for instance a chat server, implementing the server in nio is probably an advantage. similarly, if you need to keep a lot of open connections to other computers, e.g. in a p2p network, using a single thread to manage all of your outbound connections might be an advantage. this one thread, multiple connections design is illustrated in this diagram: java nio: a single thread managing multiple connections. if you have fewer connections with very high bandwidth, sending a lot of data at a time, perhaps a classic io server implementation might be the best fit. this diagram illustrates a classic io server design: java io: a classic io server design - one connection handled by one thread. from http://tutorials.jenkov.com/java-nio/nio-vs-io.html

In Eclipse, when you press Ctrl button and click on any Class names, the IDE will take you to the source file for that class. This is the normal behavior for the classes you have in your project. But, in case you want the same behavior for Java’s core classes too, you can have it by attaching the Java source with the Eclipse IDE. Once you attach the source, thereafter when you Ctrl+Click any Java class names (String for example), Eclipse will open the source code of that class. To attach the Java source code with Eclipse, When you install the JDK, you must have selected the option to install the Java source files too. This will copy the src.zip file in the installation directory. In Eclipse, go to Window -> Preferences -> Java -> Installed JREs -> Add and choose the JDK you have in your system. Eclipse will now list the JARs found in the dialog box. There, select the rt.jar and choose Source Attachment. By default, this will be pointing to the correct src.zip. If not, choose the src.zip file which you have in your java installation directory. Similarly, if you have the javadoc downloaded in your machine, you can configure that too in this dialog box. Done! Here after, for all the projects for which you are using the above JDK, you’ll be able to browse the Java’s source code just like how you browse your own code. From http://veerasundar.com/blog/2011/08/attaching-java-source-with-eclipse-ide

This article shows *one* way to mock the JMS infrastructure in a Spring JMS application. This allows us to test our JMS infrastructure without actually having to depend on a physical connection being available. If you are reading this article, chances are that you are also frustrated with failing tests in your continuous integration environment due to a JMS server being (temporarily) unavailable. By mocking the JMS provider, developers are left free to test not only the functionality of their API (unit tests) but also the plumbing of the different components, e.g. in a Spring container. In this article I show how a Spring JMS Hello World application can be fully tested without the need of a physical JMS connection. I would like to stress the fact that the code in this article is by no means meant for production and that the approach shown is just one of many. The infrastructure For this article I use the following infrastructure: Apache ActiveMQ, an open source JMS provider, running on an Ubuntu installation Spring 3 Java 6 MockRunner Eclipse as development environment, running on Windows 7 The Spring configuration It's my belief that using what I define as Spring Configuration Strategy Pattern (SCSP) is the right solution in almost all cases when there is the need for a sound testing infrastructure. I will dedicate an entire article to SCSP, for now this is how it looks: The Spring application context Here follows the content of jemosJms-appContext.xml The only important thing to note here is that there are some services which rely on an existing bean named jmsConnectionFactory but that such bean is not defined in this file. This is key to the SCSP and I will illustrate this in one of my future articles. The Spring application context implementation Here follows the content of jemosJms-appContextImpl.xml which could be seen as an implementation of the Spring application context defined above This Spring context file imports the Spring application context defined above and it is this application context which declared the connection factory. This decoupling of the bean requirement (in the super context) from its actual declaration (Spring application context implementation) represents the cornerstore of SCSP. Mocking the JMS provider - The Spring Test application context and MockRunner Following the same approach I used above, I can now declare a fake connection factory which does not require a physical connection to a JMS provider. Here follows the content of jemosJmsTest-appContext.xml. Please note that this file should reside in the test resources of your project, i.e. it should never make it to production. Here the Spring test application context file imports the Spring application context (not its implementation) and it declares a fake connection factory, thanks to the MockRunner MockQueueConnectionFactory class. A POJO listener The job of handling the message is delegated to a simple POJO, which happens to be declared also as a bean: package uk.co.jemos.experiments; public class HelloWorldHandler { /** The application logger */ private static final org.apache.log4j.Logger LOG = org.apache.log4j.Logger .getLogger(HelloWorldHandler.class); public void handleHelloWorld(String msg) { LOG.info("Received message: " + msg); } } There is nothing glamorous about this class. In real life this should have probably be the implementation of an interface, but here I wanted to keep things simple. A simple JMS message producer Here follows an example of a JMS message producer, which would use the real JMS infrastructure to send messages: package uk.co.jemos.experiments; import org.springframework.context.ApplicationContext; import org.springframework.context.support.ClassPathXmlApplicationContext; import org.springframework.jms.core.JmsTemplate; public class JmsTest { /** The application logger */ private static final org.apache.log4j.Logger LOG = org.apache.log4j.Logger .getLogger(JmsTest.class); /** * @param args */ public static void main(String[] args) { ApplicationContext ctx = new ClassPathXmlApplicationContext( "classpath:jemosJms-appContextImpl.xml"); JmsTemplate jmsTemplate = ctx.getBean(JmsTemplate.class); jmsTemplate.send("jemos.tests", new HelloWorldMessageCreator()); LOG.info("Message sent successfully"); } } The only thing of interest here is that this class retrieves the real JmsTemplate to send a message to the queue. Now if I was to run this class as is, I would obtain the following: 2011-07-31 17:09:46 ClassPathXmlApplicationContext [INFO] Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@19e0ff2f: startup date [Sun Jul 31 17:09:46 BST 2011]; root of context hierarchy 2011-07-31 17:09:46 XmlBeanDefinitionReader [INFO] Loading XML bean definitions from class path resource [jemosJms-appContextImpl.xml] 2011-07-31 17:09:46 XmlBeanDefinitionReader [INFO] Loading XML bean definitions from class path resource [jemosJms-appContext.xml] 2011-07-31 17:09:46 DefaultListableBeanFactory [INFO] Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@3479e304: defining beans [helloWorldConsumer,jmsTemplate,org.springframework.jms.listener.DefaultMessageListenerContainer#0,jmsConnectionFactory]; root of factory hierarchy 2011-07-31 17:09:46 DefaultLifecycleProcessor [INFO] Starting beans in phase 2147483647 2011-07-31 17:09:47 HelloWorldHandler [INFO] Received message: Hello World 2011-07-31 17:09:47 JmsTest [INFO] Message sent successfully Writing the integration test There are various interpretations as to what different types of tests mean and I don't pretend to have the only answer; my interpreation is that an integration test is a functional test which also wires up different components together but which does not interact with real external infrastructure (e.g. a Dao integration test fakes data, a JMS integration test fakes the JMS physical connection, an HTTP integration test fakes the remote Web host, etc). Whereas in my opinion, the main purpose of a unit (aka functional) test is to let the API emerge from the tests, the main goal of an integration test is to test that the plumbing amongst components works as expected so as to avoid surprises in a production environment. Both unit (functional) and integration tests should run very fast (e.g. under 10 minutes) as they constitute what can be considered the "development token". If unit and integration tests are green one should feel pretty confident that 90% of the functionality works as expected; in my projects when both unit and integration tests are green I let developers free to release the token. This does not mean that the other 10% (e.g. the interaction with the real infrastructure) should not be tested, but this can be delegated to system tests which run nightly and don't require the development token. Because unit and integration tests need to run fast, interaction with external infrastructure should be mocked whenever possible. Here follows an integration test for the Hello World handler: package uk.co.jemos.experiments.test.integration; import javax.annotation.Resource; import javax.jms.TextMessage; import junit.framework.Assert; import org.junit.Before; import org.junit.Test; import org.springframework.jms.core.JmsTemplate; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.AbstractJUnit4SpringContextTests; import uk.co.jemos.experiments.HelloWorldHandler; import uk.co.jemos.experiments.HelloWorldMessageCreator; import com.mockrunner.jms.DestinationManager; import com.mockrunner.mock.jms.MockQueue; /** * @author mtedone * */ @ContextConfiguration(locations = { "classpath:jemosJmsTest-appContextImpl.xml" }) public class HelloWorldHandlerIntegrationTest extends AbstractJUnit4SpringContextTests { @Resource private JmsTemplate jmsTemplate; @Resource private DestinationManager mockDestinationManager; @Resource private HelloWorldHandler helloWorldHandler; @Before public void init() { Assert.assertNotNull(jmsTemplate); Assert.assertNotNull(mockDestinationManager); Assert.assertNotNull(helloWorldHandler); } @Test public void helloWorld() throws Exception { MockQueue mockQueue = mockDestinationManager.createQueue("jemos.tests"); jmsTemplate.send(mockQueue, new HelloWorldMessageCreator()); TextMessage message = (TextMessage) jmsTemplate.receive(mockQueue); Assert.assertNotNull("The text message cannot be null!", message.getText()); helloWorldHandler.handleHelloWorld(message.getText()); } } And here follows the output: 2011-07-31 17:17:26 XmlBeanDefinitionReader [INFO] Loading XML bean definitions from class path resource [jemosJmsTest-appContextImpl.xml] 2011-07-31 17:17:26 XmlBeanDefinitionReader [INFO] Loading XML bean definitions from class path resource [jemosJms-appContext.xml] 2011-07-31 17:17:26 GenericApplicationContext [INFO] Refreshing org.springframework.context.support.GenericApplicationContext@f01a1e: startup date [Sun Jul 31 17:17:26 BST 2011]; root of context hierarchy 2011-07-31 17:17:27 DefaultListableBeanFactory [INFO] Pre-instantiating singletons in org.springframework.beans.factory.support. DefaultListableBeanFactory@39478a43: defining beans [helloWorldConsumer,jmsTemplate,org.springframework.jms.listener.DefaultMessageListener Container#0,destinationManager,configurationManager,jmsConnectionFactory,org.springframework.context.annotation.internalConfigurationAnnotation Processor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequired AnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor]; root of factory hierarchy 2011-07-31 17:17:27 DefaultLifecycleProcessor [INFO] Starting beans in phase 2147483647 2011-07-31 17:17:27 HelloWorldHandler [INFO] Received message: Hello World 2011-07-31 17:17:27 GenericApplicationContext [INFO] Closing org.springframework.context.support.GenericApplicationContext@f01a1e: startup date [Sun Jul 31 17:17:26 BST 2011]; root of context hierarchy 2011-07-31 17:17:27 DefaultLifecycleProcessor [INFO] Stopping beans in phase 2147483647 2011-07-31 17:17:32 DefaultMessageListenerContainer [WARN] Setup of JMS message listener invoker failed for destination 'jemos.tests' - trying to recover. Cause: Queue with name jemos.tests not found 2011-07-31 17:17:32 DefaultListableBeanFactory [INFO] Destroying singletons in org.springframework.beans.factory.support. DefaultListableBeanFactory@39478a43: defining beans [helloWorldConsumer,jmsTemplate,org.springframework.jms.listener.DefaultMessageListener Container#0,destinationManager,configurationManager,jmsConnectionFactory,org.springframework.context.annotation.internalConfigurationAnnotationProcessor ,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context. annotation.internalCommonAnnotationProcessor]; root of factory hierarchy In this test, although we simulated a message roundtrip to a JMS queue, the message never left the current JVM and it the whole execution did not depend on a JMS infrastructure being up. This gives us the power to simulate the JMS infrastructure, to test the integration of our business components without having to fear a red from time to time due to JMS infrastructure being down or inaccessible. Please note that in the output there are some warnings because the JMS listener container declared in the jemosJms-appContext.xml does not find a queue named "jemos.test" in the fake connection factory, but this is fine; it's a warning and does not impede the test from running successfully. The Maven configuration Here follows the Maven pom.xml to compile the example: 4.0.0 uk.co.jemos.experiments jmx-experiments 0.0.1-SNAPSHOT Jemos JMS experiments junit junit 4.8.2 test com.mockrunner mockrunner 0.3.1 test log4j log4j 1.2.16 compile org.slf4j slf4j-api 1.6.1 compile org.slf4j slf4j-simple 1.6.1 compile org.apache.activemq activemq-all 5.5.0 compile org.springframework spring-beans 3.0.5.RELEASE org.springframework spring-context 3.0.5.RELEASE org.springframework spring-core 3.0.5.RELEASE org.springframework spring-jms 3.0.5.RELEASE org.springframework spring-test 3.0.5.RELEASE test From http://tedone.typepad.com/blog/2011/07/mocking-spring-jms-with-mockrunner.html

i often get asked about ‘rest to soap’ transformation use cases these days. using an soa gateway like securespan to perform this type of transformation at runtime is trivial to setup. with securespan in front of any existing web service (in the dmz for example), you can virtualize a rest version of this same service. using an example, here is a description of the steps to perform this conversion. imagine the geoloc web service for recording geographical locations. it has two methods, one for setting a location and one for getting a location. see below what this would look like in soap. request: 34802398402 response: 52.37706 4.889721 request: 34802398402 52.37706 4.889721 response: ok here is the equivalent rest target that i want to support at the edge. payloads could be xml, but let’s use json to make it more interesting. get /position/34802398402 http 200 ok content-type: text/json { 'latitude' : 52.37706 'longitude' : 4.889721 } post /position/34802398402 content-type: text/json { 'latitude' : 52.37706 'longitude' : 4.889721 } http 200 ok ok now let’s implement this rest version of the service using securespan. i’m assuming that you already have a securespan gateway deployed between the potential rest requesters and the existing soap web service. first, i will create a new service endpoint on the gateway for this service and assign anything that comes at the uri pattern /position/* to this service. i will also allow the http verbs get and post for this service. rest geoloc service properties next, let’s isolate the resource id from the uri and save this as a context variable named ‘trackerid’. we can use a simple regex assertion to accomplish this. also, i will branch on the incoming http verb using an or statement. i am just focusing on get and post for this example but you could add additional logic for other http verbs that you want to support for this rest service. regex for rest service resource identification policy branching for get vs post for get requests, the transformation is very simple, we just declare a message variable using a soap skeleton into which we refer to the trackerid variable. soap request template this soap message is routed to the existing web service and the essential elements are isolated using xpath assertions. processing soap response the rest response is then constructed back using a template response. template json response a similar logic is performed for the post message. see below for the full policy logic. complete policy you’re done for virtualizing the rest service. setting this up with securespan took less than an hour, did not require any change on the existing soap web service and did not require the deployment of an additional component. from there, you would probably enrich the policy to perform some json schema validation , some url and query parameter validation, perhaps some authentication, authorization , etc.

martin fowler has written a really good article describing not only the disruptor , but also how it fits into the architecture at lmax . this gives some of the context that has been missing so far, but the most frequently asked question is still "what is the disruptor?". i'm working up to answering that. i'm currently on question number two: "why is it so fast?". these questions do go hand in hand, however, because i can't talk about why it's fast without saying what it does, and i can't talk about what it is without saying why it is that way. so i'm trapped in a circular dependency. a circular dependency of blogging. to break the dependency, i'm going to answer question one with the simplest answer, and with any luck i'll come back to it in a later post if it still needs explanation: the disruptor is a way to pass information between threads. as a developer, already my alarm bells are going off because the word "thread" was just mentioned, which means this is about concurrency, and concurrency is hard. concurrency 101 imagine two threads are trying to change the same value. case one: thread 1 gets there first: the value changes to "blah" then the value changes to "blahy" when thread 2 gets there. case two: thread 2 gets there first: the value changes to "fluffy" then the value changes to "blah" when thread 1 gets there. case three: thread 1 interrupts thread 2: thread 2 gets the value "fluff" and stores it as myvalue thread 1 goes in and updates value to "blah" then thread 2 wakes up and sets the value to "fluffy". case three is probably the only one which is definitely wrong, unless you think the naive approach to wiki editing is ok ( google code wiki, i'm looking at you...). in the other two cases it's all about intentions and predictability. thread 2 might not care what's in value, the intention might be to append "y" to whatever is in there regardless. in this circumstance, cases one and two are both correct. but if thread 2 only wanted to change "fluff" to "fluffy", then both cases two and three are incorrect. assuming that thread 2 wants to set the value to "fluffy", there are some different approaches to solving the problem. approach one: pessimistic locking (does the "no entry" sign make sense to people who don't drive in britain?) the terms pessimistic and optimistic locking seem to be more commonly used when talking about database reads and writes, but the principal applies to getting a lock on an object. thread 2 grabs a lock on entry as soon as it knows it needs it and stops anything from setting it. then it does its thing, sets the value, and lets everything else carry on. you can imagine this gets quite expensive, with threads hanging around all over the place trying to get hold of objects and being blocked. the more threads you have, the more chance that things are going to grind to a halt. approach two: optimistic locking in this case thread 2 will only lock entry when it needs to write to it. in order to make this work, it needs to check if entry has changed since it first looked at it. if thread 1 came in and changed the value to "blah" after thread 2 had read the value, thread 2 couldn't write "fluffy" to the entry and trample all over the change from thread 1. thread 2 could either re-try (go back, read the value, and append "y" onto the end of the new value), which you would do if thread 2 didn't care what the value it was changing was; or it could throw an exception or return some sort of failed update flag if it was expecting to change "fluff" to "fluffy". an example of this latter case might be if you have two users trying to update a wiki page, and you tell the user on the other end of thread 2 they'll need to load the new changes from thread 1 and then reapply their changes. potential problem: deadlock locking can lead to all sorts of issues, for example deadlock. imagine two threads that need access to two resources to do whatever they need to do: if you've used an over-zealous locking technique, both threads are going to sit there forever waiting for the other one to release its lock on the resource. that's when you reboot windows your computer. definite problem: locks are sloooow... the thing about locks is that they need the operating system to arbitrate the argument. the threads are like siblings squabbling over a toy, and the os kernel is the parent that decides which one gets it. it's like when you run to your dad to tell him your sister has nicked the transformer when you wanted to play with it - he's got bigger things to worry about than you two fighting, and he might finish off loading the dishwasher and putting on the laundry before settling the argument. if you draw attention to yourself with a lock, not only does it take time to get the operating system to arbitrate, the os might decide the cpu has better things to do than servicing your thread. the disruptor paper talks about an experiment we did. the test calls a function incrementing a 64-bit counter in a loop 500 million times. for a single thread with no locking, the test takes 300ms. if you add a lock (and this is for a single thread, no contention, and no additional complexity other than the lock) the test takes 10,000ms. that's, like, two orders of magnitude slower. even more astounding, if you add a second thread (which logic suggests should take maybe half the time of the single thread with a lock) it takes 224,000ms. incrementing a counter 500 million times takes nearly a thousand times longer when you split it over two threads instead of running it on one with no lock. concurrency is hard and locks are bad i'm just touching the surface of the problem, and obviously i'm using very simple examples. but the point is, if your code is meant to work in a multi-threaded environment, your job as a developer just got a lot more difficult: naive code can have unintended consequences. case three above is an example of how things can go horribly wrong if you don't realise you have multiple threads accessing and writing to the same data. selfish code is going to slow your system down. using locks to protect your code from the problem in case three can lead to things like deadlock or simply poor performance. this is why many organisations have some sort of concurrency problems in their interview process (certainly for java interviews). unfortunately it's very easy to learn how to answer the questions without really understanding the problem, or possible solutions to it. how does the disruptor address these issues? for a start, it doesn't use locks. at all. instead, where we need to make sure that operations are thread-safe (specifically, updating the next available sequence number in the case of multiple producers ), we use a cas (compare and swap/set) operation. this is a cpu-level instruction, and in my mind it works a bit like optimistic locking - the cpu goes to update a value, but if the value it's changing it from is not the one it expects, the operation fails because clearly something else got in there first. note this could be two different cores rather than two separate cpus. cas operations are much cheaper than locks because they don't involve the operating system, they go straight to the cpu. but they're not cost-free - in the experiment i mentioned above, where a lock-free thread takes 300ms and a thread with a lock takes 10,000ms, a single thread using cas takes 5,700ms. so it takes less time than using a lock, but more time than a single thread that doesn't worry about contention at all. back to the disruptor - i talked about the claimstrategy when i went over the producers . in the code you'll see two strategies, a singlethreadedstrategy and a multithreadedstrategy. you could argue, why not just use the multi-threaded one with only a single producer? surely it can handle that case? and it can. but the multi-threaded one uses an atomiclong (java's way of providing cas operations), and the single-threaded one uses a simple long with no locks and no cas. this means the single-threaded claim strategy is as fast as possible, given that it knows there is only one producer and therefore no contention on the sequence number. i know what you're thinking: turning one single number into an atomiclong can't possibly have been the only thing that is the secret to the disruptor's speed. and of course, it's not - otherwise this wouldn't be called "why it's so fast (part one )". but this is an important point - there's only one place in the code where multiple threads might be trying to update the same value. only one place in the whole of this complicated data-structure-slash-framework. and that's the secret. remember everything has its own sequence number? if you only have one producer then every sequence number in the system is only ever written to by one thread. that means there is no contention. no need for locks. no need even for cas. the only sequence number that is ever written to by more than one thread is the one on the claimstrategy if there is more than one producer. this is also why each variable in the entry can only be written to by one consumer . it ensures there's no write contention, therefore no need for locks or cas. back to why queues aren't up to the job so you start to see why queues, which may implemented as a ring buffer under the covers, still can't match the performance of the disruptor. the queue, and the basic ring buffer , only has two pointers - one to the front of the queue and one to the end: if more than one producer wants to place something on the queue, the tail pointer will be a point of contention as more than one thing wants to write to it. if there's more than one consumer, then the head pointer is contended, because this is not just a read operation but a write, as the pointer is updated when the item is consumed from the queue. but wait, i hear you cry foul! because we already knew this, so queues are usually single producer and single consumer (or at least they are in all the queue comparisons in our performance tests). there's another thing to bear in mind with queues/buffers. the whole point is to provide a place for things to hang out between producers and consumers, to help buffer bursts of messages from one to the other. this means the buffer is usually full (the producer is out-pacing the consumer) or empty (the consumer is out-pacing the producer). it's rare that the producer and consumer will be so evenly-matched that the buffer has items in it but the producers and consumers are keeping pace with each other. so this is how things really look. an empty queue: ...and a full queue: the queue needs a size so that it can tell the difference between empty and full. or, if it doesn't, it might determine that based on the contents of that entry, in which case reading an entry will require a write to erase it or mark it as consumed. whichever implementation is chosen, there's quite a bit of contention around the tail, head and size variables, or the entry itself if a consume operation also includes a write to remove it. on top of this, these three variables are often in the same cache line , leading to false sharing . so, not only do you have to worry about the producer and the consumer both causing a write to the size variable (or the entry), updating the tail pointer could lead to a cache-miss when the head pointer is updated because they're sat in the same place. i'm going to duck out of going into that in detail because this post is quite long enough as it is. so this is what we mean when we talk about "teasing apart the concerns" or a queue's "conflated concerns". by giving everything its own sequence number and by allowing only one consumer to write to each variable in the entry, the only case the disruptor needs to manage contention is where more than one producer is writing to the ring buffer. in summary the disruptor a number of advantages over traditional approaches: no contention = no locks = it's very fast. having everything track its own sequence number allows multiple producers and multiple consumers to use the same data structure. tracking sequence numbers at each individual place (ring buffer, claim strategy, producers and consumers), plus the magic cache line padding , means no false sharing and no unexpected contention. from http://mechanitis.blogspot.com/2011/07/dissecting-disruptor-why-its-so-fast.html

I’m currently working on a project that is converted from being a Mainframe application, to a Java web/batch application. We don’t ‘big bang’ into production, so the Mainframe and the Java code will work next to each other for a fairly amount of time. Since we have multiple batch processes and many simultaneous users, we start seeing deadlock errors in certain parts of the application. Some specific parts have to take a pessimistic lock, this is where it goes wrong. Since a deadlock is an error that can be solved by repeating the action, we decide to build in a retry mechanism to restart the transaction if it got rolled back. I started of with creating an Annotation. This annotation will mark the entry point that we want to retry in case of a deadlock. @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.METHOD) public @interface DeadLockRetry { /** * Retry count. default value 3 */ int retryCount() default 3; } The retry count is a value you can supply together with your annotation, so you can specify the number of times we want to retry our operation. Using AOP we can pick up this annotation an let us surround the method call with a retry mechanism. @Around(value = "@annotation(deadLockRetry)", argNames = "deadLockRetry") So lets view the aspect, we start with adding an @Aspect annotation on top of our class, this way it is configured to be an Aspect. We also want to implement the Ordered interface. This interface lets us order our aspect. We need this to surround our Transactional aspect. If we don’t surround our Transaction, we will never be able to retry in a new transaction, we would be working in the same (marked as rollback only) transaction. The rest of the code is pretty straight forward. We create a loop where we loop until we have more retries than we should have. Inside that loop we proceed our ProceedingJoinPoint and catch the PersistenceException that JPA would throw when a deadlock would occur. Inside the catch block we check if the error code is a deadlock error code. Off course we could not directly configure the database specific error codes inside our aspect, so I’ve created an interface. /** * Interface that marks a dialect aware of certain error codes. When you have to * do a low level check of the exception you are trying to handle, you can * implement this in this interface, so you can encapsulate the specific error * codes for the specific dialects. * * @author Jelle Victoor * @version 05-jul-2011 */ public interface ErrorCodeAware { Set getDeadlockErrorCodes(); } We already have custom hibernate dialects for our database and database to be, so this let me configure the error codes in the Dialect implementations. It was a bit tricky to get the current dialect. I injected the persistence unit, since we are outside a transaction, and made some casts to get my dialect. The alternative was to use a custom implementation of the ErrorCodeAware interface, not using the dialects. We could inject the needed ErrorCodeAware implementation based on our application context. This added another database specific injection, which added another point of configuration. This is why I chose to store it in our custom dialect. private Dialect getDialect() { final SessionFactory sessionFactory = ((HibernateEntityManagerFactory) emf).getSessionFactory(); return ((SessionFactoryImplementor) sessionFactory).getDialect(); } The only thing left is to configure the aspect, mind the order of the transaction manager and the retry aspect Now when I have a deadlock exception, and I’ve added this annotation, the transaction will rollback and will be reexecuted. /** * This Aspect will cause methods to retry if there is a notion of a deadlock. * * Note that the aspect implements the Ordered interface so we can set the * precedence of the aspect higher than the transaction advice (we want a fresh * transaction each time we retry). * * @author Jelle Victoor * @version 04-jul-2011 handles deadlocks */ @Aspect public class DeadLockRetryAspect implements Ordered { private static final Logger LOGGER = LoggerFactory.getLogger(DeadLockRetryAspect.class); private int order = -1; @PersistenceUnit private EntityManagerFactory emf; /** * Deadlock retry. The aspect applies to every service method with the * annotation {@link DeadLockRetry} * * @param pjp * the joinpoint * @param deadLockRetry * the concurrency retry * @return * * @throws Throwable * the throwable */ @Around(value = "@annotation(deadLockRetry)", argNames = "deadLockRetry") public Object concurrencyRetry(final ProceedingJoinPoint pjp, final DeadLockRetry deadLockRetry) throws Throwable { final Integer retryCount = deadLockRetry.retryCount(); Integer deadlockCounter = 0; Object result = null; while (deadlockCounter < retryCount) { try { result = pjp.proceed(); break; } catch (final PersistenceException exception) { deadlockCounter = handleException(exception, deadlockCounter, retryCount); } } return result; } /** * handles the persistence exception. Performs checks to see if the * exception is a deadlock and check the retry count. * * @param exception * the persistence exception that could be a deadlock * @param deadlockCounter * the counter of occured deadlocks * @param retryCount * the max retry count * @return the deadlockCounter that is incremented */ private Integer handleException(final PersistenceException exception, Integer deadlockCounter, final Integer retryCount) { if (isDeadlock(exception)) { deadlockCounter++; LOGGER.error("Deadlocked ", exception.getMessage()); if (deadlockCounter == (retryCount - 1)) { throw exception; } } else { throw exception; } return deadlockCounter; } /** * check if the exception is a deadlock error. * * @param exception * the persitence error * @return is a deadlock error */ private Boolean isDeadlock(final PersistenceException exception) { Boolean isDeadlock = Boolean.FALSE; final Dialect dialect = getDialect(); if (dialect instanceof ErrorCodeAware && exception.getCause() instanceof GenericJDBCException) { if (((ErrorCodeAware) dialect).getDeadlockErrorCodes().contains(getSQLErrorCode(exception))) { isDeadlock = Boolean.TRUE; } } return isDeadlock; } /** * Returns the currently used dialect * * @return the dialect */ private Dialect getDialect() { final SessionFactory sessionFactory = ((HibernateEntityManagerFactory) emf).getSessionFactory(); return ((SessionFactoryImplementor) sessionFactory).getDialect(); } /** * extracts the low level sql error code from the * {@link PersistenceException} * * @param exception * the persistence exception * @return the low level sql error code */ private int getSQLErrorCode(final PersistenceException exception) { return ((GenericJDBCException) exception.getCause()).getSQLException().getErrorCode(); } /** {@inheritDoc} */ public int getOrder() { return order; } /** * Sets the order. * * @param order * the order to set */ public void setOrder(final int order) { this.order = order; } } From http://styledideas.be/blog/2011/07/05/automatic-deadlock-retry-aspect-with-spring-and-jpahibernate/