I’m a huge Vaadin fan and I’ve created a Github workshop I can demo at conferences. A common issue with such kind of workshops is that attendees have to prepare their workstations in advance… and there’s always a significant part of them that comes with not everything ready. At this point, two options are available to the speaker: either wait for each of the attendee to finish the preparation – too bad for the people who took the time at home to do that, or start anyway – and lose the not-ready part. Given the current buzz around Docker, I thought that could be a very good way to make the workshop preparation quicker – only one step, and hasslefree – no problem regarding the quirks of your operation system. The required steps I ask the attendees are the following: Install Git Install Java, Maven and Tomcat Clone the git repo Build the project (to prepare the Maven repository) Deploy the built webapp Start Tomcat These should directly be automated into Docker. As I wasted much time getting this to work, here’s the tale of my journey in achieving this (be warned, it’s quite long). If you’ve got similar use-cases, I hope it will be useful in you getting things done faster. Starting with Docker The first step was to get to know the basics about Docker. Fortunately, I had the chance to attend a Docker workshop by David Gageot at Duchess Swiss. This included both Docker installation and basics of Dockerfile. I assume readers have likewise a basic understanding of Docker. For those who don’t, I guess browsing the Docker’s official documentation is a nice idea: Installation Dockerfile reference Building my first Dockerfile The Docker image can be built with the following command ran into the directory of the Dockerfile: $ docker build -t vaadinworkshop . The first issues one can encounter when playing with Docker the first time, is to get the following error message: Get http:///var/run/docker.sock/v1.14/containers/json: dial unix /var/run/docker.sock: no such file or directory The reason is because one didn’t export the required environment variables displayed by the boot2docker information message. If you lost the exact data, no worry, just use the shellinit boot2docker parameter: $ boot2docker shellinit Writing /Users/i303869/.docker/boot2docker-vm/ca.pem: Writing /Users/i303869/.docker/boot2docker-vm/cert.pem: Writing /Users/i303869/.docker/boot2docker-vm/key.pem: export DOCKER_HOST=tcp://192.168.59.103:2376 export DOCKER_CERT_PATH=/Users/i303869/.docker/boot2docker-vm Copy-paste the export lines above will solve the issue. These can also be set in one’s .bashrc script as it seems these values seldom change. Next in line is the following error: Get http://192.168.59.103:2376/v1.14/containers/json: malformed HTTP response "x15x03x01x00x02x02" This error message seems to be because of a mismatch between versions of the client and the server. It seems it is because of a bug on Mac OSX when upgrading. For a long term solution, reinstall Docker from scratch; for a quick fix, use the --tls flag with the docker command. As it is quite cumbersome to type it everything, one can alias it: $ alias docker="docker --tls" My last mistake when building the image comes from building the Dockerfile from a not empty directory. Docker sends every file it finds in the directory of the Dockerfile to the Docker container for build: $ docker --tls build -t vaadinworkshop . Sending build context to Docker daemon Too many kB Fix: do not try this at home and start from a directory container the Dockerfile only. Starting from scratch Dockerfiles describe images – images are built as a layered list of instructions. Docker images are designed around single inheritance: one image has to be set a single parent. An image requiring no parent starts from scratch, but Docker provides 4 base official distributions: busybox, debian, ubuntu and centos (operating systems are generally a good start). Whatever you want to achieve, it is necessary to choose the right parent. Given the requirements I set for myself (Java, Maven, Tomcat and Git), I tried to find the right starting image. Many Dockerfiles are already available online on the Docker hub. The browsing app is quite good, but to be really honest, the search can really be improved. My intention was to use the image that matched the most of my requirements, then fill the gap. I could find no image providing Git, but I thought the dgageot/maven Dockerfile would be a nice starting point. The problem is that the base image is a busybox and provides no installer out-of-the-box (apt-get, yum, whatever). For this reason, David uses a lot of curl to get Java 8 and Maven in his Dockerfiles. I foolishly thought I could use a different flavor of busybox that provides the opkg installer. After a while, I accumulated many problems, resolving one heading to another. In the end, I finally decided to use the OS I was most comfortable with and to install everything myself: FROM ubuntu:utopic Scripting Java installation Installing git, maven and tomcat packages is very straightforward (if you don’t forget to use the non-interactive options) with RUN and apt-get: RUN apt-get update && \ apt-get install -y --force-yes git maven tomcat8 Java doesn’t fall into this nice pattern, as Oracle wants you to accept the license. Nice people did however publish it to a third-party repo. Steps are the following: Add the needed package repository Configure the system to automatically accept the license Configure the system to add un-certified packages Update the list of repositories At last, install the package Also add a package for Java 8 system configuration. RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list && \ echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && \ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 RUN apt-get update && \ apt-get install -y --force-yes oracle-java8-installer oracle-java8-set-default Building the sources Getting the workshop’s sources and building them is quite straightforward with the following instructions: RUN git clone https://github.com/nfrankel/vaadin7-workshop.git WORKDIR /vaadin7-workshop RUN mvn package The drawback of this approach is that Maven will start from a fresh repository, and thus download the Internet the first time it is launched. At first, I wanted to mount a volume from the host to the container to share the ~/.m2/repository folder to avoid this, but I noticed this could only be done at runtime through the -v option as the VOLUME instruction cannot point to a host directory. Starting the image The simplest command to start the created Docker image is the following: $ docker run -p 8080:8080 Do not forget the port forwarding from the container to the host, 8080 for the standard HTTP port. Also, note that it’s not necessary to run the container as a daemon (with the -d option). The added value of that is that the standard output of the CMD (see below) will be redirected to the host. When running as a daemon and wanting to check the logs, one has to execute bash in the container, which requires a sequence of cumbersome manipulations. Configuring and launching Tomcat Tomcat can be launched when starting the container by just adding the following instruction to the Dockerfile: CMD ["catalina.sh", "run"] However, trying to start the container at this point will result in the following error: Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/common/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/common], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/server/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/server], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/shared/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/shared], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina initDirs SEVERE: Cannot find specified temporary folder at /usr/share/tomcat8/temp Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina load WARNING: Unable to load server configuration from [/usr/share/tomcat8/conf/server.xml] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina initDirs SEVERE: Cannot find specified temporary folder at /usr/share/tomcat8/temp Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina load WARNING: Unable to load server configuration from [/usr/share/tomcat8/conf/server.xml] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina start SEVERE: Cannot start server. Server instance is not configured. I have no idea why, but it seems Tomcat 8 on Ubuntu is not configured in any meaningful way. Everything is available but we need some symbolic links here and there as well as creating the temp directory. This translates into the following instruction in the Dockerfile: RUN ln -s /var/lib/tomcat8/common $CATALINA_HOME/common && \ ln -s /var/lib/tomcat8/server $CATALINA_HOME/server && \ ln -s /var/lib/tomcat8/shared $CATALINA_HOME/shared && \ ln -s /etc/tomcat8 $CATALINA_HOME/conf && \ mkdir $CATALINA_HOME/temp The final trick is to connect the exploded webapp folder created by Maven to Tomcat’s webapps folder, which it looks for deployments: RUN mkdir $CATALINA_HOME/webapps && \ ln -s /vaadin7-workshop/target/workshop-7.2-1.0-SNAPSHOT/ $CATALINA_HOME/webapps/vaadinworkshop At this point, the Holy Grail is not far away, you just have to browse the URL… if only we knew what the IP was. Since running on Mac, there’s an additional VM beside the host and the container that’s involved. To get this IP, type: $ boot2docker ip The VM's Host only interface IP address is: 192.168.59.103 Now, browsing http://192.168.59.103:8080/vaadinworkshop/ will bring us to the familiar workshop screen: Developing from there Everything works fine but didn’t we just forget about one important thing, like how workshop attendees are supposed to work on the sources? Easy enough, just mount the volume when starting the container: docker run -v /Users//vaadin7-workshop:/vaadin7-workshop -p 8080:8080 vaadinworkshop Note that the host volume must be part of /Users and if on OSX, it must use boot2docker v. 1.3+. Unfortunately, it seems now is the showstopper, as mounting an empty directory from the host to the container will not make the container’s directory available from the host. On the contrary, it will empty the container’s directory given that the host’s directory doesn’t exist… It seems there’s an issue in Docker on Mac. The installation of JHipster runs into the same problem, and proposes to use the Samba Docker folder sharing project. I’m afraid I was too lazy to go further at this point. However, this taught me much about Docker, its usages and use-cases (as well as OSX integration limitations). For those who are interested, you’ll find below the Docker file. Happy Docker! FROM ubuntu:utopic MAINTAINER Nicolas Frankel # Config to get to install Java 8 w/o interaction RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list && echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 RUN apt-get update && apt-get install -y --force-yes git oracle-java8-installer oracle-java8-set-default maven tomcat8 RUN git clone https://github.com/nfrankel/vaadin7-workshop.git WORKDIR /vaadin7-workshop RUN git checkout v7.2-1 RUN mvn package ENV JAVA_HOME /usr/lib/jvm/java-8-oracle ENV CATALINA_HOME /usr/share/tomcat8 ENV PATH $PATH:$CATALINA_HOME/bin # Configure Tomcat 8 directories RUN ln -s /var/lib/tomcat8/common $CATALINA_HOME/common && ln -s /var/lib/tomcat8/server $CATALINA_HOME/server && ln -s /var/lib/tomcat8/shared $CATALINA_HOME/shared && ln -s /etc/tomcat8 $CATALINA_HOME/conf && mkdir $CATALINA_HOME/temp && mkdir $CATALINA_HOME/webapps && ln -s /vaadin7-workshop/target/workshop-7.2-1.0-SNAPSHOT/ $CATALINA_HOME/webapps/vaadinworkshop VOLUME ["/vaadin7-workshop"] CMD ["catalina.sh", "run"] # docker build -t vaadinworkshop . # docker run -v ~/vaadin7-workshop training/webapp -p 8080:8080 vaadinworkshop

[This article was written by Artem Bilan.] Dear Spring community, As we promised in the Release Candidate blog post, we are pleased to announce that the Spring Integration Java DSL 1.0 GA is now available. As usual, use the Release Repository with Maven or Gradle, or download a distribution archive, to give it a spin. See the project home page for more information. First of all, we are glad to share with you that on Nov 12, 2014, DZone research recognized Spring Integration as the leader in the ESB / Integration framework space, leading with 42% marketshare, in a publication of their recent survey results. And the report is the most popular DZone Guide in November, with more than 12 000 downloads already! Don't miss it: very exciting. We hope the release of the Spring Integration Java DSL adds more excitement!. Many thanks to all contributors, including several who are new to the community. The release includes just a few bug fixes, since the release candidate, and a lot of JavaDocs! Not specifically related to the the release, I want to present here some resources on the matter. We are observing many valuable DSL questions on Stack Overflow. Josh Long's tech tip showing how we can use together Spring Boot, REST, Spring Integration 4.1 WebSocket support and Spring Integration Java DSL plus Java 8 features. The Jdbc Splitter implementation in the project tests. My gist to demonstrate how we can use Reactor Streams together with the Spring Integration Java DSL. Dave Syer has started to use Spring Integration Java DSL in the Spring Cloud Bus project. Don't miss the si4demo to see the evolution of Spring Integration including the Java DSL, as shown at the 2014 SpringOne/2GX Conference. (Video should be available soon). Especial thanks to Biju Kunjummen who has done some nice articles on DZone to introduce Spring Integration Java DSL: https://dzone.com/articles/spring-integration-java-dsl, https://dzone.com/articles/spring-integration-java-dsl-0. And of course, with the latest Spring XD, we can build Modules based on @Configuration including Spring Integration Java DSL IntegrationFlow definitions. Just after this announcement I'm going to publish a DSL Tutorial to explain concepts and features using the Java DSL version of the Cafe Demo sample as material. As always, we look forward to your comments and feedback (StackOverflow (spring-integration tag), Spring JIRA, GitHub) and we very much welcome contributions!

Spring-session is a very cool new project that aims to provide a simpler way of managing sessions in Java based web applications. One of the features that I explored with spring-session recently was the way it supports externalizing session state without needing to fiddle with the internals of specific web containers like Tomcat or Jetty. To test spring-session I have used a shopping cart type application(available here) which makes heavy use of session by keeping the items added to the cart as a session attribute, as can be seen from these screenshots: Consider first a scenario without Spring session. So this is how I have exposed my application: I am using nginx to load balance across two instances of this application. This set-up is very easy to run using Spring boot, I brought up two instances of the app up using two different server ports, this way: mvn spring-boot:run -Dserver.port=8080 mvn spring-boot:run -Dserver.port=8082 and this is my nginx.conf to load balance across these two instances: events { worker_connections 1024; } http { upstream sessionApp { server localhost:8080; server localhost:8082; } server { listen 80; location / { proxy_pass http://sessionApp; } } } I display the port number of the application in the footer just to show which instance is handling the request. If I were to do nothing to move the state of the session out the application then the behavior of the application would be erratic as the session established on one instance of the application would not be recognized by the other instance - specifically if Tomcat receives a session id it does not recognize then the behavior is to create a new session. Introducing Spring session into the application There are container specific ways to introduce a external session stores - One example is here, where Redis is configured as a store for Tomcat. PivotalGemfire provides a module to externalize Tomcat's session state. The advantage of using Spring-session is that there is no dependence on the container at all - maintaining session state becomes an application concern. The instructions on configuring an application to use Spring session is detailed very well at the Spring-session site, just to quickly summarize how I have configured my Spring Boot application, these are first the dependencies that I have pulled in: org.springframework.session spring-session 1.0.0.BUILD-SNAPSHOT org.springframework.session spring-session-data-redis 1.0.0.BUILD-SNAPSHOT org.springframework.data spring-data-redis 1.4.1.RELEASE redis.clients jedis 2.4.1 and my configuration to use Spring-session for session support, note the Spring Boot specific FilterRegistrationBean which is used to register the session repository filter: import org.springframework.boot.context.embedded.FilterRegistrationBean; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.annotation.Order; import org.springframework.data.redis.connection.jedis.JedisConnectionFactory; import org.springframework.session.data.redis.config.annotation.web.http.EnableRedisHttpSession; import org.springframework.session.web.http.SessionRepositoryFilter; import org.springframework.web.filter.DelegatingFilterProxy; import java.util.Arrays; @Configuration @EnableRedisHttpSession public class SessionRepositoryConfig { @Bean @Order(value = 0) public FilterRegistrationBean sessionRepositoryFilterRegistration(SessionRepositoryFilter springSessionRepositoryFilter) { FilterRegistrationBean filterRegistrationBean = new FilterRegistrationBean(); filterRegistrationBean.setFilter(new DelegatingFilterProxy(springSessionRepositoryFilter)); filterRegistrationBean.setUrlPatterns(Arrays.asList("/*")); return filterRegistrationBean; } @Bean public JedisConnectionFactory connectionFactory() { return new JedisConnectionFactory(); } } And that is it! magically now all session is handled by Spring-session, and neatly externalized to Redis. If I were to retry my previous configuration of using nginx to load balance two different Spring-Boot applications using the common Redis store, the application just works irrespective of the instance handling the request. I look forward to further enhancements to this excellent new project. The sample application which makes use of Spring-session is available here: https://github.com/bijukunjummen/shopping-cart-cf-app.git

Nowadays it has become mandatory to Gzipping the APIs response due to huge amount of data we are sending in response. It saves network bandwidth and delivery time and off course space over the internet. While using CXF; it provides an option to use the Gzip Compression in no of ways. Blueprint Annotation Blueprint: Annotation: First you need to register the GZIPOutInterceptor in out interceptors list. For that you need to hook into CXF initialization classes. public class InterceptorManager extends AbstractFeature { private static final Logger LOGGER = Logger.getLogger( "simcore" ); private static final Interceptor< Message > GZIP = new GZIPOutInterceptor(); //private static final Interceptor< Message > GZIP = new GZIPOutInterceptor(512); /* (non-Javadoc) * @see org.apache.cxf.feature.AbstractFeature#initializeProvider(org.apache.cxf.interceptor.InterceptorProvider, org.apache.cxf.Bus) */ @Override protected void initializeProvider( InterceptorProvider provider, Bus bus ) { /** * Adding Gzip interceptor to all outbound requests/responses */ LOGGER.debug( " ############## Adding Gzip as OUT Interceptor ##############" ); provider.getOutInterceptors().add( GZIP ); } } GZIPOutInterceptor comes with an option to set the Threshold value as no of Bytes. If response size will be below this threshold value then it will not be compressed. It is extremely useful when we will be sending empty lists and status messages/codes only. Because compressing those small responses will be overhead at server side. But there is another factor which is no of users requesting the response. So set this value by thinking over all the cases in mind. @GZIP Now we can use this annotation on any of our web-services controller to implement compression on all the APIs provided in that class. @WebService @Consumes ( { MediaType.TEXT_PLAIN, MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON } ) @Produces ( MediaType.APPLICATION_JSON ) @GZIP public interface WebServicesController { @GET @Path ( "/myGzipData" ) @Produces ( { MediaType.APPLICATION_JSON } ) Response getZipData( ); } Moreover we can set different parameters in Gzip annotation. @GZIP ( force = true, threshold = 512 )

there is currently a strong trend for microservice based architectures and frequent discussions comparing them to monoliths. there is much advice about breaking-up monoliths into microservices and also some amusing fights between proponents of the two paradigms - see the great microservices vs monolithic melee . the term 'monolith' is increasingly being used as a generic insult in the same way that 'legacy' is! however, i believe that there is a great deal of misunderstanding about exactly what a 'monolith' is and those discussing it are often talking about completely different things. a monolith can be considered an architectural style or a software development pattern (or anti-pattern if you view it negatively). styles and patterns usually fit into different viewtypes (a viewtype is a set, or category, of views that can be easily reconciled with each other [clements et al., 2010]) and some basic viewtypes we can discuss are: module - the code units and their relation to each other at compile time. allocation - the mapping of the software onto its environment. runtime - the static structure of the software elements and how they interact at runtime. a monolith could refer to any of the basic viewtypes above. module monolith if you have a module monolith then all of the code for a system is in a single codebase that is compiled together and produces a single artifact. the code may still be well structured (classes and packages that are coherent and decoupled at a source level rather than a big-ball-of-mud) but it is not split into separate modules for compilation. conversely a non-monolithic module design may have code split into multiple modules or libraries that can be compiled separately, stored in repositories and referenced when required. there are advantages and disadvantages to both but this tells you very little about how the code is used - it is primarily done for development management. allocation monolith for an allocation monolith, all of the code is shipped/deployed at the same time. in other words once the compiled code is 'ready for release' then a single version is shipped to all nodes. all running components have the same version of the software running at any point in time. this is independent of whether the module structure is a monolith. you may have compiled the entire codebase at once before deployment or you may have created a set of deployment artifacts from multiple sources and versions. either way this version for the system is deployed everywhere at once (often by stopping the entire system, rolling out the software and then restarting). a non-monolithic allocation would involve deploying different versions to individual nodes at different times. this is again independent of the module structure as different versions of a module monolith could be deployed individually. runtime monolith a runtime monolith will have a single application or process performing the work for the system (although the system may have multiple, external dependencies). many systems have traditionally been written like this (especially line-of-business systems such as payroll, accounts payable, cms etc). whether the runtime is a monolith is independent of whether the system code is a module monolith or not. a runtime monolith often implies an allocation monolith if there is only one main node/component to be deployed (although this is not the case if a new version of software is rolled out across regions, with separate users, over a period of time). note that my examples above are slightly forced for the viewtypes and it won't be as hard-and-fast in the real world. conclusion be very carefully when arguing about 'microservices vs monoliths'. a direct comparison is only possible when discussing the runtime viewtype and properties. you should also not assume that moving away from a module or allocation monolith will magically enable a microservice architecture (although it will probably help). if you are moving to a microservice architecture then i'd advise you to consider all these viewtypes and align your boundaries across them i.e. don't just code, build and distribute a monolith that exposes subsets of itself on different nodes.

When it comes to SaaS maturity model, maturity is not an all-or-nothing proposition, as a SaaS application can possess one or two important attributes and still manage to fit the typical definition and meet the essential business requirements. So in that case the app architects may choose not to meet or fulfill other attributes, especially if in so doing the action would be rendered cost ineffective. Broadly speaking, SaaS maturity can be demonstrated using a delivery model with 4 distinct levels, with each level distinguished from all other previous ones by simply adding one, two or more attributes. The four levels are briefly described below. Level I: Custom/ Ad Hoc This level of SaaS maturity resembles the conventional ASP (application service provider) software delivery model with its origin in the 1990s. At level I, each client has his/her own personalized version of a hosted application, which he/she runs an instance of the software app on the host’s servers. In terms of architecture, software at level I maturity closely resembles traditional line-of-business software sold earlier on, in that multiple clients or customers within a single organization are able to form a type of connection to a single instance running on the server. However, the instance if fully independent of other processes or instances that the host runs on behalf of all its other clients. Typically, conventional client-server apps can be relocated to a cloud-based model usually at the initial level of maturity, and with lesser development effort or without having to re-architect the whole system by building it from scratch/ ground up. While this level has few benefits of a typically mature SaaS solution, vendors can reduce costs by simply consolidating server hardware, administration, etc. Level II: Configurable This is the 2nd level SaaS maturity is where your SaaS vendor hosts a totally different instance of the SaaS application for each tenant or customer. While each instance is personally customized for each tenant, all instances at this level utilize similar code implementation. Moreover, the vendor meets the needs or requirements of the customer by offering in-depth configuration options that enable the customer to alter the look of the application as well as its behavior to its users. And while they resemble one another, particularly at the code-level, every instance remains completely isolated from the others. Migrating to a code base for clients of a vendor significantly reduces the service requirements an application, as any changes effected on the code base maybe issued to all customers of the vendor at once without upgrading or performing slipstream customized instances. In a SaaS maturity model, repositioning a conventional application as cloud-based at the maturity level may require additional re-architecting compared to the previous level, especially if this application has specially been designed for personal customization instead of configuration metadata. Just as the first level, level II requires the vendor to offer sufficient hardware or storage to accommodate multiple application instances running parallel/ concurrently. Level III (Multi-Tenant-Efficient and Configurable) Level III maturity is characterized by the vendor running a single instance serving each client with configurable metadata to provide unique, customized user experience and unique feature set. Security and authorization policies ensure the safety of each customer’s data, which is kept separate for every customer. In fact, there’s no clear indication that the instance is shared among multiple users/ tenants (from the tenant’s perspective). This eliminates the need for server space to accommodate the many instances, allowing for efficient use of scarce computing resources than level II, thus, translating to lower costs. However, a notable disadvantage of this particular approach is application’s scalability, which is limited. So unless database performance is managed by partitioning, the application may be scaled by scaling up (moving to a much more powerful server), until diminishing returns render it more costly to add extra power. Level IV (Scalable, Multi-Tenant-Efficient, Configurable) This is the final or the last level of maturity where the vendor hosts several clients on a load-balanced group of identical instances, but with each client’s data stored separate, and configurable metadata offering each customer a phenomenal user experience and unique feature set. A SaaS system can be scaled to a large number of clients, as the number of instances and servers on the backend can be adjusted to meet demand without you having to re-architect the application. Moreover, fixes or changes can be easily rolled out to multiple tenants just as easily as with a single tenant. Choosing an Appropriate Maturity Level (targeting a maturity level for your application) While you might expect this fourth or final level to act as the long-run goal for your SaaS application, this is not always the case. In fact, it could be more useful to view the maturity of SaaS as a continuum (progression of elements) between isolated data +code in one hand, and shared data+ code on another. But where your application falls along this continuum will largely depend on your business, operational and architectural needs, as well as customer considerations. Scalability Thousands of people can use large-scale software simultaneously. Anyone with experience developing enterprise applications knows the challenges of developing a scalable architecture. Scalability is a crucial aspect of a typical SaaS application as you are developing a unique internet-scale system that will actively support a broad user base that could potentially reach millions of users. Applications (in SaaS) can be quickly scaled up (moved to a larger and more powerful computer server) as well as scaled out (run on more servers). At the cloud-based level, scaling out is considered the best option for extra capacity, as portrayed in SaaS maturity model, because a properly-designed SaaS app can be easily scaled out easily to a large number of computer servers, with each running one, two, or more similar instances of that application. Conclusion SaaS represents an architectural model that is built on the foundation of massive scalability, multi-tenant efficiency,as well as metadata-driven configurability to provide great software inexpensively to both existing and potential clients. Adopting these principles can help in implementing SaaS maturity model and place you on the right path to completely transforming the manner in which you depict the long-tail business.

We can run a Gradle build without any of the task actions being executed. This is a so-called dry run of our build. We can use the dry run of a build to see if the task dependencies we have defined or are defined in a plugin are defined properly. Because all tasks and task dependencies are resolved if we use the dry run mode we can see in the output which tasks are executed. We define a simple build file with three tasks and some task dependencies: def printTaskNameAction = { println "Running ${it.name}" } task first << printTaskNameAction task second(dependsOn: first) << printTaskNameAction task third(dependsOn: [first, second]) << printTaskNameAction To run a Gradle build as a dry run we can use the command line option -m or --dry-run. So let's execute the task third with the dry run command line option: $ gradle -m third :first SKIPPED :second SKIPPED :third SKIPPED BUILD SUCCESSFUL Total time: 2.242 secs $ And we see in the output none of the tasks are really executed, because SKIPPED is shown, but we do see the task names of the tasks that are resolved. Written with Gradle 2.2.

There may be cases when your REST api provides responses that are very long, and we all know how important transfer speed and bandwidth still are on mobile devices/networks. I think this is the first performance optimization point one needs to address, when developing REST apis that support mobile apps. Guess what? Because responses are text, we can compress them. And with today’s power of smartphones and tablets uncompressing them on the client side should not be a big deal… So in this post I will present how you can SELECTIVELY compress your REST API responses, if you’ve built it in Java with Jersey, which is the JAX-RS Reference Implementation (and more)… 1. Jersey filters and interceptors Well, thanks to Jersey’s powerful Filters and Interceptors features, the implementation is fairly easy. Whereas filters are primarily intended to manipulate request and response parameters like HTTP headers, URIs and/or HTTP methods, interceptors are intended to manipulate entities, via manipulating entity input/output streams. You’ve seen the power of filters in my posts How to add CORS support on the server side in Java with Jersey, where I’ve shown how to CORS-enable a REST API and How to log in Spring with SLF4J and Logback, where I’ve shown how to log requests and responses from the REST API , but for compressing will be using a GZip WriterInterceptor. A writer interceptor is used for cases where entity is written to the “wire”, which on the server side as in this case, means when writing out a response entity. 1.1. GZip Writer Interceptor So let’s have a look at our GZip Writer Interceptor: package org.codingpedia.demo.rest.interceptors; import java.io.IOException; import java.io.OutputStream; import java.util.zip.GZIPOutputStream; import javax.ws.rs.WebApplicationException; import javax.ws.rs.core.MultivaluedMap; import javax.ws.rs.ext.WriterInterceptor; import javax.ws.rs.ext.WriterInterceptorContext; @Provider @Compress public class GZIPWriterInterceptor implements WriterInterceptor { @Override public void aroundWriteTo(WriterInterceptorContext context) throws IOException, WebApplicationException { MultivaluedMap headers = context.getHeaders(); headers.add("Content-Encoding", "gzip"); final OutputStream outputStream = context.getOutputStream(); context.setOutputStream(new GZIPOutputStream(outputStream)); context.proceed(); } } Note: it implements the WriterInterceptor, which is an interface for message body writer interceptors that wrap around calls to javax.ws.rs.ext.MessageBodyWriter.writeTo providers implementing WriterInterceptor contract must be either programmatically registered in a JAX-RS runtime or must be annotated with @Provider annotation to be automatically discovered by the JAX-RS runtime during a provider scanning phase. @Compress is the name binding annotation, which we will discuss more detailed in the coming paragraph “The interceptor gets a output stream from the WriterInterceptorContext and sets a new one which is a GZIP wrapper of the original output stream. After all interceptors are executed the output stream lastly set to the WriterInterceptorContext will be used for serialization of the entity. In the example above the entity bytes will be written to the GZIPOutputStream which will compress the stream data and write them to the original output stream. The original stream is always the stream which writes the data to the “wire”. When the interceptor is used on the server, the original output stream is the stream into which writes data to the underlying server container stream that sends the response to the client.” [2] “The overridden method aroundWriteTo() gets WriterInterceptorContext as a parameter. This context contains getters and setters for header parameters, request properties, entity, entity stream and other properties.” [2]; when you compress your response you should set the “Content-Encoding” header to “gzip” 1.2. Compress annotation Filters and interceptors can be name-bound. Name binding is a concept that allows to say to a JAX-RS runtime that a specific filter or interceptor will be executed only for a specific resource method. When a filter or an interceptor is limited only to a specific resource method we say that it is name-bound. Filters and interceptors that do not have such a limitation are called global. In our case we’ve built the @Compress annotation: package org.codingpedia.demo.rest.interceptors; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import javax.ws.rs.NameBinding; //@Compress annotation is the name binding annotation @NameBinding @Retention(RetentionPolicy.RUNTIME) public @interface Compress {} and used it to mark methods on resources which should be gzipped (e.g. when GET-ing all the podcasts with the PodcastsResource): @Component @Path("/podcasts") public class PodcastsResource { @Autowired private PodcastService podcastService; ........................... /* * *********************************** READ *********************************** */ /** * Returns all resources (podcasts) from the database * * @return * @throws IOException * @throws JsonMappingException * @throws JsonGenerationException * @throws AppException */ @GET @Compress @Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML }) public List getPodcasts( @QueryParam("orderByInsertionDate") String orderByInsertionDate, @QueryParam("numberDaysToLookBack") Integer numberDaysToLookBack) throws IOException, AppException { List podcasts = podcastService.getPodcasts( orderByInsertionDate, numberDaysToLookBack); return podcasts; } ........................... } 2. Testing 2.1. SOAPui Well, if you are testing with SOAPui, you can issue the following request against the PodcastsResource Request: GET http://localhost:8888/demo-rest-jersey-spring/podcasts/?orderByInsertionDate=DESC HTTP/1.1 Accept-Encoding: gzip,deflate Accept: application/json, application/xml Host: localhost:8888 Connection: Keep-Alive User-Agent: Apache-HttpClient/4.1.1 (java 1.5) Response: HTTP/1.1 200 OK Content-Type: application/json Content-Encoding: gzip Content-Length: 409 Server: Jetty(9.0.7.v20131107) [ { "id": 2, "title": "Quarks & Co - zum Mitnehmen", "linkOnPodcastpedia": "http://www.podcastpedia.org/quarks", "feed": "http://podcast.wdr.de/quarks.xml", "description": "Quarks & Co: Das Wissenschaftsmagazin", "insertionDate": "2014-10-29T10:46:13.00+0100" }, { "id": 1, "title": "- The Naked Scientists Podcast - Stripping Down Science", "linkOnPodcastpedia": "http://www.podcastpedia.org/podcasts/792/-The-Naked-Scientists-Podcast-Stripping-Down-Science", "feed": "feed_placeholder", "description": "The Naked Scientists flagship science show brings you a lighthearted look at the latest scientific breakthroughs, interviews with the world top scientists, answers to your science questions and science experiments to try at home.", "insertionDate": "2014-10-29T10:46:02.00+0100" } ] SOAPui recognizes the Content-Type: gzip header, we’ve added in the GZIPWriterInterceptor and automatically uncompresses the response and displays it readable to the human eye. Well, that’s it. You’ve learned how Jersey makes it straightforward to compress the REST api responses. Tip: If you want really learn how to design and implement REST API in Java read the following Tutorial – REST API design and implementation in Java with Jersey and Spring

NOTE: Apparently a bundle with a linked source will not be exported or built in an update site built. also Tycho will complain that it can't find linked sources, which severely limits the possibilities. A workaround is to export the bundles as plain old jars (this works fine for some reason), but the problem is far from ideal. See bug reports: https://bugs.eclipse.org/bugs/show_bug.cgi?id=457192 https://bugs.eclipse.org/bugs/show_bug.cgi?id=66177 Introduction I've been using Eclipse for more than ten years now, and so I like to think I know my way around it's offerings, but every now and then I get pleasantly surprised by discovering a feature – which usually had been there all along- but for which I finally have made the time to investigate. In this case, I am talking about the 'Link Source' feature in the Project Properties tab. Most experienced Eclipse users will at some point wander through the project properties, for instance when certain libraries are not found by the compiler, or when a plugin project starts to behave unexpectedly. The Project Properties tab comes into play when the Manifest.MF file no longer provides the answers for certain problems you face, and you need to delve deeper into the classpath and project settings. It also becomes topical when you need to make a custom project. At Project Chaupal we are currently maintaining and updating the code from Project JXTA. JXTA has been around for quite a bit in the open source community, and the development has had its ups and downs, so the code could do with a makeover here and there. I've been involved with keeping the code available in OSGI since 2006 or so – also with its ups and downs- and one of my ideals would be to automatically generate the OSGI bundles straight from the JXTA sources, without any handwork. The JXTA jar ships with a large number of third party libraries (e.g. Jetty and Log4j), some of which are available as OSGI bundles, so I don't want to include them in the JXTA OSGI bundle I make. A list of dependencies in the Manifest should be enough! Some of the third party libraries also aim to provide the same functionality (e.g. the database functionality provided by Derby and H2), so I would prefer to divide this over two bundles, and then just select the bundle that is needed. Ever since JXTA 2.6, the code has been mavenised, and so the code now conforms to the typical structure that Maven requires, with a specific location for JAVA code (src/main/java), resources (src/main/resources) and tests (src/main/test). I prefer to use Tycho for my OSGI bundles, so the regular Eclipse tooling is leading. As a result my goal is to: Create separate bundles for the core and the test source code Add the required resources, such as .properties files and the likes Split the core source code over different bundles, so that every bundle depends on one third party library at the most. It took me a day or two to get everything the way I wanted it, but in the end it was surprisingly easy, so I thought I'd share the experience.This tutorial assumes that you are well-versed in Eclipse and OSGI development. If not, a good tutorial on the subject can be found here. Preparation As was described earlier, the plan is to use a Maven project (available on GitHub) as a source for a number of OSGI bundles. For starters, we need to do the following; Prepare an Eclipse IDE with Egit and, optionally, with support for GitHub. As always, Lars Vogel's tutorial provides an excellent guide to achieve this. With the GitHub support you can actually search for the required repository, and clone it in your workspace within minutes. Add Maven Support for Eclipse. The tutorial can be found here. We now have an Eclipse IDE with one project loaded in the workspace, which conforms to the Maven structure. Now we can start to do the magic! Extracting Source Files in an OSGI Project First create a new plugin project, using the wizard (File → new → Plugin Project). Fill in the required details as requested (target platform→OSGI framework) and press 'finish'. We now have a standard textbook OSGI bundle project. For the sake of argument, let's call this bundle org.mybundle.host. Now we are going to add java source files and resources from the Maven project: open the project properties tab (right mouse click → properties) select the 'Java build path' option and choose the 'source' tab press 'Link Source' and browse to the 'main/java' subfolder of the Maven project. Close the project properties Include the source folder in the build.properties file and clean the workspace Update the Manifest to include the required dependencies, and export the packages as needed As you can see, the java files have been included in the bundle project, and will compile in a normal fashion. TIP: Currently the source file will by default have the same name as the folder. You can change this in the 'link source' wizard. For instance, you can delete the 'src' folder that is created by default, and replace it with the linked source if you want. This should only be considered if you are not going to make specific java files in the bundle. Next we are going to include the resources, such as .properties files that are included in the Maven project. As an exercise, we will exclude all html files that may be included. Open the project properties and follow the steps described previously, but now select the main/resources folder. Then press the 'next' tab, instead of closing You can now select which files to include or exclude in your bundle. Select the 'exclude' tab, and enter the following pattern: **/*.htm*. Close the project properties, update the build path and clean the project We have now included the desired resources, and in principle the bundle should now work as desired. With the include and exclude tabs, you can determine which files and folders you want to add to your bundle. The inclusion and exclusion patterns follow the conventions used by Apache ANT. TIP: You can check if the bundle has the correct source and resource files by opening a file explorer (in Windows) and browsing to the 'bin' folder of your bundle project. If you first refresh your bundle project (F5 in the Eclipse IDE), the correct class and resource files should be present there. NOTE: Although I would not recommend it, it is possible to link the sources of multiple non-OSGI code sources this way. Even though the folders need different names, they will be built as if they are one source folder. Now we create a second plugin project for the test files. We will call this bundle org.mybundle.test Open the project properties and follow the steps described previously, but now select the main/test folder. If required, you can exclude certain tests in the 'exclude' tab Close the project properties, update the build path and clean the project In the manifest editor, include the dependencies to org.mybundle.host, and for instance JUnit and JMock. When there are no more compiler errors, your two bundles should behave as regular OSGI bundles, with the only difference that the sources are extracted from the Maven project. TIP: It is also possible to make fragment bundles this way, and you can include library resources in your bundle (such as third party jars). This way you can restructure a non-OSGI project at will Using Variables When you store your projects in the cloud, such as on gitHub, you may often find multiple versions of your workspace scattered over different computers, and your repositories stored on different drives. This means that linking your sources with absolute paths, as we have done previously, is not a very versatile approach. Especially with linking this may become problematic, as the linked source project (e.g. the maven project) can be stored on a different location than the project that uses the source files. Luckily eclipse allows you to define variables in your project, which can help either to standardise the relative locations or, if this is not possible, to easily modify the links. In order to achive this, follow these steps: Select the project properties, and select the 'Java Build Path' option. Add or Edit a link source, and select the 'variables' button. Add one or more new variables by pressing the 'New' button and entering the locations. Then press 'OK' As an example, the following three variables point to two GitHub projects, one which holds the Maven project, while the bundle project is located in the same subfolder: - GITHUB_LOC: C:/Users/MyName/MyGithubLocation - MYPROJECT_LOC: ${GITHUB_LOC}/MyProject - MYSOURCEPROJECT_LOC: ${MYPROJECT_LOC}/MySource Now all you have to do is change the 'Linked folder location' to: MYSOURCEPROJECT_LOC/src/main/java in order to include the Maven project in your bundle.You can then also add a resource folder: MYSOURCEPROJECT_LOC/src/main/resources TIP: In the above example with a Maven project, the linked folder name will default to 'java' (and the resources to 'resources'). It is recommended to leave it that way, because you can then use the 'src' folder for bundle specific code that yuo may want to add, like a decalarative service. Also remember to update the build path to include the folders your project needs. Conclusion The 'link source' option provides a powerful way to make non-OSGI code accessible as OSGI bundles. The inclusion and exclusion patterns allow you to customise the bundles to your needs.

Originally Written by Peiran Song A deadlock in MySQL happens when two or more transactions mutually hold and request for locks, creating a cycle of dependencies. In a transaction system, deadlocks are a fact of life and not completely avoidable. InnoDB automatically detects transaction deadlocks, rollbacks a transaction immediately and returns an error. It uses a metric to pick the easiest transaction to rollback. Though an occasional deadlock is not something to worry about, frequent occurrences call for attention. Before MySQL 5.6, only the latest deadlock can be reviewed using SHOW ENGINE INNODB STATUS command. But with Percona Toolkit’s pt-deadlock-logger you can have deadlock information retrieved from SHOW ENGINE INNODB STATUS at a given interval and saved to a file or table for late diagnosis. For more information on using pt-deadlock-logger, see this post. With MySQL 5.6, you can enable a new variable innodb_print_all_deadlocks to have all deadlocks in InnoDB recorded in mysqld error log. Before and above all diagnosis, it is always an important practice to have the applications catch deadlock error (MySQL error no. 1213) and handle it by retrying the transaction. How to diagnose a MySQL deadlock A MySQL deadlock could involve more than two transactions, but the LATEST DETECTED DEADLOCK section only shows the last two transactions. Also it only shows the last statement executed in the two transactions, and locks from the two transactions that created the cycle. What are missed are the earlier statements that might have really acquired the locks. I will show some tips on how to collect the missed statements. Let’s look at two examples to see what information is given. Example 1: 1 141013 6:06:22 2 *** (1) TRANSACTION: 3 TRANSACTION 876726B90, ACTIVE 7 sec setting auto-inc lock 4 mysql tables in use 1, locked 1 5 LOCK WAIT 9 lock struct(s), heap size 1248, 4 row lock(s), undo log entries 4 6 MySQL thread id 155118366, OS thread handle 0x7f59e638a700, query id 87987781416 localhost msandbox update 7 INSERT INTO t1 (col1, col2, col3, col4) values (10, 20, 30, 'hello') 8 *** (1) WAITING FOR THIS LOCK TO BE GRANTED: 9 TABLE LOCK table `mydb`.`t1` trx id 876726B90 lock mode AUTO-INC waiting 10 *** (2) TRANSACTION: 11 TRANSACTION 876725B2D, ACTIVE 9 sec inserting 12 mysql tables in use 1, locked 1 13 876 lock struct(s), heap size 80312, 1022 row lock(s), undo log entries 1002 14 MySQL thread id 155097580, OS thread handle 0x7f585be79700, query id 87987761732 localhost msandbox update 15 INSERT INTO t1 (col1, col2, col3, col4) values (7, 86, 62, "a lot of things"), (7, 76, 62, "many more") 16 *** (2) HOLDS THE LOCK(S): 17 TABLE LOCK table `mydb`.`t1` trx id 876725B2D lock mode AUTO-INC 18 *** (2) WAITING FOR THIS LOCK TO BE GRANTED: 19 RECORD LOCKS space id 44917 page no 529635 n bits 112 index `PRIMARY` of table `mydb`.`t2` trx id 876725B2D lock mode S locks rec but not gap waiting 20 *** WE ROLL BACK TRANSACTION (1) Line 1 gives the time when the deadlock happened. If your application code catches and logs deadlock errors,which it should, then you can match this timestamp with the timestamps of deadlock errors in application log. You would have the transaction that got rolled back. From there, retrieve all statements from that transaction. Line 3 & 11, take note of Transaction number and ACTIVE time. If you log SHOW ENGINE INNODB STATUS output periodically(which is a good practice), then you can search previous outputs with Transaction number to hopefully see more statements from the same transaction. The ACTIVE sec gives a hint on whether the transaction is a single statement or multi-statement one. Line 4 & 12, the tables in use and locked are only with respect to the current statement. So having 1 table in use does not necessarily mean that the transaction involves 1 table only. Line 5 & 13, this is worth of attention as it tells how many changes the transaction had made, which is the “undo log entries” and how many row locks it held which is “row lock(s)”. These info hints the complexity of the transaction. Line 6 & 14, take note of thread id, connecting host and connecting user. If you use different MySQL users for different application functions which is another good practice, then you can tell which application area the transaction comes from based on the connecting host and user. Line 9, for the first transaction, it only shows the lock it was waiting for, in this case the AUTO-INC lock on table t1. Other possible values are S for shared lock and X for exclusive with or without gap locks. Line 16 & 17, for the second transaction, it shows the lock(s) it held, in this case the AUTO-INC lock which was what TRANSACTION (1) was waiting for. Line 18 & 19 shows which lock TRANSACTION (2) was waiting for. In this case, it was a shared not gap record lock on another table’s primary key. There are only a few sources for a shared record lock in InnoDB: 1) use of SELECT … LOCK IN SHARE MODE 2) on foreign key referenced record(s) 3) with INSERT INTO… SELECT, shared locks on source table The current statement of trx(2) is a simple insert to table t1, so 1 and 3 are eliminated. By checking SHOW CREATE TABLE t1, you could confirm that the S lock was due to a foreign key constraint to the parent table t2. Example 2: With MySQL community version, each record lock has the record content printed: 1 2014-10-11 10:41:12 7f6f912d7700 2 *** (1) TRANSACTION: 3 TRANSACTION 2164000, ACTIVE 27 sec starting index read 4 mysql tables in use 1, locked 1 5 LOCK WAIT 3 lock struct(s), heap size 360, 2 row lock(s), undo log entries 1 6 MySQL thread id 9, OS thread handle 0x7f6f91296700, query id 87 localhost ro ot updating 7 update t1 set name = 'b' where id = 3 8 *** (1) WAITING FOR THIS LOCK TO BE GRANTED: 9 RECORD LOCKS space id 1704 page no 3 n bits 72 index `PRIMARY` of table `tes t`.`t1` trx id 2164000 lock_mode X locks rec but not gap waiting 10 Record lock, heap no 4 PHYSICAL RECORD: n_fields 5; compact format; info bit s 0 11 0: len 4; hex 80000003; asc ;; 12 1: len 6; hex 000000210521; asc ! !;; 13 2: len 7; hex 180000122117cb; asc ! ;; 14 3: len 4; hex 80000008; asc ;; 15 4: len 1; hex 63; asc c;; 16 17 *** (2) TRANSACTION: 18 TRANSACTION 2164001, ACTIVE 18 sec starting index read 19 mysql tables in use 1, locked 1 20 3 lock struct(s), heap size 360, 2 row lock(s), undo log entries 1 21 MySQL thread id 10, OS thread handle 0x7f6f912d7700, query id 88 localhost r oot updating 22 update t1 set name = 'c' where id = 2 23 *** (2) HOLDS THE LOCK(S): 24 RECORD LOCKS space id 1704 page no 3 n bits 72 index `PRIMARY` of table `tes t`.`t1` trx id 2164001 lock_mode X locks rec but not gap 25 Record lock, heap no 4 PHYSICAL RECORD: n_fields 5; compact format; info bit s 0 26 0: len 4; hex 80000003; asc ;; 27 1: len 6; hex 000000210521; asc ! !;; 28 2: len 7; hex 180000122117cb; asc ! ;; 29 3: len 4; hex 80000008; asc ;; 30 4: len 1; hex 63; asc c;; 31 32 *** (2) WAITING FOR THIS LOCK TO BE GRANTED: 33 RECORD LOCKS space id 1704 page no 3 n bits 72 index `PRIMARY` of table `tes t`.`t1` trx id 2164001 lock_mode X locks rec but not gap waiting 34 Record lock, heap no 3 PHYSICAL RECORD: n_fields 5; compact format; info bit s 0 35 0: len 4; hex 80000002; asc ;; 36 1: len 6; hex 000000210520; asc ! ;; 37 2: len 7; hex 17000001c510f5; asc ;; 38 3: len 4; hex 80000009; asc ;; 39 4: len 1; hex 62; asc b;; Line 9 & 10: The ‘space id’ is tablespace id, ‘page no’ gives which page the record lock is on inside the tablespace. The ‘n bits’ is not the page offset, instead the number of bits in the lock bitmap. The page offset is the ‘heap no’ on line 10, Line 11~15: It shows the record data in hex numbers. Field 0 is the cluster index(primary key). Ignore the highest bit, the value is 3. Field 1 is the transaction id of the transaction which last modified this record, decimal value is 2164001 which is TRANSACTION (2). Field 2 is the rollback pointer. Starting from field 3 is the rest of the row data. Field 3 is integer column, value 8. Field 4 is string column with character ‘c’. By reading the data, we know exactly which row is locked and what is the current value. What else can we learn from analysis? Since most MySQL deadlocks happen between two transactions, we could start the analysis based on that assumption. In Example 1, trx (2) was waiting on a shared lock, so trx (1) either held a shared or exclusive lock on that primary key record of table t2. Let’s say col2 is the foreign key column, by checking the current statement of trx(1), we know it did not require the same record lock, so it must be some previous statement in trx(1) that required S or X lock(s) on t2’s PK record(s). Trx (1) only made 4 row changes in 7 seconds. Then you learned a few characteristics of trx(1): it does a lot of processing but a few changes; changes involve table t1 and t2, a single record insertion to t2. These information combined with other data could help developers to locate the transaction. Where else can we find previous statements of the transactions? Besides application log and previous SHOW ENGINE INNODB STATUS output, you may also leverage binlog, slow log and/or general query log. With binlog, if binlog_format=statement, each binlog event would have the thread_id. Only committed transactions are logged into binlog, so we could only look for Trx(2) in binlog. In the case of Example 1, we know when the deadlock happened, and we know Trx(2) started 9 seconds ago. We can run mysqlbinlog on the right binlog file and look for statements with thread_id = 155097580. It is always good to then cross refer the statements with the application code to confirm. $ mysqlbinlog -vvv --start-datetime=“2014-10-13 6:06:12” --stop-datatime=“2014-10-13 6:06:22” mysql-bin.000010 > binlog_1013_0606.out With Percona Server 5.5 and above, you can set log_slow_verbosity to include InnoDB transaction id in slow log. Then if you have long_query_time = 0, you would be able to catch all statements including those rolled back into slow log file. With general query log, the thread id is included and could be used to look for related statements. How to avoid a MySQL deadlock There are things we could do to eliminate a deadlock after we understand it. – Make changes to the application. In some cases, you could greatly reduce the frequency of deadlocks by splitting a long transaction into smaller ones, so locks are released sooner. In other cases, the deadlock rises because two transactions touch the same sets of data, either in one or more tables, with different orders. Then change them to access data in the same order, in another word, serialize the access. That way you would have lock wait instead of deadlock when the transactions happen concurrently. – Make changes to the table schema, such as removing foreign key constraint to detach two tables, or adding indexes to minimize the rows scanned and locked. – In case of gap locking, you may change transaction isolation level to read committed for the session or transaction to avoid it. But then the binlog format for the session or transaction would have to be ROW or MIXED.

In the modern world of microservices it's important to provide strict and polyglot clients for your service. It's better if your API is self-documented. One of the best tools for it is Apache Thrift. I want to explain how to use it with my favorite platform for microservices - Spring Boot. All project source code is available on GitHub: https://github.com/bsideup/spring-boot-thrift Project skeleton I will use Gradle to build our application. First, we need our main build.gradle file: buildscript { repositories { jcenter() } dependencies { classpath("org.springframework.boot:spring-boot-gradle-plugin:1.1.8.RELEASE") } } allprojects { repositories { jcenter() } apply plugin:'base' apply plugin: 'idea' } subprojects { apply plugin: 'java' } Nothing special for a Spring Boot project. Then we need a gradle file for thrift protocol modules (we will reuse it in next part): import org.gradle.internal.os.OperatingSystem repositories { ivy { artifactPattern "http://dl.bintray.com/bsideup/thirdparty/[artifact]-[revision](-[classifier]).[ext]" } } buildscript { repositories { jcenter() } dependencies { classpath "ru.trylogic.gradle.plugins:gradle-thrift-plugin:0.1.1" } } apply plugin: ru.trylogic.gradle.thrift.plugins.ThriftPlugin task generateThrift(type : ru.trylogic.gradle.thrift.tasks.ThriftCompileTask) { generator = 'java:beans,hashcode' destinationDir = file("generated-src/main/java") } sourceSets { main { java { srcDir generateThrift.destinationDir } } } clean { delete generateThrift.destinationDir } idea { module { sourceDirs += [file('src/main/thrift'), generateThrift.destinationDir] } } compileJava.dependsOn generateThrift dependencies { def thriftVersion = '0.9.1'; Map platformMapping = [ (OperatingSystem.WINDOWS) : 'win', (OperatingSystem.MAC_OS) : 'osx' ].withDefault { 'nix' } thrift "org.apache.thrift:thrift:$thriftVersion:${platformMapping.get(OperatingSystem.current())}@bin" compile "org.apache.thrift:libthrift:$thriftVersion" compile 'org.slf4j:slf4j-api:1.7.7' } We're using my Thrift plugin for Gradle. Thrift will generate source to the "generated-src/main/java" directory. By default, Thrift uses slf4j v1.5.8, while Spring Boot uses v1.7.7. It will cause an error in runtime when you will run your application, that's why we have to force a slf4j api dependency. Calculator service Let's start with a simple calculator service. It will have 2 modules: protocol and app.We will start with protocol. Your project should look as follows: calculator/ protocol/ src/ main/ thrift/ calculator.thrift build.gradle build.gradle settings.gradle thrift.gradle Where calculator/protocol/build.gradle contains only one line: apply from: rootProject.file('thrift.gradle') Don't forget to put these lines to settings.gradle, otherwise your modules will not be visible to Gradle: include 'calculator:protocol' include 'calculator:app' Calculator protocol Even if you're not familiar with Thrift, its protocol description file (calculator/protocol/src/main/thrift/calculator.thrift) should be very clear to you: namespace cpp com.example.calculator namespace d com.example.calculator namespace java com.example.calculator namespace php com.example.calculator namespace perl com.example.calculator namespace as3 com.example.calculator enum TOperation { ADD = 1, SUBTRACT = 2, MULTIPLY = 3, DIVIDE = 4 } exception TDivisionByZeroException { } service TCalculatorService { i32 calculate(1:i32 num1, 2:i32 num2, 3:TOperation op) throws (1:TDivisionByZeroException divisionByZero); } Here we define TCalculatorService with only one method - calculate. It can throw an exception of type TDivisionByZeroException. Note how many languages we're supporting out of the box (in this example we will use only Java as a target, though) Now run ./gradlew generateThrift, you will get generated Java protocol source in the calculator/protocol/generated-src/main/java/ folder. Calculator application Next, we need to create the service application itself. Just create calculator/app/ folder with the following structure: src/ main/ java/ com/ example/ calculator/ handler/ CalculatorServiceHandler.java service/ CalculatorService.java CalculatorApplication.java build.gradle Our build.gradle file for app module should look like this: apply plugin: 'spring-boot' dependencies { compile project(':calculator:protocol') compile 'org.springframework.boot:spring-boot-starter-web' testCompile 'org.springframework.boot:spring-boot-starter-test' } Here we have a dependency on protocol and typical starters for Spring Boot web app. CalculatorApplication is our main class. In this example I will configure Spring in the same file, but in your apps you should use another config class instead. package com.example.calculator; import com.example.calculator.handler.CalculatorServiceHandler; import org.apache.thrift.protocol.*; import org.apache.thrift.server.TServlet; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.context.annotation.*; import javax.servlet.Servlet; @Configuration @EnableAutoConfiguration @ComponentScan public class CalculatorApplication { public static void main(String[] args) { SpringApplication.run(CalculatorApplication.class, args); } @Bean public TProtocolFactory tProtocolFactory() { //We will use binary protocol, but it's possible to use JSON and few others as well return new TBinaryProtocol.Factory(); } @Bean public Servlet calculator(TProtocolFactory protocolFactory, CalculatorServiceHandler handler) { return new TServlet(new TCalculatorService.Processor(handler), protocolFactory); } } You may ask why Thrift servlet bean is called "calculator". In Spring Boot, it will register your servlet bean in context of the bean name and our servlet will be available at /calculator/. After that we need a Thrift handler class: package com.example.calculator.handler; import com.example.calculator.*; import com.example.calculator.service.CalculatorService; import org.apache.thrift.TException; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; @Component public class CalculatorServiceHandler implements TCalculatorService.Iface { @Autowired CalculatorService calculatorService; @Override public int calculate(int num1, int num2, TOperation op) throws TException { switch(op) { case ADD: return calculatorService.add(num1, num2); case SUBTRACT: return calculatorService.subtract(num1, num2); case MULTIPLY: return calculatorService.multiply(num1, num2); case DIVIDE: try { return calculatorService.divide(num1, num2); } catch(IllegalArgumentException e) { throw new TDivisionByZeroException(); } default: throw new TException("Unknown operation " + op); } } } In this example I want to show you that Thrift handler can be a normal Spring bean and you can inject dependencies in it. Now we need to implement CalculatorService itself: package com.example.calculator.service; import org.springframework.stereotype.Component; @Component public class CalculatorService { public int add(int num1, int num2) { return num1 + num2; } public int subtract(int num1, int num2) { return num1 - num2; } public int multiply(int num1, int num2) { return num1 * num2; } public int divide(int num1, int num2) { if(num2 == 0) { throw new IllegalArgumentException("num2 must not be zero"); } return num1 / num2; } } That's it. Well... almost. We still need to test our service somehow. And it should be an integration test. Usually, even if your application is providing JSON REST API, you still have to implement a client for it. Thrift will do it for you. We don't have to care about it. Also, it will support different protocols. Let's use a generated client in our test: package com.example.calculator; import org.apache.thrift.protocol.*; import org.apache.thrift.transport.THttpClient; import org.apache.thrift.transport.TTransport; import org.junit.*; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.*; import org.springframework.boot.test.IntegrationTest; import org.springframework.boot.test.SpringApplicationConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import org.springframework.test.context.web.WebAppConfiguration; import static org.junit.Assert.*; @RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration(classes = CalculatorApplication.class) @WebAppConfiguration @IntegrationTest("server.port:0") public class CalculatorApplicationTest { @Autowired protected TProtocolFactory protocolFactory; @Value("${local.server.port}") protected int port; protected TCalculatorService.Client client; @Before public void setUp() throws Exception { TTransport transport = new THttpClient("http://localhost:" + port + "/calculator/"); TProtocol protocol = protocolFactory.getProtocol(transport); client = new TCalculatorService.Client(protocol); } @Test public void testAdd() throws Exception { assertEquals(5, client.calculate(2, 3, TOperation.ADD)); } @Test public void testSubtract() throws Exception { assertEquals(3, client.calculate(5, 2, TOperation.SUBTRACT)); } @Test public void testMultiply() throws Exception { assertEquals(10, client.calculate(5, 2, TOperation.MULTIPLY)); } @Test public void testDivide() throws Exception { assertEquals(2, client.calculate(10, 5, TOperation.DIVIDE)); } @Test(expected = TDivisionByZeroException.class) public void testDivisionByZero() throws Exception { client.calculate(10, 0, TOperation.DIVIDE); } } This test will run your Spring Boot application, bind it to a random port and test it. All client-server communications will be performed in the same way real world clients are. Note how easy to use our service is from the client side. We're just calling methods and catching exceptions.

I was involved in a project recently where we needed to capture the http session id for a websocket request - the reason was to determine the number of websocket sessions utilizing the same underlying http session The way to do this is based on a sample utilizing the new spring-session module and is described here. The trick to capturing the http session id is in understanding that before a websocket connection is established between the browser and the server, there is a handshake phase negotiated over http and the session id is passed to the server during this handshake phase. Spring Websocket support provides a nice way to register a HandShakeInterceptor, which can be used to capture the http session id and set this in the sub-protocol(typically STOMP) headers. First, this is the way to capture the session id and set it to a header: public class HttpSessionIdHandshakeInterceptor implements HandshakeInterceptor { @Override public boolean beforeHandshake(ServerHttpRequest request, ServerHttpResponse response, WebSocketHandler wsHandler, Map attributes) throws Exception { if (request instanceof ServletServerHttpRequest) { ServletServerHttpRequest servletRequest = (ServletServerHttpRequest) request; HttpSession session = servletRequest.getServletRequest().getSession(false); if (session != null) { attributes.put("HTTPSESSIONID", session.getId()); } } return true; } public void afterHandshake(ServerHttpRequest request, ServerHttpResponse response, WebSocketHandler wsHandler, Exception ex) { } } And to register this HandshakeInterceptor with Spring Websocket support: @Configuration @EnableWebSocketMessageBroker public class WebSocketDefaultConfig extends AbstractWebSocketMessageBrokerConfigurer { @Override public void configureMessageBroker(MessageBrokerRegistry config) { config.enableSimpleBroker("/topic/", "/queue/"); config.setApplicationDestinationPrefixes("/app"); } @Override public void registerStompEndpoints(StompEndpointRegistry registry) { registry.addEndpoint("/chat").withSockJS().setInterceptors(httpSessionIdHandshakeInterceptor()); } @Bean public HttpSessionIdHandshakeInterceptor httpSessionIdHandshakeInterceptor() { return new HttpSessionIdHandshakeInterceptor(); } } Now that the session id is a part of the STOMP headers, this can be grabbed as a STOMP header, the following is a sample where it is being grabbed when subscriptions are registered to the server: @Component public class StompSubscribeEventListener implements ApplicationListener { private static final Logger logger = LoggerFactory.getLogger(StompSubscribeEventListener.class); @Override public void onApplicationEvent(SessionSubscribeEvent sessionSubscribeEvent) { StompHeaderAccessor headerAccessor = StompHeaderAccessor.wrap(sessionSubscribeEvent.getMessage()); logger.info(headerAccessor.getSessionAttributes().get("HTTPSESSIONID").toString()); } } or it can be grabbed from a controller method handling websocket messages as a MessageHeaders parameter: @MessageMapping("/chats/{chatRoomId}") public void handleChat(@Payload ChatMessage message, @DestinationVariable("chatRoomId") String chatRoomId, MessageHeaders messageHeaders, Principal user) { logger.info(messageHeaders.toString()); this.simpMessagingTemplate.convertAndSend("/topic/chats." + chatRoomId, "[" + getTimestamp() + "]:" + user.getName() + ":" + message.getMessage()); }

daniel bryant , simon and i recently had a discussion about how to represent system communication with external apis. the requirement for integration with external apis is now extremely common but it's not immediately obvious how to clearly show them in architectural diagrams. how to represent an external system? the first thing we discussed was what symbol to use for a system supplying an api. traditionally, uml has used the actor (stick man) symbol to represent a "user or any other system that interacts with the subject" (uml superstructure specification, v2.1.2). therefore a system providing an api may look like this: i've found that this symbol tends to confuse those who aren't well versed in uml as most people assume that the actor symbol always represents a *person* rather than a system. sometimes this is stereotyped to make it more obvious e.g. however the symbol is very powerful and tends to overpower the stereotype. therefore i prefer to use a stereotyped box for an external system supplying an api. let's compare two context diagrams using boxes vs stick actors. in which diagram is it more obvious what are systems or people? note that archimate has a specific symbol for application service that can be used to represent an api: (application service notation from the open group's archimate 2.1 specification) an api or the system that supplies it? whatever symbol we choose, what we've done is to show the *system* rather than the actual api. the api is a definition of a service provided by the system in question. how should we provide more details about the api? there are a number of ways we could do this but my preference is to give details of the api on the connector (line connecting two elements/boxes). in c4 the guidelines for a container diagram includes listing protocol information on the connector and an api can be viewed as the layer above the protocol. for example: multiple apis per external system many api providers supply multiple services/apis (i'm not referring to different operations within an api but multiple sets of operations in different apis, which may even use different underlying protocols.) for example a financial marketplace may have apis that do the following: allow a bulk, batch download of static data (such as details of companies listed on a stock market) via xml over http. supply real time, low latency updates of market prices via bespoke messages over udp. allow entry of trades via industry standard fpml over a queuing system. supply a bulk, batch download of trades for end-of-day reconciliation via fpml over http. two of the services use the same protocol (xml over http) but have very different content and use. one of the apis is used to constantly supply information after user subscription (market data) and the last service involves the user supplying all the information with no acknowledgment (although it should reconcile at eod). there are multiple ways of showing this. we could: have a single service element, list the apis on it and have all components linking to it. show each service/api as a separate box and connect the components that use the individual service to the relevant box. show a single service element with multiple connections. each connection is labeled and represents an api. use a port and connector style notation to represent each api from the service provider. provide a key for the ports. use a uml style 'cup and ball' notation to define interfaces and their usage. some examples are below: a single service element and simple description in the above diagram the containers are stating what they are using but contain no information about how to use the apis. we don't know if it is a single api (with different operations) or anything about the mechanisms used to transport the data. this isn't very useful for anyone implementing a solution or resolving operational issues. single, service box with descriptive connectors in this diagram there is a single, service box with descriptive connectors. the above diagram shows all the information so is much more useful as a diagnostic or implementation tool. however it does look quite crowded. services/apis shown as separate boxes here the external system has its services/apis shown as separate boxes. this contains all the information but might be mistaken as defining the internal structure of the external system. we want to show the services it provides but we know nothing about the internal structure. using ports to represent apis in the above diagram the services/apis are shown as 'ports' on the external system and the details have been moved into a separate key/table. this is less likely to be mistaken as showing any internal structure of the external service. (note that i could have also shown outgoing rports from the brokerage system.) uml interfaces this final diagram is using a uml style interface provider and requirer. this is a clean diagram but requires the user to be aware of what the cup and ball means (although i could have explained this in the key). conclusion any of these solutions could be appropriate depending on the complexity of the api set you are trying to represent. i'd suggest starting with a simple representation (i.e. fully labeled connections) and moving to a more complex one if needed but remember to use a key to explain any elements you use!

REST Easy with SQL/NoSQL Integration and CQRS Pattern implementation New demands are being put on IT organizations everyday to deliver agile, high-performance, integrated mobile and web applications. In the meantime, the technology landscape is getting complex everyday with the advent of new technologies like REST, NoSQL, Cloud while existing technologies like SOAP and SQL still rule everyday work. Rather than taking religious side of the debate, NoSQL can successfully co-exist with SQL in this ‘polyglot’ of data storage and formats. However, this integration also adds another layer of complexity both in architecture and implementation. This document offers a guide on how some of the relatively newer technologies like REST can help bridge the gap between SQL and NoSQL with an example of a well known pattern called CQRS. This document is organized as follows: Introduction to SQL development process NoSQL Do I have to choose between SQL and NoSQL? CQRS Pattern How to implement CQRS pattern using REST services Introduction to SQL development process Developers have been using SQL Databases for decades to build and deliver enterprise business applications. The process of creating tables, attributes,and relationships is second nature for most developers. Data architects think in terms of tables and columns and navigate relationships for data. The basic concepts of delivery and transformation takes place at the web server level which means the server developer is reading and ‘binding’ to the tables and mapping attributes to a REST response. Application development lifecycle meant changes to the database schema first, followed by the bindings, then internal schema mapping, and finally the SOAP or JSON services, and eventually the client code. This all costs the project time and money. It also means that the ‘code’ (pick your language here) and the business logic would also need to be modified to handle the changes to the model. NoSQL NoSQL is gaining supporters among many SQL shops for various reasons including: Low cost Ability to handle unstructured dataa Scalability Performance The first thing database folks notice is that there is no schema. These document style storage engines can handle huge volumes of structured, semi-structured, and unstructured data. The very nature of schema-less documents allows change to a document structure without having to go through the formal change management process (or data architect). The other major difference is that NoSQL (no-schema) also means no joins or relationships. The document itself contains the embedded information by design. So an order entry would contain the customer with all the orders and line items for each order in a single document. There are many different NoSQL vendors (popular NoSQL databases include MongoDB, Casandra) that are being used for BI and Analytics (read-only) purposes. We are also seeing many customers starting to use NoSQL for auditing, logging, and archival transactions. Do I have to choose between SQL and NoSQL? The purpose of this article is to not get into the religious debate about whether to use SQL or NoSQL. Bottom line is both have their place and are suited for certain type of data – SQL for structured data and NoSQL for unstructured data. So why not have the capability to mix and match this data depending on the application. This can be done by creating a single REST API across both SQL and NoSQL databases. Why a single REST API? The answer is simple – the new agile and mobile world demands this ‘mashup’ of data into a document style JSON response. CQRS (Command Query Responsibility Segmentation) Pattern There are many design patterns for delivery of high performance RESTful services but the one that stands out was described in an article written by Martin Fowler, one of the software industry veterans. He described the pattern called CQRS that is more relevant today in a ‘polyglot’ of servers, data, services, and connections. “We may want to look at the information in a different way to the record store, perhaps collapsing multiple records into one, or forming virtual records by combining information for different places. On the update side we may find validation rules that only allow certain combinations of data to be stored, or may even infer data to be stored that’s different from that we provide.” – Martin Fowler 2011 In this design pattern, the REST API requests (GET) return documents from multiple sources (e.g. mashups). In the update process, the data is subject to business logic derivations, validations, event processing, and database transactions. This data may then be pushed back into the NoSQL using asynchronous events. With the wide-spread adoption of NoSQL databases like MongoDB and schema-less, high capacity data store; most developers are challenged with providing security, business logic, event handling, and integration to other systems. MongoDB; one the popular NoSQL databases and SQL databases share many similar concepts. However the MongoDB programming language itself is very different from the SQL we all know. How to implement CQRS pattern using a RESTFul Architecture A REST server should meet certain requirements to support the CQRS pattern. The server should run on-premise or in the cloud and appears to the mobile and web developer as an HTTP endpoint. The server architecture should implement the following: Connections and Mapping necessary for SQL and NoSQL connectivity and API services needed to create and return GET, PUT, POST, and DELETE REST responses Security Business Logic Connections and Mapping There are two main approaches to creating REST Servers and APIs for SQL and NoSQL databases: Open source frameworks like Apache Tomcat, Spring/Hibernate Commercial framework like Espresso Logic Open source Frameworks Using various open source frameworks like Tomcat, Spring/Hibernate, Node.js, JDBC and MongoDB drivers, a REST server can be created, but we would still be left with the following tasks: Creation and mapping of the necessary SQL objects Create a REST server container and configurations Create Jersey/Jackson classes and annotations Create and define REST API for tables, views, and procedures Hand write validation, event and business logic Handle persistence, optimistic locking, transaction paging Adding identity management and security by roles Now we can start down the same path to connect to MongoDB and write code to connect, select, and return data in JSON and then create the REST calls to merge these two different document styles into a single RESTful endpoint. This is a lot of work for a development team to manage and control and frankly pretty boring and repetitive and is better done by a well designed framework Commercial Frameworks Many commercial frameworks may take care of this complexity without the need to do extensive programming. Here is an example from Espresso Logic and how it handles this complexity with a point and click interface: Running REST server in the cloud or on-premise Connections to external SQL databases Object mapping to tables, views, and procedures Automatic creation of RESTful endpoints from model Reactive business rules and rich event model Integrated role-based security and authentication services. Point-and-click document API creation for SQL and MongoDB endpoints In the example below, the editor shows an SQL (customersTransactions) joined with archived details from MongoDB (archivedTransactions). The MongoDB document for each customer may include transaction details, check images, customer service notes and other relevant account information. This new mashup becomes a single REST call that can be published to mobile and web application developer. Security Security is an important part of building and delivery of RESTful services which can be broken down into two parts; authentication and access control. Authentication Before allowing anyone access to corporate data you want to use the existing corporate identity management (some call this authentication services) to capture and validate the user. This identity management service is based on using existing corporate standards such as LDAP, Windows AD, SQL Database. Role-based Access Control Each user may be assigned one or more corporate roles and these roles are then assigned specific access privileges to each resource (e.g. READ, INSERT, UPDATE, and DELETE). Role-based access should also be able to restrict permissions to specific rows and columns of the API (e.g. only sales reps can see their own orders or a manager can see and change his department salaries but cannot change his own). This restriction should be applied regardless of how or where the API is used or called. Remember, the SQL database already provides some level of security and access which must be considered when designing and delivering new front-end services to internal and external users. Business Logic for REST When data is updated to a REST Server several things need to happen. First, the authentication and access control should determine if this is a valid request and if the user has rights to the endpoint. In addition, the server may need to de-alias REST attributes back to the actual SQL column names. In a full featured business logic server, there should be a series of events and business rules to perform various calculations, validations, and fire other events on dependent tables. Finally, the entire multi-table transaction is written back to the SQL database in a single transaction. Updates are then sent asynchronously to MongoDB as part of the commit event (after the SQL transaction has completed). Conclusion In the real-world of API services, the demand for more complex document style RESTful services is a requirement. That is, the ability to create ‘mashups’ of data from multiple tables, NoSQL collections, and other external systems is a large part of this new design pattern. In addition, the ability to alias attribute names and formats from these source fields has become critical for partners and customers systems. Using REST with the CQRS pattern to blend MongoDB and SQL seamlessly to your existing data will become a major part of your future mobile strategy. To implement these REST services, one can use open source tools and spend a lot of time or select a right commercial framework. This framework should support cloud or on-premise connectivity, security, API integration, as well as business logic. This will make the design and delivery of new application services more rapid and agile in the heterogeneous world of information.

[This article was written by Barak Merimovich.] We have discussed OpenStack networking extensively in previous posts. In this post, I’d like to dive into a more advanced OpenStack networking scenario. Many cloud images are not configured to automatically bring up all network cards that are available. They will usually only have a single network card configured. To correctly set up a host in the cloud with multiple network cards, log on to the machine and bring up the additional interfaces. echo $'auto eth1\niface eth1 inet dhcp' | sudo tee /etc/network/interfaces.d/eth1.cfg > /dev/null sudo ifup eth1 Networks in the cloud A complex network architecture is a mainstay of modern IaaS clouds. Understanding how to configure your cloud-based networks, and hosts, is critical to getting your application working in the cloud. This is especially true with Cloudify, the open source cloud orchestration platform I work on. The cloud, like the world, used to be flat It was not that long a time ago that most IaaS providers only supported flat networks – all of your hosts were in one large network. Separation between services running in the cloud was enforced in software or with firewalls/security-groups. But technically, all of the hosts were connected to the same network and visible to each other. The flat network model is simple, and therefore easy to reason and understand. It was a good choice for the early days of the IaaS cloud and no doubt helped with getting applications into the cloud in the first place. It was one of the things that made EC2 so easy to use for anyone just starting out with the ‘cloud’. This model is in fact still available on Amazon Web Services under the title ‘EC2-Classic’. And for many applications, a flat network is good enough. But as cloud adoption increases, more complex applications are moving into the clouds, and issues like network separation, security, SLA and broadcast domains make more complex networks models a must. Software Defined Networks (SDN) fill that gap. They are now a staple of most major IaaS clouds. AWS has AWS-VPC, OpenStack has the Neutron project and there are many other implementations. Working with SDN requires knowing a bit more about how information moves around between your cloud resources. In this post I am going to discuss how to set up a host in the cloud so it will play nice with complex networks. I’ll be using OpenStack, but the concepts are similar for other cloud infrastructures. Openstack configuration I am going to start with an empty tenant, only the public network is available. First, lets set up out networks and router: neutron router-create demo-router neutron net-create demo-network-1 neutron net-create demo-network-2 neutron subnet-create --name demo-subnet-1 demo-network-1 10.0.0.0/24 neutron subnet-create --name demo-subnet-2 demo-network-2 10.0.1.0/24 neutron router-interface-add demo-router demo-subnet-1 neutron router-interface-add demo-router demo-subnet-2 neutron router-gateway-set demo-router public Note the network IDs: neutron net-list | id | name | subnets | | 2c33efe2-6204-4125-9716-3bc525630016 | demo-network-1 | 928dafa0-83ef-459c-b20d-71d8ea596fa2 10.0.0.0/24 | | aa30627e-c181-4a4b-89bf-5dd7c26c244e | demo-network-2 | 26d573f7-7953-4a54-825b-ed7bbc0661c7 10.0.1.0/24 | | e502de8d-929a-4ee0-bd18-efa297875cf6 | public | d40dab51-a729-452c-9ee6-b9ad08d10808 | We’ll start with a standard Ubuntu cloud image: glance image-create --name "Ubuntu 12.04 Standard" --location "http://uec-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img" --disk-format qcow2 --container-format bare Create the keypair and security group: nova keypair-add demo-keypair > demo-keypair.pem chmod 400 demo-keypair.pem nova secgroup-create demo-security-group "Security group for demo" nova secgroup-add-rule demo-security-group tcp 22 22 0.0.0.0/0 Let’s spin up an instance connected to both our networks: nova boot -flavor m1.small --image "Ubuntu 12.04 Standard" --nic net-id=2c33efe2-6204-4125-9716-3bc525630016 --nic net-id=aa30627e-c181-4a4b-89bf-5dd7c26c244e --security-groups demo-security-group --key-name demo-keypair demo-vm And set up floating IPs for the first network: nova list | ID | Name | Status | Task State | Power State | Networks | 2b17588b-8980-4489-9a04-6539a159dc3c | demo-vm | ACTIVE | None | Running | demo-network-1=10.0.0.2; demo-network-2=10.0.1.2 | neutron floatingip-create public neutron floatingip-list | id | fixed_ip_address | floating_ip_address | port_id | | 49c8b05e-bb8f-4b07-80ed-3155ab6ffc09 | | 192.168.15.42 | | neutron port-list | id | name | mac_address | fixed_ips | | 1ccfd334-7328-4b22-b93e-24a0888276ab | | fa:16:3e:14:39:39 | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.3"} | | a482c4f6-fa74-476e-b1ce-cd8dd0c70815 | | fa:16:3e:18:92:79 | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.2"} | | b23d7836-30c5-4bff-b873-15c87ba051f6 | | fa:16:3e:3a:28:40 | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.3"} | | d421b447-2adf-406f-876b-142238683344 | | fa:16:3e:9d:fc:7f | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.2"} | | dcf8696b-cc80-4b48-b09c-61c0f8ab02ac | | fa:16:3e:5b:39:fb | {"subnet_id": "94598487-c1fc-4f55-ac1f-ef2545d5cfeb", "ip_address": "10.0.1.1"} | | f6a1666e-495a-4d3f-afa3-754b3cb3cfc0 | | fa:16:3e:8a:1b:fb | {"subnet_id": "dec6ec74-cfa9-4a08-8792-54900631b98e", "ip_address": "10.0.0.1"} | neutron floatingip-associate 49c8b05e-bb8f-4b07-80ed-3155ab6ffc09 d421b447-2adf-406f-876b-142238683344 Note how we matched the VM’s IP to its port, and associated the floating IP to the port. I wish there was an easier way to do this from the CLI… If everything worked correctly, you should have the following setup: Let’s make sure ssh works correctly: ssh -i demo-keypair.pem [email protected] hostname demo-vm Cool, ssh works. Now, we should have two network cards, right? ssh -i demo-keypair.pem [email protected] hostname demo-vm Cool, ssh works. Now, we should have two network cards, right? ssh -i demo-keypair.pem [email protected] ifconfig eth0 Link encap:Ethernet HWaddr fa:16:3e:5f:a2:5f inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe5f:a25f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:230 errors:0 dropped:0 overruns:0 frame:0 TX packets:224 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:46297 (46.2 KB) TX bytes:31130 (31.1 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Huh?! The VM only has one working network interface! Where is my second NIC? Was there a configuration problem with the OpenStack network setup? The answer is here: ssh -i demo-keypair.pem [email protected] ifconfig -a eth0 Link encap:Ethernet HWaddr fa:16:3e:5f:a2:5f inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe5f:a25f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:324 errors:0 dropped:0 overruns:0 frame:0 TX packets:332 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:69973 (69.9 KB) TX bytes:47218 (47.2 KB) eth1 Link encap:Ethernet HWaddr fa:16:3e:29:6d:22 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) The second NIC exists, but is not running. The issue is not with the OpenStack network configuration – it’s with the image. The image itself should be configured to work correctly with multiple NICs. All we have to do is bring up the NIC. So we ssh into the instance: ssh -i demo-keypair.pem [email protected] And run the following commands: echo $'auto eth1\niface eth1 inet dhcp' | sudo tee /etc/network/interfaces.d/eth1.cfg > /dev/null sudo ifup eth1 The second NIC should now be running: ifconfig eth1 eth1 Link encap:Ethernet HWaddr fa:16:3e:18:92:79 inet addr:10.0.1.2 Bcast:10.0.1.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe18:9279/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:81 errors:0 dropped:0 overruns:0 frame:0 TX packets:45 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:15376 (15.3 KB) TX bytes:3960 (3.9 KB) And there you go – your VM can access both networks. This issue can make life complicated when setting up a complex, or even a not very complex, application. When will this issue hurt you? Well, imagine a scenario where you have a web server and a database server. The web server is connected to both Network1 and Network2, and the database server is only connected to Network2. Network1 is connected to the external world over a router, and Network 2 is completely internal, adding another layer of security to the critical database server. So what happens if the web server only has one network card? If only the NIC for Network1 is up, the web server can’t access the database. If only the NIC for Network2 is up, the web server can’t be reached from the external world. Even worse, if this web server is accessed via a floating IP, this IP will also not work, so you won’t be able to access the web server and fix the issue. Tricky. In conclusion The above commands will bring up your additional network card. You will of-course need to repeat this process for each additional network card, and for each VM. You can use a start-up script (a.k.a. user-data script) or system service to run these commands, but there are better ways. I’ll discuss how to automate the network setup in a follow-up post. This was originally posted at Barak's blog Head in the Clouds, find it here.

The last couple of weeks I've been playing around with docker and kubernetes. If you are not familiar with kubernetes let's just say for now that its an open source container cluster management implementation, which I find really really awesome. One of the first things I wanted to try out was running an Apache ZooKeeper ensemble inside kubernetes and I thought that it would be nice to share the experience. For my experiments I used Docker v. 1.3.0 and Openshift V3, which I built from source and includes Kubernetes. ZooKeeper on Docker Managing a ZooKeeper ensemble is definitely not a trivial task. You usually need to configure an odd number of servers and all of the servers need to be aware of each other. This is a PITA on its own, but it gets even more painful when you are working with something as static as docker images. The main difficulty could be expressed as: "How can you create multiple containers out of the same image and have them point to each other?" One approach would be to use docker volumes and provide the configuration externally. This would mean that you have created the configuration for each container, stored it somewhere in the docker host and then pass the configuration to each container as a volume at creation time. I've never tried that myself, I can't tell if its a good or bad practice, I can see some benefits, but I can also see that this is something I am not really excited about. It could look like this: docker run -p 2181:2181 -v /path/to/my/conf:/opt/zookeeper/conf my/zookeeper An other approach would be to pass all the required information as environment variables to the container at creation time and then create a wrapper script which will read the environment variables, modify the configuration files accordingly, launch zookeeper. This is definitely easier to use, but its not that flexible to perform other types of tuning without rebuilding the image itself. Last but not least one could combine the two approaches into one and do something like: Make it possible to provide the base configuration externally using volumes. Use env and scripting to just configure the ensemble. There are plenty of images out there that take one or the other approach. I am more fond of the environment variables approach and since I needed something that would follow some of the kubernetes conventions in terms of naming, I decided to hack an image of my own using the env variables way. Creating a custom image for ZooKeeper I will just focus on the configuration that is required for the ensemble. In order to configure a ZooKeeper ensemble, for each server one has to assign a numeric id and then add in its configuration an entry per zookeeper server, that contains the ip of the server, the peer port of the server and the election port. The server id is added in a file called myid under the dataDir. The rest of the configuration looks like: server.1=server1.example.com:2888:3888 server.2=server2.example.com:2888:3888 server.3=server3.example.com:2888:3888 ... server.current=[bind address]:[peer binding port]:[election biding port]Note that if the server id is X the server.X entry needs to contain the bind ip and ports and not the connection ip and ports. So what we actually need to pass to the container as environment variables are the following: The server id. For each server in the ensemble: The hostname or ip The peer port The election port If these are set, then the script that updates the configuration could look like: if [ ! -z "$SERVER_ID" ]; then echo "$SERVER_ID" > /opt/zookeeper/data/myid #Find the servers exposed in env. for i in `echo {1..15}`;do HOST=`envValue ZK_PEER_${i}_SERVICE_HOST` PEER=`envValue ZK_PEER_${i}_SERVICE_PORT` ELECTION=`envValue ZK_ELECTION_${i}_SERVICE_PORT` if [ "$SERVER_ID" = "$i" ];then echo "server.$i=0.0.0.0:2888:3888" >> conf/zoo.cfg elif [ -z "$HOST" ] || [ -z "$PEER" ] || [ -z "$ELECTION" ] ; then #if a server is not fully defined stop the loop here. break else echo "server.$i=$HOST:$PEER:$ELECTION" >> conf/zoo.cfg fi done fi For simplicity the function that read the keys and values from env are excluded. The complete image and helping scripts to launch zookeeper ensembles of variables size can be found in the fabric8io repository. ZooKeeper on Kubernetes The docker image above, can be used directly with docker, provided that you take care of the environment variables. Now I am going to describe how this image can be used with kubernetes. But first a little rambling... What I really like about using kubernetes with ZooKeeper, is that kubernetes will recreate the container, if it dies or the health check fails. For ZooKeeper this also means that if a container that hosts an ensemble server dies, it will get replaced by a new one. This guarantees that there will be constantly a quorum of ZooKeeper servers. I also like that you don't need to worry about the connection string that the clients will use, if containers come and go. You can use kubernetes services to load balance across all the available servers and you can even expose that outside of kubernetes. Creating a Kubernetes confing for ZooKeeper I'll try to explain how you can create 3 ZooKeeper Server Ensemble in Kubernetes. What we need is 3 docker containers all running ZooKeeper with the right environment variables: { "image": "fabric8/zookeeper", "name": "zookeeper-server-1", "env": [ { "name": "ZK_SERVER_ID", "value": "1" } ], "ports": [ { "name": "zookeeper-client-port", "containerPort": 2181, "protocol": "TCP" }, { "name": "zookeeper-peer-port", "containerPort": 2888, "protocol": "TCP" }, { "name": "zookeeper-election-port", "containerPort": 3888, "protocol": "TCP" } ] } The env needs to specify all the parameters discussed previously. So we need to add along with the ZK_SERVER_ID, the following: ZK_PEER_1_SERVICE_HOST ZK_PEER_1_SERVICE_PORT ZK_ELECTION_1_SERVICE_PORT ZK_PEER_2_SERVICE_HOST ZK_PEER_2_SERVICE_PORT ZK_ELECTION_2_SERVICE_PORT ZK_PEER_3_SERVICE_HOST ZK_PEER_3_SERVICE_PORT ZK_ELECTION_3_SERVICE_PORT An alternative approach could be instead of adding all these manual configuration, to expose peer and election as kubernetes services. I tend to favor the later approach as it can make things simpler when working with multiple hosts. It's also a nice exercise for learning kubernetes. So how do we configure those services? To configure them we need to know: the name of the port the kubernetes pod the provide the service The name of the port is already defined in the previous snippet. So we just need to find out how to select the pod. For this use case, it make sense to have a different pod for each zookeeper server container. So we just need to have a label for each pod, the designates that its a zookeeper server pod and also a label that designates the zookeeper server id. "labels": { "name": "zookeeper-pod", "server": 1 } Something like the above could work. Now we are ready to define the service. I will just show how we can expose the peer port of server with id 1, as a service. The rest can be done in a similar fashion: { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-peer-1", "kind": "Service", "port": 2888, "containerPort": "zookeeper-peer-port", "selector": { "name": "zookeeper-pod", "server": 1 } } The basic idea is that in the service definition, you create a selector which can be used to query/filter pods. Then you define the name of the port to expose and this is pretty much it. Just to clarify, we need a service definition just like the one above per zookeeper server container. And of course we need to do the same for the election port. Finally, we can define an other kind of service, for the client connection port. This time we are not going to specify the sever id, in the selector, which means that all 3 servers will be selected. In this case kubernetes will load balance across all ZooKeeper servers. Since ZooKeeper provides a single system image (it doesn't matter on which server you are connected) then this is pretty handy. { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-client", "kind": "Service", "port": 2181, "createExternalLoadBalancer": "true", "containerPort": "zookeeper-client-port", "selector": { "name": "zookeeper-pod" } } The basic idea is that in the service definition, you create a selector which can be used to query/filter pods. Then you define the name of the port to expose and this is pretty much it. Just to clarify, we need a service definition just like the one above per zookeeper server container. And of course we need to do the same for the election port. Finally, we can define an other kind of service, for the client connection port. This time we are not going to specify the sever id, in the selector, which means that all 3 servers will be selected. In this case kubernetes will load balance across all ZooKeeper servers. Since ZooKeeper provides a single system image (it doesn't matter on which server you are connected) then this is pretty handy. { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-client", "kind": "Service", "port": 2181, "createExternalLoadBalancer": "true", "containerPort": "zookeeper-client-port", "selector": { "name": "zookeeper-pod" } } I hope you found it useful. There is definitely room for improvement so feel free to leave comments.

This article explains how errors are handled when using the messaging system with Spring Integration and how to handle route and redirect to specific channel.

if someone asked you to develop a rest api on the jvm, which frameworks would you use? i was recently tasked with such a project. my client asked me to implement a rest api to ingest requests from a 3rd party. the project entailed consuming xml requests, storing the data in a database, then exposing the data to internal application with a json endpoint. finally, it would allow taking in a json request and turning it into an xml request back to the 3rd party. with the recent release of apache camel 2.14 and my success using it , i started by copying my apache camel / cxf / spring boot project and trimming it down to the bare essentials. i whipped together a simple hello world service using camel and spring mvc. i also integrated swagger into both. both implementations were pretty easy to create ( sample code ), but i decided to use spring mvc. my reasons were simple: its rest support was more mature, i knew it well, and spring mvc test makes it easy to test apis. camel's swagger support without web.xml as part of the aforementioned spike, i learned out how to configure camel's rest and swagger support using spring's javaconfig and no web.xml. i made this into a sample project and put it on github as camel-rest-swagger . this article shows how i built a rest api with java 8, spring boot/mvc, jaxb and spring data (jpa and rest components). i stumbled a few times while developing this project, but figured out how to get over all the hurdles. i hope this helps the team that's now maintaining this project (my last day was friday) and those that are trying to do something similar. xml to java with jaxb the data we needed to ingest from a 3rd party was based on the ncpdp standards. as a member, we were able to download a number of xsd files, put them in our project and generate java classes to handle the incoming/outgoing requests. i used the maven-jaxb2-plugin to generate the java classes. org.jvnet.jaxb2.maven2 maven-jaxb2-plugin 0.8.3 generate -xtostring -xequals -xhashcode -xcopyable org.jvnet.jaxb2_commons jaxb2-basics 0.6.4 src/main/resources/schemas/ncpdp the first error i ran into was about a property already being defined. [info] --- maven-jaxb2-plugin:0.8.3:generate (default) @ spring-app --- [error] error while parsing schema(s).location [ file:/users/mraible/dev/spring-app/src/main/resources/schemas/ncpdp/structures.xsd{1811,48}]. com.sun.istack.saxparseexception2; systemid: file:/users/mraible/dev/spring-app/src/main/resources/schemas/ncpdp/structures.xsd; linenumber: 1811; columnnumber: 48; property "multipletimingmodifierandtimingandduration" is already defined. use to resolve this conflict. at com.sun.tools.xjc.errorreceiver.error(errorreceiver.java:86) i was able to workaround this by upgrading to maven-jaxb2-plugin version 0.9.1. i created a controller and stubbed out a response with hard-coded data. i confirmed the incoming xml-to-java marshalling worked by testing with a sample request provided by our 3rd party customer. i started with a curl command, because it was easy to use and could be run by anyone with the file and curl installed. curl -x post -h 'accept: application/xml' -h 'content-type: application/xml' \ --data-binary @sample-request.xml http://localhost:8080/api/message -v this is when i ran into another stumbling block: the response wasn't getting marshalled back to xml correctly. after some research, i found out this was caused by the lack of @xmlrootelement annotations on my generated classes. i posted a question to stack overflow titled returning jaxb-generated elements from spring boot controller . after banging my head against the wall for a couple days, i figured out the solution . i created a bindings.xjb file in the same directory as my schemas. this causes jaxb to generate @xmlrootelement on classes. to add namespaces prefixes to the returned xml, i had to modify the maven-jaxb2-plugin to add a couple arguments. -extension -xnamespace-prefix and add a dependency: org.jvnet.jaxb2_commons jaxb2-namespace-prefix 1.1 then i modified bindings.xjb to include the package and prefix settings. i also moved into a global setting. i eventually had to add prefixes for all schemas and their packages. i learned how to add prefixes from the namespace-prefix plugins page . finally, i customized the code-generation process to generate joda-time's datetime instead of the default xmlgregoriancalendar . this involved a couple custom xmladapters and a couple additional lines in bindings.xjb . you can see the adapters and bindings.xjb with all necessary prefixes in this gist . nicolas fränkel's customize your jaxb bindings was a great resource for making all this work. i wrote a test to prove that the ingest api worked as desired. @runwith(springjunit4classrunner.class) @springapplicationconfiguration(classes = application.class) @webappconfiguration @dirtiescontext(classmode = dirtiescontext.classmode.after_class) public class initiaterequestcontrollertest { @inject private initiaterequestcontroller controller; private mockmvc mockmvc; @before public void setup() { mockitoannotations.initmocks(this); this.mockmvc = mockmvcbuilders.standalonesetup(controller).build(); } @test public void testgetnotallowedonmessagesapi() throws exception { mockmvc.perform(get("/api/initiate") .accept(mediatype.application_xml)) .andexpect(status().ismethodnotallowed()); } @test public void testpostpainitiationrequest() throws exception { string request = new scanner(new classpathresource("sample-request.xml").getfile()).usedelimiter("\\z").next(); mockmvc.perform(post("/api/initiate") .accept(mediatype.application_xml) .contenttype(mediatype.application_xml) .content(request)) .andexpect(status().isok()) .andexpect(content().contenttype(mediatype.application_xml)) .andexpect(xpath("/message/header/to").string("3rdparty")) .andexpect(xpath("/message/header/sendersoftware/sendersoftwaredeveloper").string("hid")) .andexpect(xpath("/message/body/status/code").string("010")); } } spring data for jpa and rest with jaxb out of the way, i turned to creating an internal api that could be used by another application. spring data was fresh in my mind after reading about it last summer. i created classes for entities i wanted to persist, using lombok's @data to reduce boilerplate. i read the accessing data with jpa guide, created a couple repositories and wrote some tests to prove they worked. i ran into an issue trying to persist joda's datetime and found jadira provided a solution. i added its usertype.core as a dependency to my pom.xml: org.jadira.usertype usertype.core 3.2.0.ga ... and annotated datetime variables accordingly. @column(name = "last_modified", nullable = false) @type(type="org.jadira.usertype.dateandtime.joda.persistentdatetime") private datetime lastmodified; with jpa working, i turned to exposing rest endpoints. i used accessing jpa data with rest as a guide and was looking at json in my browser in a matter of minutes. i was surprised to see a "profile" service listed next to mine, and posted a question to the spring boot team. oliver gierke provided an excellent answer . swagger spring mvc's integration for swagger has greatly improved since i last wrote about it . now you can enable it with a @enableswagger annotation. below is the swaggerconfig class i used to configure swagger and read properties from application.yml . @configuration @enableswagger public class swaggerconfig implements environmentaware { public static final string default_include_pattern = "/api/.*"; private relaxedpropertyresolver propertyresolver; @override public void setenvironment(environment environment) { this.propertyresolver = new relaxedpropertyresolver(environment, "swagger."); } /** * swagger spring mvc configuration */ @bean public swaggerspringmvcplugin swaggerspringmvcplugin(springswaggerconfig springswaggerconfig) { return new swaggerspringmvcplugin(springswaggerconfig) .apiinfo(apiinfo()) .genericmodelsubstitutes(responseentity.class) .includepatterns(default_include_pattern); } /** * api info as it appears on the swagger-ui page */ private apiinfo apiinfo() { return new apiinfo( propertyresolver.getproperty("title"), propertyresolver.getproperty("description"), propertyresolver.getproperty("termsofserviceurl"), propertyresolver.getproperty("contact"), propertyresolver.getproperty("license"), propertyresolver.getproperty("licenseurl")); } } after getting swagger to work, i discovered that endpoints published with @repositoryrestresource aren't picked up by swagger. there is an open issue for spring data support in the swagger-springmvc project. liquibase integration i configured this project to use h2 in development and postgresql in production. i used spring profiles to do this and copied xml/yaml (for maven and application*.yml files) from a previously created jhipster project. next, i needed to create a database. i decided to use liquibase to create tables, rather than hibernate's schema-export. i chose liquibase over flyway based of discussions in the jhipster project . to use liquibase with spring boot is dead simple: add the following dependency to pom.xml, then place changelog files in src/main/resources/db/changelog . org.liquibase liquibase-core i started by using hibernate's schema-export and changing hibernate.ddl-auto to "create-drop" in application-dev.yml . i also commented out the liquibase-core dependency. then i setup a postgresql database and started the app with "mvn spring-boot:run -pprod". i generated the liquibase changelog from an existing schema using the following command (after downloading and installing liquibase). liquibase --driver=org.postgresql.driver --classpath="/users/mraible/.m2/repository/org/postgresql/postgresql/9.3-1102-jdbc41/postgresql-9.3-1102-jdbc41.jar:/users/mraible/snakeyaml-1.11.jar" --changelogfile=/users/mraible/dev/spring-app/src/main/resources/db/changelog/db.changelog-02.yaml --url="jdbc:postgresql://localhost:5432/mydb" --username=user --password=pass generatechangelog i did find one bug - the generatechangelog command generates too many constraints in version 3.2.2 . i was able to fix this by manually editing the generated yaml file. tip: if you want to drop all tables in your database to verify liquibase creation is working in postgesql, run the following commands: psql -d mydb drop schema public cascade; create schema public; after writing minimal code for spring data and configuring liquibase to create tables/relationships, i relaxed a bit, documented how everything worked and added a loggingfilter . the loggingfilter was handy for viewing api requests and responses. @bean public filterregistrationbean loggingfilter() { loggingfilter filter = new loggingfilter(); filterregistrationbean registrationbean = new filterregistrationbean(); registrationbean.setfilter(filter); registrationbean.seturlpatterns(arrays.aslist("/api/*")); return registrationbean; } accessing api with resttemplate the final step i needed to do was figure out how to access my new and fancy api with resttemplate . at first, i thought it would be easy. then i realized that spring data produces a hal -compliant api, so its content is embedded inside an "_embedded" json key. after much trial and error, i discovered i needed to create a resttemplate with hal and joda-time awareness. @bean public resttemplate resttemplate() { objectmapper mapper = new objectmapper(); mapper.configure(deserializationfeature.fail_on_unknown_properties, false); mapper.registermodule(new jackson2halmodule()); mapper.registermodule(new jodamodule()); mappingjackson2httpmessageconverter converter = new mappingjackson2httpmessageconverter(); converter.setsupportedmediatypes(mediatype.parsemediatypes("application/hal+json")); converter.setobjectmapper(mapper); stringhttpmessageconverter stringconverter = new stringhttpmessageconverter(); stringconverter.setsupportedmediatypes(mediatype.parsemediatypes("application/xml")); list> converters = new arraylist<>(); converters.add(converter); converters.add(stringconverter); return new resttemplate(converters); } the jodamodule was provided by the following dependency: com.fasterxml.jackson.datatype jackson-datatype-joda with the configuration complete, i was able to write a messagesapiitest integration test that posts a request and retrieves it using the api. the api was secured using basic authentication, so it took me a bit to figure out how to make that work with resttemplate. willie wheeler's basic authentication with spring resttemplate was a big help. @runwith(springjunit4classrunner.class) @contextconfiguration(classes = integrationtestconfig.class) public class messagesapiitest { private final static log log = logfactory.getlog(messagesapiitest.class); @value("http://${app.host}/api/initiate") private string initiateapi; @value("http://${app.host}/api/messages") private string messagesapi; @value("${app.host}") private string host; @inject private resttemplate resttemplate; @before public void setup() throws exception { string request = new scanner(new classpathresource("sample-request.xml").getfile()).usedelimiter("\\z").next(); responseentity response = resttemplate.exchange(gettesturl(initiateapi), httpmethod.post, getbasicauthheaders(request), org.ncpdp.schema.transport.message.class, collections.emptymap()); assertequals(httpstatus.ok, response.getstatuscode()); } @test public void testgetmessages() { httpentity request = getbasicauthheaders(null); responseentity> result = resttemplate.exchange(gettesturl(messagesapi), httpmethod.get, request, new parameterizedtypereference>() {}); httpstatus status = result.getstatuscode(); collection messages = result.getbody().getcontent(); log.debug("messages found: " + messages.size()); assertequals(httpstatus.ok, status); for (message message : messages) { log.debug("message.id: " + message.getid()); log.debug("message.datecreated: " + message.getdatecreated()); } } private httpentity getbasicauthheaders(string body) { string plaincreds = "user:pass"; byte[] plaincredsbytes = plaincreds.getbytes(); byte[] base64credsbytes = base64.encodebase64(plaincredsbytes); string base64creds = new string(base64credsbytes); httpheaders headers = new httpheaders(); headers.add("authorization", "basic " + base64creds); headers.add("content-type", "application/xml"); if (body == null) { return new httpentity<>(headers); } else { return new httpentity<>(body, headers); } } } to get spring data to populate the message id, i created a custom restconfig class to expose it. i learned how to do this from tommy ziegler . /** * used to expose ids for resources. */ @configuration public class restconfig extends repositoryrestmvcconfiguration { @override protected void configurerepositoryrestconfiguration(repositoryrestconfiguration config) { config.exposeidsfor(message.class); config.setbaseuri("/api"); } } summary this article explains how i built a rest api using jaxb, spring boot, spring data and liquibase. it was relatively easy to build, but required some tricks to access it with spring's resttemplate. figuring out how to customize jaxb's code generation was also essential to make things work. i started developing the project with spring boot 1.1.7, but upgraded to 1.2.0.m2 after i found it supported log4j2 and configuring spring data rest's base uri in application.yml. when i handed the project off to my client last week, it was using 1.2.0.build-snapshot because of a bug when running in tomcat . this was an enjoyable project to work on. i especially liked how easy spring data makes it to expose jpa entities in an api. spring boot made things easy to configure once again and liquibase seems like a nice tool for database migrations. if someone asked me to develop a rest api on the jvm, which frameworks would i use? spring boot, spring data, jackson, joda-time, lombok and liquibase. these frameworks worked really well for me on this particular project.

While the difference between SaaS VS ASP is quite significant, most people often confuse the two models because they are both “hosted.” However, Application Service Provider (ASP) is much closer to Legacy Software than Software-as-a-Service (SaaS). For many people, it is very difficult to differentiate whether a web-based solution is a SaaS offering or just another ASP-hosted instance of an on-site application. However, there is a significant difference as you are about to realize. Typically, ASP is used to provide computer-based services to clients over a network. It allows customers to gain access to an application program via a standard protocol, for instance, CRM over the Internet or via HTTP. One of the major advantages of ASPs is that small and medium-sized businesses can get specialized software. However, it is often unaffordable for many. It also eliminates the physical need for having to distribute the software, including the upgrades. On top of the software, ASPs will also maintain up-to-date services, including physical and electronic security, 24/7 technical support, and in-built support for continuity of business activities and promote flexible working. ASPs provide a way for businesses to outsource some or nearly all aspects of their IT needs. The following are 5 subcategories of ASPs: Enterprise ASPs – deliver not only high-end business applications, but also broad spectrum solutions Specialist ASPs – provide application to meet a specific business need, such as human resources, Web site services, credit card payment processing, etc. Local/Regional ASPs – supply various application services for small businesses within a limited area Volume Business ASPs – generally supply low cost prepackaged application services in volume – from their own site – to small and medium-sized businesses. A good example is PayPal Vertical Market ASPs – offer support to a particular industry, delivering a solution package for a certain type of customer, such as dental practice or healthcare in general. Similarly, SaaS – as a software deliver model – provides a platform in which software, as well as the associated data, is centrally hosted on the Cloud and the users can quickly access the software from their web browser. Just like ASPs, SaaS also provides software, available over the internet. Regardless, there are some minor differences between the two. Differences Between SaaS vs ASP Ideally, SaaS extends the ASP model idea. While ASPs try to focus on managing and hosting 3rd-party ISV software, SaaS vendors manage the software they have developed on their own. In addition, ASPs provide more traditional client-server applications, requiring installation of software on users’ PCs. On the other hand, SaaS rely solely on the Web and can be accessed via a web browser. Additionally, ASPs’ software architecture required that, for each business, you must maintain a separate instance of the application. However, SaaS does not maintain such requirements, as SaaS solutions use a multi-tenant architecture in which the application serves multiple users and businesses. Users access SaaS over the internet and it works in maintenance and service operation. Also, users pay per use and not as per a license, while the provider is responsible for maintenance and storage of data and business logic in the cloud. A major advantage of SaaS is that businesses can potentially reduce IT support costs by outsourcing their hardware and software maintenance and support needs to the SaaS provider. Typically, SaaS is mostly utilized as a delivery model for several business applications, including Office & Messaging software, Management software, DBMS software, Development software, CAD software, Virtualization, collaboration, accounting, human resource management (HRM), customer relationship management (CRM), enterprise resource planning (ERP), management information system (MIS), invoicing, service desk management, and content management (CM). However, ASP is a failed model because of the following reasons: It lacks scalability for the vendor No inbuilt aggregation of data Too much customization Generally a single revenue model No network effect data for collection and aggregation While there are some vendors who have found success with ASP model, this success has been limited due to issues of scalability and customization between systems. Always Up-to-Date With a SaaS offering, the software you are using will always be the most current, as your software providers always apply regular updates, maintenance, and latest enhancements. ASP, on the other hand, would require the provider to update each customer’s instance one by one, making regular maintenance and updates costly and time consuming. What this means for ASP customers is that necessary maintenance and market-driven enhancements are batched up and often delayed for months. Designed for the Web Since a typical SaaS offering is built from the ground up rather than being retrofitted for the web, it can take full advantage of today’s web capabilities. In ASP, vendors offer the choice between an on-site application and a web-based instance. Ending the SaaS VS ASP Confusion SaaS is an all-inclusive business architecture and a value delivery model other than a software delivery method. As explained earlier, SaaS is characterized by an inbuilt multi-tenancy, allowing for shared resources and infrastructure. Since SaaS is scalable, vendors can take full advantage of economies of scale to reduce complexities common with customizations, as well as reduce overall operational costs. So, the SaaS vendor enjoys many benefits, including the fact that resources are utilized efficiently and overall costs are reduced. This is because the inbuilt multi-tenancy of the SaaS business architecture can be leveraged to help reduce sales cycles and in turn accelerate revenue, improve customer service and retention, gain and maintain competitive advantage, directly monetize beyond the SaaS application, and improve strategic planning abilities. An application deployed in a single-tenant/ ASP model, in most cases, means the product does not support multi-tenancy, and the vendor is not willing to invest his/her time in re-architecting the product. It also means the product was not properly commercialized, but thought of as software rather than as a business, and probably lacks inbuilt revenue model support, billing, advanced metering, etc. Conclusion Between SaaS and ASP, the reality on the ground is that single-tenant applications, such as those in ASP models, are not architected properly to support the demanding business requirements surrounding the SaaS business architect. Overall, between SaaS vs ASP, an application designed and created specifically as a SaaS offering is safer if you want to use a web-based application, as it will also be easy to scale without incurring further costs.

In Parts 1 and 2 we have covered a number of common issues people run into when managing a sharded MongoDB cluster. In this final post of the series we will cover a subtle, but important distinction in terms of balancing a sharded cluster as well as an interesting limitation that can be worked around relatively easily, but is nonetheless surprising when it comes up. 6. Chunk balancing != data balancing != traffic balancing The balancer in a sharded cluster cares about just one thing: Are chunks for a given collection evenly balanced across all shards? If they are not, then it will take steps to rectify that imbalance. This all sounds perfectly logical, and even with extra complexity like tagging involved the logic is pretty straight forward. If we assume that all chunks are equal, then we can rest assured that our data is being evenly balanced across all the shards in our cluster and rest easy at night. Although that is sometimes, perhaps even frequently, the case it is not always true - chunks are not always equal. There can be massive “jumbo” chunks that exceed the maximum chunk size (64MiB), completely empty chunks and everything in between. Let’s use an example from our first pitfall, the monotonically increasing shard key. For our example, we have picked just such a key to shard on (date), and up until this point we have had just one shard and had not sharded the collection. We are about to add a second shard to our cluster and so we enable sharding on the collection and do the necessary admin work to add the new shard into the cluster. Once the collection is enabled for sharding, the first shard contains all the newly minted chunks. Let’s represent them in a simplified table of 10 chunks. This is not representative of a real data set, but it will do for illustrative purposes: Table 1 - Initial Chunk Layout Now we add our second shard. The balancer will kick in and attempt to distribute the chunks evenly. It will do this by moving the lowest range chunks to the new shard until the counts are identical. Once it is finished balancing, our table now looks like this: Table 2 - Balanced Chunk Layout That looks pretty good at the moment, but lets imagine that more recent chunks are more likely to have more activity (updates say) than older chunks. Adding the traffic share estimates for each chunk shows that shard1 is taking far more traffic (72%) than shard2 (28%) despite the chunks seeming balanced overall based on the approximate size. Hence, chunk balancing is not equal to traffic balancing. Using that same example, let’s add another wrinkle - periodic deletion of old data. Every 3 months we run a job to delete any data older than 12 months. Let’s look at the impact of that on our table after we run it for the first time (assuming the first run happens on July 1st 2015). Table 3 - Post-Delete Chunk Layout The distribution of data is now completely skewed toward shard1 - shard2 is in fact empty! However, the balancer is completely unaware of this imbalance - the chunk count has remained the same the entire time, and as far as it is concerned the system is in a steady state. With no data on shard2, our traffic imbalance as seen above will be even worse, and we have essentially negated the benefit of having a second shard for this collection. Possible Mitigation Strategies If data and traffic balance are important, select an appropriate shard key Move chunks manually to address the imbalances - swap “hot” chunks for “cool” chunks, empty chunks for larger chunks 7. Waiting too long to shard a collection (collection too large) This is not very common, but when it falls on your shoulders, it can be quite challenging to solve. There is a maximum data size for a collection when when it is initially split which is a function of the chunk size and data size as noted on the limits page. If your collection contains less than 256GiB of data, then there will be no issue. If the collection size exceeds 256GiB but is less than 400GiB, then MongoDB may be able to do an initial split without any special measures being taken. Otherwise, with larger initial data sizes and the default settings, the initial split will fail. It is worth noting that once split the collection may grow as needed and without any real limitations as long as you can continue to add shards as data size grows. Possible Mitigation Strategies Since the limit is dictated by the chunk size and the data size, and assuming there is not much to be done about the data size, then the remaining variable is the chunk size. This is adjustable (default is 64MiB) and can be raised in order to let a large collection split initially and then reduced once that has been completed. The required chunk size increase will depend on the actual data size. However, this is relatively easy to work out - simply divide your data size by 256GB and then multiply that figure by 64MiB (and round up if it is not a nice even number). As an example, let’s consider a 4TiB collection: 4TiB divided by 256GiB = 16 64MiB x 16 = 1024MiB Hence, set the max chunk size to 1024MiB, then perform the initial sharding of the collection, and then finally reduce the chunk size back to 64MiB using the same procedure. . Thanks for reading through the Sharding Pitfall series! If you want to learn more about managing MongoDB deployments at scale, sign up for my online education course, MongoDB Advanced Deployment and Operations. Planning for scale? No problem: MongoDB is here to help. Get a preview of what it’s like to work with MongoDB’s Technical Services Team. Give us some details on your deployment and we can set you up with an expert who can provide detailed guidance on all aspects of scaling with MongoDB, based on our experience with hundreds of deployments.