This is the first in a series of posts on the new “Red Deer” (https://github.com/jboss-reddeer/reddeer) open source testing framework for Eclipse. In this post, we’ll introduce Red Deer, and take a look at the some of the advantages that it offers by building a sample test program from scratch. Some of the features that Red Deer automated offers are: An easy to use, high-level API for testing standard Eclipse components Support for creating custom extensions for your own applications A requirements validation mechanism to assist you in configuring complex tests Eclipse Tooling to Assist in Creating new Projects A record and playback tool to enable you to quickly create automated tests An integration with Selenium for testing web based applications Support for running tests in a Jenkins CI environment Note that as of this writing, Red Deer is in an incubation stage. The current release is at level 0.5. The target date for the 1.0 release of Red Deer is late 2014. But, as a community-based, open source project, now is a great time to try Red Deer and make suggestions or even contribute code! A Look at Red Deer’s Architecture The Red Deer project itself is comprised of utilities and the API that supports the development and execution of automated tests. The API (the parts of the above diagram that are enclosed in dashed line boxes) can be thought of as having three layers: The top layer consists of extensions to Red Deer’s abstract classes or implementations for Eclipse components such as Views, Editors, Wizards, or Shells. For example, if you are writing tests for a feature that uses a custom Eclipse View, you can extend Red Deer’s View class by adding support for the specific functions of the feature. The advantage that this API layer gives you is that your test programs do not have to focus on manipulating the individual UI elements directly to perform operations. Your programs can instead instantiate an instance of an Eclipse component such as a View, and then use that instance’s methods to perform operations on the View. This layer of abstraction makes your test programs easier to write, understand, and maintain. The middle layer consists of the Red Deer implementations for SWT UI elements such as: Button, Combo, Label, Menu, Shell, TabItem, Table, ToolBar, Tree. This API layer supports the API’s higher level by providing the building blocks for the API’s Views, Editors, Shells, and WIzards. This middle layer of the API also provides Red Deer packages that enable your tests to enforce requirements, so that necessary setup tasks are performed before a test is run. The bottom layer consists of Red Deer packages that support the execution of tests such as: Conditions, Matchers, Widgets, Workbench, and Red Deer extensions to JUnit. What Makes Red Deer different from other Tools? A Layer of Abstraction The top-most layer of the API enables you to instantiate Eclipse UI elements as objects, and then manipulate them through their methods. The resulting code is easier to read and maintain, instead of being brittle and subject to failures when the UI changes. For example, for a test that has to open a view and press a button, without Red Deer, the test would have to navigate the top level menu, find the view menu, then the view type in that menu, then find the view open dialog, then locate the “OK” button, etc. Your test would have to spend a lot of time navigating through the UI elements before it could even begin to perform the test’s steps. With Red Deer, the code to open a view (in this case, the servers view) is simply: ServersView view = new ServersView(); view.open(); Furthermore, within that ServersView, your test program can perform operations on the View through methods which are defined in the view (and are incidentally also well debugged by the Red Deer team), instead of having to explicitly locate and manipulate the UI elements directly. For example, to obtain a list of all the servers, instead of locating the UI tree that contains the server list, and extracting that list of servers into an array, your Red Deer program can simply call the “getServers()” method. Likewise, the code to open a PackageExplorer, and then select a project within that PackageExplorer is as follows: PackageExplorer packageExplorer = new PackageExplorer(); packageExplorer.open(); packageExplorer.getProject("myTestProject").select(); And, the code to retrieve all the projects within that PackageExplorer is simply: packageExplorer.getProjects(); The result are that your tests are easier to write and maintain and you can focus on testing your application’s logic instead of writing brittle code to navigate through the application. Installing Red Deer The only prerequisites to using Red Deer are Eclipse and Java. In this post, we’ll use Eclipse Kepler and OpenJDK 1.7, running on Red Hat Enterprise Linux (RHEL) 6. To install Red Deer 0.4 (this is the latest stable milestone version as of this writing) follow these steps: Open up Eclipse Navigate to: Help->Install New Software Define a new download site using the Red Deer update site URL: http://download.jboss.org/jbosstools/updates/stable/kepler/core/reddeer/0.4.0/ Select Red Deer, click on the Finish button and Red Deer will install Now that you have Red Deer installed, let’s move onto building a new Red Deer test. Building your First Red Deer Test To create a new Red Deer test project, you make use of the Red Deer UI tooling and select New->Project->Other->Red Deer Test: Before we move on, let’s take a look at the WEB-INF/MANIFEST.MF file that is created in the project: Manifest-Version: 1.0 Bundle-ManifestVersion: 2 Bundle-Name: com.example.reddeer.sample Bundle-SymbolicName: com.example.reddeer.sample;singleton:=true Bundle-Version: 1.0.0.qualifier Bundle-ActivationPolicy: lazy Bundle-Vendor: Sample Co Bundle-RequiredExecutionEnvironment: JavaSE-1.6 Require-Bundle: org.junit, org.jboss.reddeer.junit, org.jboss.reddeer.swt, org.jboss.reddeer.eclipse The line we’re interested in is the final line in the file. These are the bundles that are required by Red Deer. After the empty project is created by the wizard, you can define a package and create a test class. Here's the code for a minimal functional test. The test will verify that the eclipse configuration is not empty. package com.example.reddeer.sample; import static org.junit.Assert.assertFalse; import java.util.List; import org.jboss.reddeer.swt.api.TreeItem; import org.jboss.reddeer.swt.impl.button.PushButton; import org.jboss.reddeer.swt.impl.menu.ShellMenu; import org.jboss.reddeer.swt.impl.tree.DefaultTree; import org.junit.Test; import org.junit.runner.RunWith; import org.jboss.reddeer.junit.runner.RedDeerSuite; @RunWith(RedDeerSuite.class) public class SimpleTest { @Test public void TestIt() { new ShellMenu("Help", "About Eclipse Platform").select(); new PushButton("Installation Details").click(); DefaultTree ConfigTree = new DefaultTree(); List ConfigItems = ConfigTree.getAllItems(); assertFalse ("The list is empty!", ConfigItems.isEmpty()); for (TreeItem item : ConfigItems) { System.out.println ("Found: " + item.getText()); } } } After you save the test's source file, you can run the test. To run the test, select the Run As->Red Deer Test option: And - there's the green bar! Simplifying Tests with Requirements Red Deer requirements enable you to define actions that you want happen before a test is executed. The advantage to using requirements is that you define the actions with annotations instead of using a @BeforeClass method. The result is that your test code is easier to read and maintain. The biggest difference between a Red Deer requirement and the the @BeforeClass annotation from the JUnit framework is that if a requirement cannot be fulfilled the test is not executed. Like everything else in Red Deer, you can make use of predefined requirements, or you can extend the feature by adding your own custom requirements. These custom requirements can be made complex and for convenience can be stored in external properties files. (We’ll take a look at defining custom requirements in a later post in this series when we examine how to create and contribute extensions to Red Deer.) The current milestone release of Red Deer provides predefined requirements that enable you to clean out your current workspace and open a perspective. Let’s add these to our example. To do this, we need to add these import statements: import org.jboss.reddeer.eclipse.ui.perspectives.JavaBrowsingPerspective; import org.jboss.reddeer.requirements.cleanworkspace.CleanWorkspaceRequirement.CleanWorkspace; import org.jboss.reddeer.requirements.openperspective.OpenPerspectiveRequirement.OpenPerspective; And these annotations: @CleanWorkspace @OpenPerspective(JavaBrowsingPerspective.class) And, we also have to a reference to org.jboss.reddeer.requirements to the required bundle list in our example’s MANIFEST.MF file: Require-Bundle: org.junit, org.jboss.reddeer.junit, org.jboss.reddeer.swt, org.jboss.reddeer.eclipse, org.jboss.reddeer.requirements When we’re done, our example looks like this: package com.example.reddeer.sample; import static org.junit.Assert.assertFalse; import java.util.List; import org.jboss.reddeer.swt.api.TreeItem; import org.jboss.reddeer.swt.impl.button.PushButton; import org.jboss.reddeer.swt.impl.menu.ShellMenu; import org.jboss.reddeer.swt.impl.tree.DefaultTree; import org.junit.Test; import org.junit.runner.RunWith; import org.jboss.reddeer.junit.runner.RedDeerSuite; import org.jboss.reddeer.eclipse.ui.perspectives.JavaBrowsingPerspective; import org.jboss.reddeer.requirements.cleanworkspace.CleanWorkspaceRequirement.CleanWorkspace; import org.jboss.reddeer.requirements.openperspective.OpenPerspectiveRequirement.OpenPerspective; @RunWith(RedDeerSuite.class) @CleanWorkspace @OpenPerspective(JavaBrowsingPerspective.class) public class SimpleTest { @Test public void TestIt() { new ShellMenu("Help", "About Eclipse Platform").select(); new PushButton("Installation Details").click(); DefaultTree ConfigTree = new DefaultTree(); List ConfigItems = ConfigTree.getAllItems(); assertFalse ("The list is empty!", ConfigItems.isEmpty()); for (TreeItem item : ConfigItems) { System.out.println ("Found: " + item.getText()); } } } Notice how we were able to add those functions to the test code, while only adding a very small amount of actual new code? Yes, it can pay to be a lazy programmer. ;-) What’s Next? What’s next for Red Deer is its continued development as it progresses through its incubation stage until its 1.0 release. What’s next for this series of posts will be discussions about: The Red Deer Recorder - To enable you to capture manual actions and convert them into test programs How you can Extend Red Deer - To provide test coverage for your plugins’ specific functions. And How you can Contribute these extensions to the Red Deer project. How you can Define Complex Requirements - To enable you to perform setup tasks for your tests. Red Deer’s Integration with Selenium - To enable you to test web interfaces provided by your plugins. Running Red Deer tests with Jenkins - To enable you to take advantage of Jenkins’ Continuous Integration (CI) test framework. Author’s Acknowledgements I’d like to thank all the contributors to Red Deer for their vision and contributions. It’s a new project, but it is growing fast! The contributors (in alphabetic order) are: Stefan Bunciak, Radim Hopp, Jaroslav Jankovic, Lucia Jelinkova, Marian Labuda, Martin Malina, Jan Niederman, Vlado Pakan, Jiri Peterka, Andrej Podhradsky, Milos Prchlik, Radoslav Rabara, Petr Suchy, and Rastislav Wagner.

If you need to process large database result sets from Java, you can opt for JDBC to give you the low level control required. On the other hand, if you are already using an ORM in your application, falling back to JDBC might imply some extra pain. You would be losing features such as optimistic locking, caching, automatic fetching when navigating the domain model and so forth. Fortunately most ORMs, like Hibernate, have some options to help you with that. While these techniques are not new, there are a couple of possibilities to choose from. A simplified example; let's assume we have a table (mapped to class "DemoEntity") with 100.000 records. Each record consists of a single column (mapped to the property "property" in DemoEntity) holding some random alphanumerical data of about ~2KB. The JVM is ran with -Xmx250m. Let's assume that 250MB is the overall maximum memory that can be assigned to the JVM on our system. Your job is to read all records currently in the table, doing some not further specified processing, and finally store the result. We'll assume that the entities resulting from our bulk operation are not modified. To start we'll try the obvious first, performing a query to simply retrieve all data: new TransactionTemplate(txManager).execute(new TransactionCallback() { @Override public Void doInTransaction(TransactionStatus status) { Session session = sessionFactory.getCurrentSession(); List demoEntitities = (List) session.createQuery("from DemoEntity").list(); for(DemoEntity demoEntity : demoEntitities){ //Process and write result } return null; } }); After a couple of seconds: Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded Clearly this won't cut it. To fix this we will be switching to Hibernate scrollable result sets as probably most developers are aware of. The above example instructs hibernate to execute the query, map the entire results to entities and return them. When using scrollable result sets records are transformed to entities one at a time: new TransactionTemplate(txManager).execute(new TransactionCallback() { @Override public Void doInTransaction(TransactionStatus status) { Session session = sessionFactory.getCurrentSession(); ScrollableResults scrollableResults = session.createQuery("from DemoEntity").scroll(ScrollMode.FORWARD_ONLY); int count = 0; while (scrollableResults.next()) { if (++count > 0 && count % 100 == 0) { System.out.println("Fetched " + count + " entities"); } DemoEntity demoEntity = (DemoEntity) scrollableResults.get()[0]; //Process and write result } return null; } }); After running this we get: ... Fetched 49800 entities Fetched 49900 entities Fetched 50000 entities Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded Although we are using a scrollable result set, every returned object is an attached object and becomes part of the persistence context (aka session). The result is actually the same as our first example in which we used "session.createQuery("from DemoEntity").list()". However, with that approach we had no control; everything happens behind the scenes and you get a list back with all the data if hibernate has done its job. using a scrollable result set on the other hand gives us a hook into the retrieval process and allows us to free memory up when needed. As we have seen it does not free up memory automatically, you have to instruct Hibernate to actually do it. Following options exist: Evicting the object from the persistent context after processing it Clearing the entire session every now and then We will opt for the first. In the above example under line 13 (//Process and write result) we'll add: session.evict(demoEntity); Important: If you were to perform any modification to the entity (or entities it has associations with that are cascade evicted alongside), make sure to flush the session PRIOR evicting or clearing, otherwise queries hold back because of Hibernate's write behind will not be sent to the database Evicting or clearing does not remove the entities from second level cache. If you enabled second level cache and are using it and you want to remove them as well use the desired sessionFactory.getCache().evictXxx() method From the moment you evict an entity it will be no longer attached (no longer associated with a session). Any modification done to the entity at that stage will no longer be reflected to the database automatically. If you are using lazy loading, accessing any property that was not loaded prior the eviction will yield the famous org.hibernate.LazyInitializationException. So basically, make sure the processing for that entity is done (or it is at least initialized for further needs) before you evict or clear After we run the application again, we see that it now successfully executes: ... Fetched 99800 entities Fetched 99900 entities Fetched 100000 entities Btw; you can also set the query read-only allowing hibernate to perform some extra optimizations: ScrollableResults scrollableResults = session.createQuery("from DemoEntity").setReadOnly(true).scroll(ScrollMode.FORWARD_ONLY); Doing this only gives a very marginal difference in memory usage, in this specific test setup it enabled us to read about 300 entities extra with the given amount of memory. Personally I would not use this feature merely for memory optimizations alone but only if it suits in your overall immutability strategy. With hibernate you have different options to make entities read-only: on the entity itself, the overall session read-only and so forth. Setting read only false on the query individually is probably the least preferred approach. (eg. entities loaded in the session before will remain unaffected, possibly modifiable. Lazy associations will be loaded modifiable even if the root objects returned by the query are read only). Ok, we were able to process our 100.000 records, life is good. But as it turns out Hibernate has another another option for bulk operations: the stateless session. You can obtain a scrollable result set from a stateless session the same way as from a normal session. A stateless session lies directly above JDBC. Hibernate will run in nearly "all features disabled" mode. This means no persistent context, no 2nd level caching, no dirty detection, no lazy loading, basically no nothing. From the javadoc: /** * A command-oriented API for performing bulk operations against a database. * A stateless session does not implement a first-level cache nor interact with any * second-level cache, nor does it implement transactional write-behind or automatic * dirty checking, nor do operations cascade to associated instances. Collections are * ignored by a stateless session. Operations performed via a stateless session bypass * Hibernate's event model and interceptors. Stateless sessions are vulnerable to data * aliasing effects, due to the lack of a first-level cache. For certain kinds of * transactions, a stateless session may perform slightly faster than a stateful session. * * @author Gavin King */ The only thing it does is transforming records to objects. This might be an appealing alternative because it helps you getting rid of that manual evicting/flushing: new TransactionTemplate(txManager).execute(new TransactionCallback() { @Override public Void doInTransaction(TransactionStatus status) { sessionFactory.getCurrentSession().doWork(new Work() { @Override public void execute(Connection connection) throws SQLException { StatelessSession statelessSession = sessionFactory.openStatelessSession(connection); try { ScrollableResults scrollableResults = statelessSession.createQuery("from DemoEntity").scroll(ScrollMode.FORWARD_ONLY); int count = 0; while (scrollableResults.next()) { if (++count > 0 && count % 100 == 0) { System.out.println("Fetched " + count + " entities"); } DemoEntity demoEntity = (DemoEntity) scrollableResults.get()[0]; //Process and write result } } finally { statelessSession.close(); } } }); return null; } }); Besides the fact that the stateless session has the most optimal memory usage, using the it has some side effects. You might have noticed that we are opening a stateless session and closing it explicitly: there is no sessionFactory.getCurrentStatelessSession() nor (at the time of writing) any Spring integration for managing the stateless session.Opening a stateless session allocates a new java.sql.Connection by default (if you use openStatelessSession()) to perform its work and therefore indirectly spawns a second transaction. You can mitigate these side effects by using the Hibernate work API as in the example which supplies the current Connection and pass it along to openStatelessSession(Connection connection). Closing the session in the finally has no impact on the physical connection since that is captured by the Spring infrastructure: only the logical connection handle is closed and a new logical connection handle was created when opening the stateless session. Also note that you have to deal with closing the stateless session yourself and that the above example is only good for read-only operations. From the moment you are going to modify using the stateless session there are some more caveats. As said before, hibernate runs in "all feature disabled" mode and as a direct consequence entities are returned in detached state. For each entity you modify, you'll have to call: statelessSession.update(entity) explicitly. First I tried this for modifying an entity: new TransactionTemplate(txManager).execute(new TransactionCallback() { @Override public Void doInTransaction(TransactionStatus status) { sessionFactory.getCurrentSession().doWork(new Work() { @Override public void execute(Connection connection) throws SQLException { StatelessSession statelessSession = sessionFactory.openStatelessSession(connection); try { DemoEntity demoEntity = (DemoEntity) statelessSession.createQuery("from DemoEntity where id = 1").uniqueResult(); demoEntity.setProperty("test"); statelessSession.update(demoEntity); } finally { statelessSession.close(); } } }); return null; } }); The idea is that we open a stateless session with the existing database Connection. As the StatelessSession javadoc indicates that no write behind occurs, I was convinced that each statement performed by the stateless session would be sent directly to the database. Eventually when the transaction (started by the TransactionTemplate) would be committed the results would become visible in the database. However, hibernate does BATCH statements using a stateless session. I'm not 100% sure what the difference is between batching and write behind, but the result is the same and thus contra dictionary with the javadoc as statements are queued and flushed at a later time. So, if you don't do anything special, statements that are batched will not be flushed and this is what happened in my case: the "statelessSession.update(demoEntity);" was batched and never flushed. One way to force the flush is to use the hibernate transaction API: StatelessSession statelessSession = sessionFactory.openStatelessSession(); statelessSession.beginTransaction(); ... statelessSession.getTransaction().commit(); ... While this works, you probably don't want to start controlling your transactions programatically just because you are using a stateless session. Also, doing this we are again running our stateless session work in a second transaction scenario since we didn't pass along our Connection and thus a new database connection will be acquired. The reason we can't pass along the outer Connection is because if we commit the inner transaction (the "stateless session transaction") and it would be using the same connection as the outer transaction (started by the TransactionTemplate) it would break the outer transaction atomicity as statements from the outer transaction sent to database would be committed along with the inner transaction. So not passing along the connections means opening a new connection and thus creating a second transaction. A better alternative would be just to trigger Hibernate to flush the stateless session. However, statelessSession has no "flush" method to manually trigger a flush. A solution here is to depend a bit on the Hibernate internal API. This solution makes the manual transaction handling and the second transaction obsolete: all statements become part of our (one and only) outer transaction: StatelessSession statelessSession = sessionFactory.openStatelessSession(connection); try { DemoEntity demoEntity = (DemoEntity) statelessSession.createQuery("from DemoEntity where id = 1").uniqueResult(); demoEntity.setProperty("test"); statelessSession.update(demoEntity); ((TransactionContext) statelessSession).managedFlush(); } finally { statelessSession.close(); } Fortunately there is an even better solution very recently posted on the Spring jira: https://jira.springsource.org/browse/SPR-2495 This is not yet part of Spring, but the factory bean implementation is pretty straight forward: StatelessSessionFactoryBean.java when using this you could simple inject the StatelessSession: @Autowired private StatelessSession statelessSession; It will inject a stateless session proxy which is equivalent to the way the normal "current" session works (with the minor difference that you inject a SessionFactory and need to obtain the currentSession each time). When the proxy is invoked it will lookup the stateless session bound to the running transaction. If none exists already it will create one with the same connection as the normal session (like we did in the example) and register a custom transaction synchronization for the stateless session. When the transaction is committed the stateless session is flushed thanks to the synchronization and finally closed. Using this you can inject the stateless session directly and use it as a current session (or the same way as you would inject a JPA PeristentContext for that matter). This relieves you from dealing with the opening and closing of the stateless session and having to deal with one way or the other to make it flush. The implementation is JPA aimed, but the JPA part is limited to obtaining the physical connection in obtainPhysicalConnection(). You can easily leave out the EntityManagerFactory and get the physical connection directly from the Hibernate session. Very careful conclusion: it is clear that the best approach will depend on your situation. If you use the normal session you will have to deal with eviction yourself when reading or persisting entities. Besides the fact you have to do this manually, it might also impact further use of the session if you have a mixed transaction; you both perform 'bulk' and 'normal' operations in the same transaction. If you continue with the normal operations you will have detached entities in your session which might lead to unexpected results (as dirty detection will no longer work and so forth). On the other hand you will still have the major hibernate benefits (as long as the entity isn't evicted) such as lazy loading, caching, dirty detection and the likes. Using the stateless session at the time of writing requires some extra attention on managing it (opening, closing and flushing) which can also be error prone. In the assumption you can proceed with the proposed factory bean, you have a very bare bone session which is separately from your normal session but still participating in the same transaction. With this you have a powerful tool to perform bulk operations without having to think about memory management. The downside is that you don't have any other hibernate functionality available.

what is codenvy exactly? well, their website states: codenvy is a cloud environment for coding, building, and debugging apps. basically, it’s an ide in the cloud (“ide as a service?”) accessible by all the major browsers . it started out as an additional feature to the exo platform in early 2009 and gained a lot of traction after the first paas (openshift) and git integration was added mid-2011. codenvy targets me as a (java) software developer to run and debug applications in their hosted cloud ide, while being able to share and collaborate during development and finally publish to a repository – e.g. git – or a number of deployment platforms – e.g. amazon, openshift or google app engine. i first encountered their booth at javaone last september, but they couldn’t demo their product right there on the spot over the wifi, because their on-line demo workspace never finished loading well i got the t-shirt instead then, but now’s the time to see what codenvy has in store as a cloud ide. signing up signing up took 3 seconds. all you have to do is go to codenvy.com , use the “sign up” button, choose an email address and a name for your workspace , confirm the email they’ll send you and you’re done. the “workspace” holds all your projects and is part of the url codenvy will create for you, like “ https://codenvy.com/ide/ . although not very clear during the registration process – which of course nowadays is usually minimalistic as can be – it seems that i’ve signed up for codenvy’s free community plan , which gives me an unlimited number of public projects. you can even start coding without registration. after confirming the registration mail, i’m in. finally i’ll end up in the browser where your (empty) workspace has been opened. empty workspace a few options a possible for here on, as seen in the figure above: create a new project from scratch – generate an empty project from predefined project types import from github – import projects from your github account clone a git repository – create a new project from any public git reposiroty browse documentation invite people – get team members on board support – questions, feedback and troubleshooting let’s… create a new project from scratch this option allows you to name the new project – e.g. “myproject”, choose a technology and a paas . the technology is a defined set of languages of frameworks to develop with. available technologies at the moment the technologies are: java jar java war java spring javascript ruby on rails python php node.js android maven multi-module at the time of writing java 1.6 is supported. available paas at the moment the available platforms are: amazon webservices (aws) elastic beanstalk savvis cloud appfrog cloudbees google app engine (gae) heroku manymo android emulator red hat’s openshift none depending on the choice of technology, or or more paas options become available. a single jar can not be deployed onto any of the platforms, leaving only the option “none” available. a java web application (war) can be deployed onto any number of platforms, except heroku and manymo. node.js can only be deployed to openshift. creating a simple jar project after having selected a jar (and no platform) one can select a project template . e.g. if webapplication (war) would have been selected, codenvy would present project templates, such as google app engine java project illustrating simple examples that use the search api , java web project with datasource usage or a demonstration of accessing amazon s3 buckets using the java sdk . the jar technology has only one project: simple jar project . after having finished the wizard, our jar project has been created in our workspace. we’ll see two views of our project: a project explorer and a package explorer. project- and package explorer what we can see is that our jar project has been given a maven pom.xml with the following content: view source print ? 01. < project xmlns = " http://maven.apache.org/pom/4.0.0 " xmlns:xsi = " http://www.w3.org/2001/xmlschema-instance " 02. xsi:schemalocation = " http://maven.apache.org/pom/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd " > 03. < modelversion >4.0.0 04. < groupid >com.codenvy.workspaceyug8g52wjwb5im13 05. < artifactid >testjarproject 06. < version >1.0-snapshot 07. < packaging >jar 08. 09. < name >sample-lib 10. 11. < properties > 12. < project.build.sourceencoding >utf-8 13. 14. 15. < dependencies > 16. < dependency > 17. < groupid >junit 18. < artifactid >junit 19. < version >3.8.1 20. < scope >test 21. 22. 23. we have a generated group id com.codenvy.workspaceyug8g52wjwb5im13 , our own artifact id and the junit dependency, which is a decent choice for many java developers use it as a testing framework. the source encoding has already been set to utf-8, which is also a sensible choice. as a convenience we’ve also been given a hello.sayhello class, so we know we’re actually in a java project say hello file & project management so what about the browser-based editor we’re working in? on top we’re seeing a few menu’s, like file, project, edit, view, run, git, paas, window, share and help . i’ll be highlighting a few. file- and project menu the file menu allows to creating folders , packages and various kind of filetypes , such as text, xml (1.0 at time of writing) , html (4.1) , css (2.0), java classes and jsp’s (2.1). although i’m in a jar project, i am still also able to create here e.g. ruby, php or python files. a very convenient feature is to upload existing files to the workspace, either separately or in zip archives. i’ve tried dropping a file onto the package explorer from the file system, but the browser (in this case, chrome) tries to open it instead the project menu allows to create new projects, either by launching the create project wizard again, but also allows for importing from github . in order to clone a repository, you’ll have to authorize codenvy to access github.com to be able to import a project. after having authorized github, codeenvy presents me with a list of projects to choose from. after having imported all necessary stuff, it somehow needs to know what kind of project i’m importing. selecting a file type after importing a project from github the project i imported didn’t give codenvy any clues as to what kind of project it is (which is right since i only had a readme.md in it), so it lists a few options to choose from. i chose the maven multi-module type after which the output window shows: [email protected]:tvinke/examples.git was successfully cloned. [info] project type updated. if you’d have a pom.xml in the root of your project, it would immediately recognize it a s a maven project. apart from going through the project > import from github option, you can also go directly to the git menu, and choose clone repository . this allows you to manually enter the remote repository uri, wanted project name and the remote name (e.g. “origin”). cloning a repository one you have pulled in a git project, the git menu allows all kinds of common operations, such as adding and removing files, committing, pushing, pulling and much more. git menu the ssh keys can be found under menu window > preferences where you can view the github.com entry, where one can view the details or delete it. also a new key can be either generated or uploaded here. sharing the project one of the unique selling points of codenvy are their collaboration possibilities which come along with any project. you can: invite other developers with read-only rights or full read-write rights to your workspace and every project in it.when you’re pair-programming like this, or co-editing a file with a colleague, you can also send each other code pointers – small shortcuts to code lines. use factories to create temporary workspaces , through cloning, off one source project (“factory”) and represent the cloning mechanism as a url which can be given to other developers. a use case might be to get a colleague quickly started on a project by providing a fully working development environment.there’s a lot more about creating factories in the docs (such as through rest), but the nice thing is that once you have a factory url, you can embed it as a button, send it through email of publish it somewhere for others! a factory url to load up e.g. their twitter bootstrap sample – as they use on their website themselves – looks like: https://codenvy.com/factory?v=1.0&pname=sample-twitterbootstrap&wname=codenvy-factories&vcs=git&vcsurl=http%3a%2f%2fcodenvy.com%2fgit%2f04%2f0f%2f7f%2fworkspacegcpv6cdxy1q34n1i%2fsample-twitterbootstrap&idcommit=c1443ecea63471f5797f172c081cd802bac6e6b0&action=openproject&ptype=javascript conclusion applications are run in the cloud nowadays, so why not create them there too? codenvy brings some interesting features, such as being able to instantly provision workspaces (through factory urls) and share projects in real-time. it supports common operations with projects, files and version control. with a slew of languages and platforms and as an ide being always accessible through the internet, it could lower the barrier to actually code anytime and anywhere. in a future post i will try and see whether or not it can actually replace my conventional desktop ide for java development.

Is Tomcat is a web server or an application server? Let me tell you how I convinced my self regarding this.

I first came across Reactive Extensions about 4 years ago on Matthew Podwysocki’s blog but then haven’t heard much about it until I saw Matthew give a talk at Code Mesh a few weeks ago. It seems to have grown in popularity recently and I noticed that’s there’s now a Java version called RxJavawritten by Netflix. I thought I’d give it a try by changing some code I wrote while exploring cypher’s MERGE function to expose an Observable instead of Futures. To recap, we have 50 threads and we do 100 iterations where we create random (user, event) pairs. We create a maximum of 10 users and 50 events and the goal is to concurrently send requests for the same pairs. In the example of my other post I was throwing away the result of each query whereas here I returned the result back so I had something to subscribe to. The outline of the code looks like this: public class MergeTimeRx { public static void main( final String[] args ) throws InterruptedException, IOException { String pathToDb = "/tmp/foo"; FileUtils.deleteRecursively( new File( pathToDb ) ); GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase( pathToDb ); final ExecutionEngine engine = new ExecutionEngine( db ); int numberOfThreads = 50; int numberOfUsers = 10; int numberOfEvents = 50; int iterations = 100; Observable events = processEvents( engine, numberOfUsers, numberOfEvents, numberOfThreads, iterations ); events.subscribe( new Action1() { @Override public void call( ExecutionResult result ) { for ( Map row : result ) { } } } ); .... } } The nice thing about using RxJava is that there’s no mention of how we got our collection of ExecutionResults, it’s not important. We just have a stream of them and by calling the subscribe function on the Observablewe’ll be informed whenever another one is made available. Most of the examples I found show how to generate events from a single thread but I wanted to use a thread pool so that I could fire off lots of requests at the same time. The processEvents method ended up looking like this: private static Observable processEvents( final ExecutionEngine engine, final int numberOfUsers, final int numberOfEvents, final int numberOfThreads, final int iterations ) { final Random random = new Random(); final List userIds = generateIds( numberOfUsers ); final List eventIds = generateIds( numberOfEvents ); return Observable.create( new Observable.OnSubscribeFunc() { @Override public Subscription onSubscribe( final Observer observer ) { final ExecutorService executor = Executors.newFixedThreadPool( numberOfThreads ); List> jobs = new ArrayList<>(); for ( int i = 0; i < iterations; i++ ) { Future job = executor.submit( new Callable() { @Override public ExecutionResult call() { Integer userId = userIds.get( random.nextInt( numberOfUsers ) ); Integer eventId = eventIds.get( random.nextInt( numberOfEvents ) ); return engine.execute( "MERGE (u:User {id: {userId})\n" + "MERGE (e:Event {id: {eventId})\n" + "MERGE (u)-[:HAS_EVENT]->(e)\n" + "RETURN u, e", MapUtil.map( "userId", userId, "eventId", eventId ) ); } } ); jobs.add( job ); } for ( Future future : jobs ) { try { observer.onNext( future.get() ); } catch ( InterruptedException | ExecutionException ignored ) { } } observer.onCompleted(); executor.shutdown(); return Subscriptions.empty(); } } ); } I’m not sure if that’s the correct way of using Observables so please let me know in the comments if I’ve got it wrong. I wasn’t sure what the proper way of handling errors was. I initially had a call to observer#onError in the catch block but that means that no further events are produced which wasn’t what I wanted. The code is available as a gist if you want to play around with it. I added the following dependency to get the RxJava library: com.netflix.rxjava rxjava-core 0.15.1

I’ll review Google’s most important Big Data publications and discuss where they are (as far as they’ve disclosed).

This article is an introduction to using the specification pattern in Java. We also will see how we can combine classic specifications with JPA Criteria queries to retrieve objects from a relational database. Within this post we will use the following Poll class as an example entity for creating specifications. It represents a poll that has a start and end date. In the time between those two dates users can vote among different choices. A poll can also be locked by an administrator before the end date has been reached. In this case, a lock date will be set. @Entity public class Poll { @Id @GeneratedValue private long id; private DateTime startDate; private DateTime endDate; private DateTime lockDate; @OneToMany(cascade = CascadeType.ALL) private List votes = new ArrayList<>(); } For better readability I skipped getters, setters, JPA annotations for mapping Joda DateTime instances and fields that aren't needed in this example (like the question being asked in the poll). Now assume we have two constraints we want to implement: A poll is currently running if it is not locked and if startDate < now < endDate A poll is popular if it contains more than 100 votes and is not locked We could start by adding appropriate methods to Poll like: poll.isCurrentlyRunning(). Alternatively we could use a service method like pollService.isCurrentlyRunning(poll). However, we also want to be able to query the database to get all currently running polls. So we might add a DAO or repository method like pollRepository.findAllCurrentlyRunningPolls(). If we follow this way we implement the isCurrentlyRunning constraint two times in two different locations. Things become worse if we want to combine constraints. What if we want to query the database for a list of all popular polls that are currently running? This is where the specification pattern come in handy. When using the specification pattern we move business rules into extra classes called specifications. To get started with specifications we create a simple interface and an abstract class: public interface Specification { boolean isSatisfiedBy(T t); Predicate toPredicate(Root root, CriteriaBuilder cb); Class getType(); } abstract public class AbstractSpecification implements Specification { @Override public boolean isSatisfiedBy(T t) { throw new NotImplementedException(); } @Override public Predicate toPredicate(Root poll, CriteriaBuilder cb) { throw new NotImplementedException(); } @Override public Class getType() { ParameterizedType type = (ParameterizedType) this.getClass().getGenericSuperclass(); return (Class) type.getActualTypeArguments()[0]; } } Please ignore the AbstractSpecification class with the mysterious getType() method for a moment (we come back to it later). The central part of a specification is the isSatisfiedBy() method, which is used to check if an object satisfies the specification. toPredicate() is an additional method we use in this example to return the constraint as javax.persistence.criteria.Predicate instance which can be used to query a database. For each constraint we create a new specification class that extends AbstractSpecification and implements isSatisfiedBy() and toPredicate(). The specification implementation to check if a poll is currently running looks like this: public class IsCurrentlyRunning extends AbstractSpecification { @Override public boolean isSatisfiedBy(Poll poll) { return poll.getStartDate().isBeforeNow() && poll.getEndDate().isAfterNow() && poll.getLockDate() == null; } @Override public Predicate toPredicate(Root poll, CriteriaBuilder cb) { DateTime now = new DateTime(); return cb.and( cb.lessThan(poll.get(Poll_.startDate), now), cb.greaterThan(poll.get(Poll_.endDate), now), cb.isNull(poll.get(Poll_.lockDate)) ); } } Within isSatisfiedBy() we check if the passed object matches the constraint. In toPredicate() we construct a Predicate using JPA's CriteriaBuilder. We will use the resulting Predicate instance later to build a CriteriaQuery for querying the database. The specification for checking if a poll is popular looks similar: public class IsPopular extends AbstractSpecification { @Override public boolean isSatisfiedBy(Poll poll) { return poll.getLockDate() == null && poll.getVotes().size() > 100; } @Override public Predicate toPredicate(Root poll, CriteriaBuilder cb) { return cb.and( cb.isNull(poll.get(Poll_.lockDate)), cb.greaterThan(cb.size(poll.get(Poll_.votes)), 5) ); } } If we now want to test if a Poll instance matches one of these constraints we can use our newly created specifications: boolean isPopular = new IsPopular().isSatisfiedBy(poll); boolean isCurrentlyRunning = new IsCurrentlyRunning().isSatisfiedBy(poll); For querying the database we need to extend our DAO / repository to support specifications. This can look like the following: public class PollRepository { private EntityManager entityManager = ... public List findAllBySpecification(Specification specification) { CriteriaBuilder criteriaBuilder = entityManager.getCriteriaBuilder(); // use specification.getType() to create a Root instance CriteriaQuery criteriaQuery = criteriaBuilder.createQuery(specification.getType()); Root root = criteriaQuery.from(specification.getType()); // get predicate from specification Predicate predicate = specification.toPredicate(root, criteriaBuilder); // set predicate and execute query criteriaQuery.where(predicate); return entityManager.createQuery(criteriaQuery).getResultList(); } } Here we finally use the getType() method implemented in AbstractSpecification to createCriteriaQuery and Root instances. getType() returns the generic type of theAbstractSpecification instance defined by the subclass. For IsPopular andIsCurrentlyRunning it returns the Poll class. Without getType() we would have to create theCriteriaQuery and Root instances inside toPredicate() of every specification we create. So it is just a small helper to reduce boiler plate code inside specifications. Feel free to replace this with your own implementation if you come up with better approaches. Now we can use our repository to query the database for polls that match a certain specification: List popularPolls = pollRepository.findAllBySpecification(new IsPopular()); List currentlyRunningPolls = pollRepository.findAllBySpecification(new IsCurrentlyRunning()); At this point the specifications are the only components that contain the constraint definitions. We can use it to query the database or to check if an object fulfills the required rules. However one question remains: How do we combine two or more constraints? For example we would like to query the database for all popular polls that are still running. The answer to this is a variation of the composite design pattern called composite specifications. Using a composite specification we can combine specifications in different ways. To query the database for all running and popular pools we need to combine the isCurrentlyRunning with theisPopular specification using the logical and operation. Let's create another specification for this. We name itAndSpecification: public class AndSpecification extends AbstractSpecification { private Specification first; private Specification second; public AndSpecification(Specification first, Specification second) { this.first = first; this.second = second; } @Override public boolean isSatisfiedBy(T t) { return first.isSatisfiedBy(t) && second.isSatisfiedBy(t); } @Override public Predicate toPredicate(Root root, CriteriaBuilder cb) { return cb.and( first.toPredicate(root, cb), second.toPredicate(root, cb) ); } @Override public Class getType() { return first.getType(); } } An AndSpecification is created out of two other specifications. In isSatisfiedBy() and toPredicate()we return the result of both specifications combined by a logical and operation. We can use our new specification like this: Specification popularAndRunning = new AndSpecification<>(new IsPopular(), new IsCurrentlyRunning()); List polls = myRepository.findAllBySpecification(popularAndRunning); To improve readability we can add an and() method to the Specification interface: public interface Specification { Specification and(Specification other); // other methods } and implement it within our abstract implementation: abstract public class AbstractSpecification implements Specification { @Override public Specification and(Specification other) { return new AndSpecification<>(this, other); } // other methods } Now we can chain multiple specification by using the and() method: Specification popularAndRunning = new IsPopular().and(new IsCurrentlyRunning()); boolean isPopularAndRunning = popularAndRunning.isSatisfiedBy(poll); List polls = myRepository.findAllBySpecification(popularAndRunning); When needed we can easily extend this further with other composite specifications (for exampleOrSpecification or NotSpecification). Conclusion When using the specification pattern we move business rules in separate specification classes. These specification classes can be easily combined by using composite specifications. In general, specification improve reusability and maintainability. Additionally specifications can easily be unit tested. For more detailed information about the specification pattern I recommend this article by Eric Evans and Martin Fowler. You can find the source of this example project on GitHub.

Finally the wait is over, Spring 4 is here and one of the best features is that now has a fully support for Groovy. The Spring team did a fantastic job bringing the same concept from Grails into the Spring Framework. In the past if you needed to incorporate Spring in your Groovy Applications or Scripts , normally you did something like this: @Grab('org.springframework:spring-context:4.0.0.RELEASE') import org.springframework.context.support.GenericApplicationContext import org.springframework.context.annotation.ClassPathBeanDefinitionScanner import org.springframework.beans.factory.annotation.Autowired import org.springframework.stereotype.Component @Component class Login { def authorize(User user){ if(user.credentials.username == "guest" && user.credentials.password == "guest"){ "${user.greetings} ${user.credentials.username}" }else "You are not ${user.greetings}" } } @Component class Credentials { String username = "guest" String password = "guest" } @Component class User{ @Autowired Credentials credentials String greetings = "Welcome" } def ctx = new GenericApplicationContext() new ClassPathBeanDefinitionScanner(ctx).scan('') // scan root package for components ctx.refresh() def login = ctx.getBean("login") def user = ctx.getBean("user") println login.authorize(user) One of the benefits that I see is that Groovy removes all the unnecessary boiler plate from Java. Then, if you wanted to add more flavor to your Groovy Apps using the Grail's BeanBuilder , you needed to do something like this: @Grab(group='org.slf4j', module='slf4j-simple', version='1.7.5') @Grab(group='org.grails', module='grails-web', version='2.3.4') import grails.spring.BeanBuilder import groovy.transform.ToString class Login { def authorize(User user){ if(user.credentials.username == "John" && user.credentials.password == "Doe"){ "${user.greetings} ${user.credentials.username}" }else "You are not ${user.greetings}" } } @ToString(includeNames=true) class Credentials{ String username String password } @ToString(includeNames=true) class User{ Credentials credentials String greetings } def bb = new BeanBuilder() bb.beans { login(Login) user(User){ credentials = new Credentials(username:"John", password:"Doe") greetings = 'Welcome!!' } } def ctx = bb.createApplicationContext() def u = ctx.getBean("user") println u def l = ctx.getBean("login") println l.authorize(u) And now with the new Spring 4 you can add all the Groovy flavor to your apps (without all the Grails Dependencies) and using the new GroovyBeanDefinitionReader from Spring 4: @Grab('org.springframework:spring-context:4.0.0.RELEASE') import org.springframework.context.support.GenericApplicationContext import org.springframework.beans.factory.groovy.GroovyBeanDefinitionReader import groovy.transform.ToString class Login { def authorize(User user){ if(user.credentials.username == "John" && user.credentials.password == "Doe"){ "${user.greetings} ${user.credentials.username}" }else "You are not ${user.greetings}" } } @ToString(includeNames=true) class Credentials{ String username String password } @ToString(includeNames=true) class User{ Credentials credentials String greetings } def ctx = new GenericApplicationContext() def reader = new GroovyBeanDefinitionReader(ctx) reader.beans { login(Login) user(User){ credentials = new Credentials(username:"John", password:"Doe") greetings = 'Welcome!!' } } ctx.refresh() def u = ctx.getBean("user") println u def l = ctx.getBean("login") println l.authorize(u) And that's not all, also SpringBoot gives you even more power using Groovy: //File: app.groovy import org.springframework.boot.* import org.springframework.boot.autoconfigure.* import org.springframework.stereotype.* import org.springframework.web.bind.annotation.* import org.springframework.beans.factory.annotation.* @Component class Login { def authorize(User user){ if(user.credentials.username == "John" && user.credentials.password == "Doe"){ "${user.greetings} ${user.credentials.username}" }else "You are not ${user.greetings}" } } @Component class Credentials{ String username String password } @Component class User{ Credentials credentials = new Credentials(username:"John", password:"Doe") String greetings = "Welcome!!" } @Controller @EnableAutoConfiguration class SampleController { @Autowired def login @Autowired def user @RequestMapping("/") @ResponseBody String home() { return login.authorize(user) } } To run the above code, install springboot and then just execute: $ spring run app.groovy and then go to your browser and open http://localhost:8080 and you see: Welcome!! John Congratulations to the Spring Team!! Keep grooving!! References: [1] https://spring.io/blog/2013/12/12/announcing-spring-framework-4-0-ga-release [2] http://groovy.codehaus.org/Using+Spring+Factories+with+Groovy [3] http://grails.org/doc/latest/guide/spring.html [4] http://projects.spring.io/spring-boot/ [5] https://spring.io/guides

a common and difficult problem acquiring data is extracting tables from a pdf. previously, i described how to extract the text from a pdf with pdf.js , a pdf rendering library made by mozilla labs. the rendering process requires an html canvas object, and then draws each object (character, line, rectangle, etc) on it. the easiest way to get a list of these is to to intercept all the calls pdf.js makes to drawing functions on the canvas object. (see “ self modifying javascripts ” for a similar technique). the “set” method below adds a wrapper closure to each function, which logs the call. function replace(ctx, key) { var val = ctx[key]; if (typeof(val) == "function") { ctx[key] = function() { var args = array.prototype.slice.call(arguments); console.log("called " + key + "(" + args.join(",") + ")"); return val.apply(ctx, args); } } } for (var k in context) { replace(context, k); } var rendercontext = { canvascontext: context, viewport: viewport }; page.render(rendercontext); this lets us see a series of calls: called transform(1,0,0,1,150.42,539.67) called translate(0,0) called scale(1,-1) called scale(0.752625,0.752625) called measuretext(c) called save() called scale(0.9701818181818181,1) called filltext(c,0,0) called restore() called restore() called save() called transform(1,0,0,1,150.42,539.6 we can easily retrieve the text by noting the first argument to each “filltext” call: "congregations ranked by growth and decline in membership and worship attendance, 2006 to 2011philadelphia presbytery - table 16net membership changenet worship changepercent changepercent changeworship 2006worship 2011membership 2006membership 2011abington, abington- 143(74)-13.18%(57)0(15)0.00%(22)numberrank3003001,085942anchor, wrightstown0(23)0.00%(27)-12(25)-21.43%(52)numberrank56449797arch street, philadelphia-117(71)-68.42%(117)27(5)90.00% (2)numberrank305717154aston, aston3(21)3.53%(22)-5(19)-9.43% (31)numberrank53488588beaconno reportboth yearsno reportboth yearsnumberrankbensalem, bensalem-23(39)-13.94%(62)-28(36)-28.57% (64)numberrank9870165142berean, philadelphia106(4)44.92%(4)no reportboth yearsnumberrank00236342bethany collegiate, havertown- 188(76)-42.44%(110)43(3)21.29%(7)numberrank202245443255bethel, philadelphia-13(33)-13.68%(60)-27(35)-35.06% (71)numberrank77509582bethesda, philadelphia9(18)5.56%(18)no reportboth yearsnumberrank1150162171beverly hills, upper darby-3(26)-3.03% (32)-11(24)-20.00%(48)numberrank55449996bridesburg, philadelphia0(23)0.00%(27)no reportboth yearsnumberrank004444bristol, bristolno reportboth yearsno reportboth yearsnumberrankpage 1 of 10report prepared by research services, presbyterian church (u.s.a.)1- 800-728-7228, ext #204006-oct-12" notable, this doesn’t track line endings, and not all the characters are recorded in the expected order (the first line is rendered after the second). the calls to transform, translate, and scale control where text is placed. the filltext method also takes an (x, y) parameter set that moves the individual letters between words. the exact position is a combination of successive operations, which are modeled as a stack of matrix operations. thankfully, pdf.js tracks the output of these operations as it renders, so we don’t have to recalculate it. thus, we can make a method that records the letters and their real positions. this method takes the internal context object, the type of state transition, and the arguments to the transition. this method is then called from the ‘record’ function listed above. var chars = []; var cur = {}; function record(ctx, state, args) { if (state === 'filltext') { var c = args[0]; cur.c = c; cur.x = ctx._transformmatrix[4] + args[1]; cur.y = ctx._transformmatrix[5] + args[2]; chars[chars.length] = cur; cur = {}; } } these results can be sorted by position (x and y). the sort method arranges letters by position – if they are shifted up or down a small amount, they are considered to be on one line. chars.sort( function(a, b) { var dx = b.x - a.x; var dy = b.y - a.y; if (math.abs(dy) < 0.5) { return dx * -1; } else { return dy * -1; } } ); this presents several difficulties: this doesn’t detect right-to-left text, and it’s becoming clear that we’re going to have a hard time knowing when you’re in a table and when we aren’t. to do this, we define a function which can transform the array of letters and positions into a csv style output. this tracks from letter to letter – if it sees a “large” change in y, it makes a new line. if it sees a “large” change in x, it treats it as a new column. the real challenge is defining “large” which for my test pdf were around 15 and 20, for dx and dy. function gettext(marks, ex, ey, v) { var x = marks[0].x; var y = marks[0].y; var txt = ''; for (var i = 0; i < marks.length; i++) { var c = marks[i]; var dx = c.x - x; var dy = c.y - y; if (math.abs(dy) > ey) { txt += "\"\n\""; if (marks[i+1]) { // line feed - start from position of next line x = marks[i+1].x; } } if (math.abs(dx) > ex) { txt += "\",\""; } if (v) { console.log(dx + ", " + dy); } txt += c.c; x = c.x; y = c.y; } return txt; } this algorithm doesn’t handle newlines in rows, and oddly, the columns don’t come out in the right order, but they appear to be consistently out of order. line with large spaces (e.g. an em-dash) are detected as having multiple columns, but this can be cleaned up later – here is some sample output. you can see an example below, and the final source is available on github . congregations ranked by growth and decline in m","embership and w","orship attendance, 2006 to 2011" "","philadelphia presbytery"," - table 16" "","net ","membership ","change" "","net worship ","change","percent ","change","percent ","change","worship"," 2006","worship"," 2011","membership"," 2006","membership"," 2011" "","abington, abington","-143","(74)","-13.18%(57)","0","(15)","0.00%(22)","number","rank","300","300","1,085","942" "","anchor, wrightstown","0","(23)","0.00%(27)","-12","(25)","-21.43%(52)","number","rank","56","44","97","97" "","arch street, philadelphia","-117","(71)","-68.42%","(117)","27(5)","90.00%(2)","number","rank","30","57","171","54" "","aston, aston","3","(21)","3.53%(22)","-5","(19)","-9.43%(31)","number","rank","53","48","85","88" "","beacon","no report","both years","no report","both years","number","rank" "","bensalem, bensalem","-23","(39)","-13.94%(62)","-28","(36)","-28.57%(64)","number","rank","98","70","165","142" "","berean, philadelphia","106(4)","44.92%(4)","no report","both years","number","rank","0","0","236","342" "","bethany collegiate, havertown","-188","(76)","-42.44%","(110)","43(3)","21.29%(7)","number","rank","202","245","443","255" "","bethel, philadelphia","-13","(33)","-13.68%(60)","-27","(35)","-35.06%(71)","number","rank","77","50","95","82" "","bethesda, philadelphia","9","(18)","5.56%(18)","no report","both years","number","rank","115","0","162","171" "","beverly hills, upper darby","-3","(26)","-3.03%(32)","-11","(24)","-20.00%(48)","number","rank","55","44","99","96" "","bridesburg, philadelphia","0","(23)","0.00%(27)","no report","both years","number","rank","0","0","44","44" "","bristol, bristol","no report","both years","no report","both years","number","rank" "","page 1 of 10","report prepared by research services, presbyterian church (u.s.a.)","1-800-728-7228, ext #2040","06-oct-12"

this post is not about variables in my application code (which i debug). it is about using variables in eclipse for building projects. eclipse variables allow me to make my projects ‘position independent’ whenever i cannot use a path relative to my projects or workspace. eclipse variables which variables are used where in eclipse might be sometimes not very clear. depending in which context variables are used, not everything might be available. this link for example gives a list of variables which can be used to invoke an external tool. build variables eclipse comes with many built-in variables, especially for the build system. if i want to see what variables are already defined, i can show them in the project properties, under c/c++ build > build variables with enabled option ‘show system variables’: system build variables with the ‘add…’ button i can define and add my own variables, available for that project: define a new build variable if above operation is done on a project, then the setting is for the project only. if i want to add a variable for the workspace, i can do this using the menu window > preferences : workspace build variables global system variables eclipse automatically includes the system (e.g. windows) environment variables. many dialogs have the ‘variables…’ button where i can use my variables, including the variables defined on system level: system variables system variables: one way or the other so if i want to have a variable for every workspace, one way is to define it at the system level. however, this is not a good way as this clutter the variables for every application. batch file a solution to this to create my custom batch file where i define my variables, and at the end of this batch file i launch eclipse. that way the extra variables are only for this eclipse session. cwide-env file another very nice way codewarrior eclipse offers is using the cwide-env file located in the eclipse sub-folder of the installation: cwide-env file i can define variables here, or extend existing ones: -add : add string to the variable at the end -prepend : add string to the variable at the beginning that way i can easily manipulate existing system variables or create new ones which then are used by eclipse. summary variables in eclipse help me to define paths to source files and folders outside of a project or workspace. with variables i avoid using absolute paths which would make porting projects from one machine to another difficult. i can define variables for projects, for the workspace or use system variables. with codewarrior i have a cwide-env file which is used to extend the system variables. happy variabling

the upcoming version of jboss application server will no longer use tomcat as the integrated webserver, but instead, will replace it with undertow . the architecture of undertow is based on handlers that can be added dynamically via a builder api to the server. this approach is similar to the way of constructing a webserver in node.js . it allows developers to embed the undertow webserver easily into their applications. as the addition of features is done via the builder api, one can only add the features that are really required in one’s application. beyond that undertow supports websockets and the servlet api in version 3.1. it can be run as a blocking or non-blocking server and it is reported that first benchmarks have proven that undertow is the fastest webserver written in java. as all of this sounds very promising, so let’s try to set up a simple websocket server. as usual we start by creating a simple java project and add the undertow maven dependency: io.undertow undertow-core 1.0.0.beta20 with undertow’s builder api our buildandstartserver() method looks like this: public void buildandstartserver(int port, string host) { server = undertow.builder() .addlistener(port, host) .sethandler(getwebsockethandler()) .build(); server.start(); } we just add a listener that specifies the port and host to listen for incoming connections and afterwards add a websocket handler. as the websocket handler code is a little bit more comprehensive, i have put it into its own method: private pathhandler getwebsockethandler() { return path().addpath("/websocket", websocket(new websocketconnectioncallback() { @override public void onconnect(websockethttpexchange exchange, websocketchannel channel) { channel.getreceivesetter().set(new abstractreceivelistener() { @override protected void onfulltextmessage(websocketchannel channel, bufferedtextmessage message) { string data = message.getdata(); lastreceivedmessage = data; logger.info("received data: "+data); websockets.sendtext(data, channel, null); } }); channel.resumereceives(); } })) .addpath("/", resource(new classpathresourcemanager(websocketserver.class.getclassloader(), websocketserver.class.getpackage())) .addwelcomefiles("index.html")); } let’s go line by line through this code snippet. first of all, we add a new path: /websocket. the second argument of the addpath() methods lets us specify what kind of protocol we want to use for this path. in our case we create a new websocket. the anonymous implementation has a onconnect() method in which we set an implementation of abstractreceivelistener. here we have a convenient method onfulltextmessage() that is called when a client has sent us a text message. a call of getdata() fetches the actual message we have received. in this simple example we just echo this string back to client to validate that the roundtrip from the client to server and back works. to perform some simple manual tests we also add a second resource under the path / which serves some static html and javascript files. the directory that contains these files is given as an instance of classpathresourcemanager. the call of addwelcomefiles() tells undertow which file to server when the client asks for the path /. the index.html looks like this: our javascript code is swapped out to the websocket.js file. we use jquery and the jquery-plugin gracefulwebsocket to ease the client side development: var ws = $.gracefulwebsocket("ws://127.0.0.1:8080/websocket"); ws.onmessage = function(event) { var messagefromserver = event.data; $('#output').append('received: '+messagefromserver+''); } function send(message) { ws.send(message); } after having created a websocket object by calling $.gracefulwebsocket() we can register a callback function for incoming messages. in this method we only append the message string to the dom of the page. the send() method is just a call to gracefulwebsocket’s send() method. when we now start our application and open the url http://127.0.0.1:8080/ in our webbrowser we see the following page: entering some string and hitting the “send web socket data” button sends the message to the server, which in response echos it back to the client. now that we know that everything works as expected, we want to protect our code against regression with a junit test case. as a websocket client i have chosen the library jetty-websocket: org.eclipse.jetty jetty-websocket 8.1.0.rc5 test in the test case we build and start the websocket server to open a new connection to the websocket port. the websocket implementation of jetty-websocket allows us to implement two callback methods for the open and close events. within the open callback we send the test message to the client. the rest of the code waits for the connection to be established, closes it and asserts that the server has received the message: @test public void teststartandbuild() throws exception { subject = new websocketserver(); subject.buildandstartserver(8080, "127.0.0.1"); websocketclient client = new websocketclient(); future connectionfuture = client.open(new uri("ws://localhost:8080/websocket"), new websocket() { @override public void onopen(connection connection) { logger.info("onopen"); try { connection.sendmessage("testmessage"); } catch (ioexception e) { logger.error("failed to send message: "+e.getmessage(), e); } } @override public void onclose(int i, string s) { logger.info("onclose"); } }); websocket.connection connection = connectionfuture.get(2, timeunit.seconds); assertthat(connection, is(notnullvalue())); connection.close(); subject.stopserver(); thread.sleep(1000); assertthat(subject.lastreceivedmessage, is("testmessage")); } as usual you can find the source code on github . conclusion: undertow’s builder api makes it easy to construct a websocket server and in general an embedded webserver that fits your needs. this also eases automatic testing as you do not need any specific maven plugin that starts and stops your server before and after your integration tests. beyond that the jquery plugin jquery-graceful-websocket lets you send and receive messages over websockets with only a few lines of code.

One alternative to using SQLite on Android is to store Java objects in SharedPreferences.

Your mission-critical projects need complex event processing, realtime management and monitoring. Talend 5.4 (released in December 2013, https://www.talend.com) offers a great new feature: Talend Event Logging. It allows logging, processing and monitoring of all technical events and business data. In this article, I will focus on how to process, filter and monitor business data. You can find more details about monitoring technical events and logs (e.g. OSGi events of the ESB container) in Talend’s documentation (www.help.talend.com). This new feature is very powerful, but also extendable to fit custom requirements. You can solve and monitor much more complex scenarios than the one I describe here. Talend Event Logging with Logstash, ElasticSearch and Kibana First, let’s take a look at the components / projects which are integrated and extended into Talend’s products for implementing this new feature. logstash (http://logstash.net) is a tool for managing events and logs. You can use it to collect different (distributed) logs, parse them, and store them for later use. Speaking of searching, logstash comes with a simple, but fine web interface for searching and drilling into all of your logs. It uses ElasticSearch under the hood. So, you can easily query through all your logs for specific errors or business analytics (e.g. searching for all lines matching an unique order id). Additional to the pure collection of events, the Event Logging feature supports custom processing (e.g. custom filtering, customer data enrichment/reduction), aggregation, signing and also server side custom pre- and post-processing of events - e.g. to send them to an intrusion detection system or to any other kind of potential higher level log processing /management system. Kibana (http://www.elasticsearch.org/overview/kibana) is a browser based analytics and search interface to logstash and other timestamped data sets stored in ElasticSearch. Kibana strives to be easy to get started with, while also being flexible and powerful, just like logstash and ElasticSearch. Main difference to logstash is a much more powerful HTML5 based web interface. You can • use multiple concurrent search inputs • highlight to drill down bar charts • create line charts, stacked, unstacked, filled or unfilled, with or without points • create Pie and donut charts that compare top terms or the results of multiple queries • create custom dashboards with multiple charts • and much more… Therefore, logstash, Elasticsearch and Kibana are a perfect combination. You can use Kibana to analyze and monitor your data as you do with logstash, however, Kibana’s web interface is much more powerful and comfortable than logstash. There is a great book about logstash: “The logstash book” – for just 9.99 USD. I can really recommend this book for getting started: http://www.logstashbook.com/. For Elasticsearch, you can find several books on Amazon. Unfortunately, Kibana has no good and extensive documentation yet. I heard from its developers that this topic is addressed for Q1 2014. Integration of Event Logging and Monitoring into Talend’s Unified Platform Talend 5.4 has integrated logstash and Kibana into its Unified Platform. Talend Administration Center (TAC) is Talend’s central web application for management and monitoring. It got a new logging view: Here, you can use very flexible and powerful realtime search capabilities of Elasticsearch. Some dashboards are available by default. You can also create your own custom dashboards easily within this site thanks to Kibana. Many panel types are available, such as pie, histogram, table, hits or trends. You can analyze every technical event or business data down to the message level: By default, you see fields such as message, source, timestamp, type, and others. Of course, you can also add custom fields suitable for your business case. This way, you have a central monitoring capability which allows analyzing data on distributed clusters easily. Under the hood, this data comes from logstash. Many alternative inputs are available for logstash, such as log4j input or tcp input. For business data, I often use file input (http://logstash.net/docs/1.2.2/inputs/file) to analyze files such as CSV. Adding new inputs is very simple. You just have to add an input to your logstash configuration file of your logstash server. As I mentioned already, all this is integrated into the Talend’s Unified Platform: In this example, there are three log4j inputs and one file input. I use the file input to analyze text files in a specific directory. In this case, output is an embedded Elasticsearch instance. In production, you should use an external Elasticsearch cluster, of course. Processing such as filtering can also be configured in this file. Thus, you can process, analyze and monitor all your different inputs within one central monitoring application thanks to Kibana. Building Talend Integration Jobs, Routes and Web Services As mentioned before, you can analyze almost all data with logstash, Elasticsearch and Kibana. It does not matter if your input is technical events from a container (e.g. OSGi events) or any business data such as CSV files, log4j logs, or something else. Talend implicitly supports technical events which are created by the ESB container, by MDM, etc. However, you can also add your custom business data from your Talend jobs (integration perspective), SOAP / REST Web Services (integration perspective) or Talend routes (mediation perspective), easily: This is just one example (part of Talend’s DI demos which are included in every DI installation). The job generates some random data and stores it to a CSV file. You just have to add the configured file or directory of tFileOutputDelimited to your logstash configuration using file input (with wild cards for more complex scenarios). That’s it. You can now monitor and analyze the business data in realtime. This example showed a Talend DI Job (i.e. ETL job). However, you can also monitor your business data from SOAP / REST Web Services or Talend Routes the same way. Conclusion Your mission-critical projects need management and monitoring. Today, this is not just possible with complex and expensive tools of large vendors, but also with Talend’s Unified Platform products such as Talend ESB or Talend MDM. Under the hood, Talend integrates and extends widely used open source products: logstash, Elasticsearch and Kibana. Have fun with Talend 5.4’s new event logging and monitoring features… Best regards, Kai Wähner (@KaiWaehner) CONTENT FROM MY BLOG: http://www.kai-waehner.de/blog/2013/12/17/realtime-event-logging-complex-event-processing-cep-and-monitoring-with-talends-unified-platform-5-4-di-esb-dq-mdm-bpm-using-elasticsearch-logstash-and-kibana/

Editor's note: Be sure to check out part 2 as well. This is the fourth of an introductory series of blogs on Apache HBase. In the third part, we saw a high level view of HBase architecture . In this part, we'll use the HBase Java API to create tables, insert new data, and retrieve data by row key. We'll also see how to setup a basic table scan which restricts the columns retrieved and also uses a filter to page the results. Having just learned about HBase high-level architecture, now let's look at the Java client API since it is the way your applications interact with HBase. As mentioned earlier you can also interact with HBase via several flavors of RPC technologies like Apache Thrift plus a REST gateway, but we're going to concentrate on the native Java API. The client APIs provide both DDL (data definition language) and DML (data manipulation language) semantics very much like what you find in SQL for relational databases. Suppose we are going to store information about people in HBase, and we want to start by creating a new table. The following listing shows how to create a new table using the HBaseAdmin class. Configuration conf = HBaseConfiguration.create(); HBaseAdmin admin = new HBaseAdmin(conf); HTableDescriptor tableDescriptor = new HTableDescriptor(TableName.valueOf("people")); tableDescriptor.addFamily(new HColumnDescriptor("personal")); tableDescriptor.addFamily(new HColumnDescriptor("contactinfo")); tableDescriptor.addFamily(new HColumnDescriptor("creditcard")); admin.createTable(tableDescriptor); The people table defined in preceding listing contains three column families: personal, contactinfo, and creditcard. To create a table you create an HTableDescriptor and add one or more column families by adding HColumnDescriptor objects. You then call createTable to create the table. Now we have a table, so let's add some data. The next listing shows how to use the Put class to insert data on John Doe, specifically his name and email address (omitting proper error handling for brevity). Configuration conf = HBaseConfiguration.create(); HTable table = new HTable(conf, "people"); Put put = new Put(Bytes.toBytes("doe-john-m-12345")); put.add(Bytes.toBytes("personal"), Bytes.toBytes("givenName"), Bytes.toBytes("John")); put.add(Bytes.toBytes("personal"), Bytes.toBytes("mi"), Bytes.toBytes("M")); put.add(Bytes.toBytes("personal"), Bytes.toBytes("surame"), Bytes.toBytes("Doe")); put.add(Bytes.toBytes("contactinfo"), Bytes.toBytes("email"), Bytes.toBytes("[email protected]")); table.put(put); table.flushCommits(); table.close(); In the above listing we instantiate a Put providing the unique row key to the constructor. We then add values, which must include the column family, column qualifier, and the value all as byte arrays. As you probably noticed, the HBase API's utility Bytes class is used a lot; it provides methods to convert to and from byte[] for primitive types and strings. (Adding a static import for the toBytes() method would cut out a lot of boilerplate code.) We then put the data into the table, flush the commits to ensure locally buffered changes take effect, and finally close the table. Updating data is also done via the Put class in exactly the same manner as just shown in the prior listing. Unlike relational databases in which updates must update entire rows even if only one column changed, if you only need to update a single column then that's all you specify in the Put and HBase will only update that column. There is also a checkAndPut operation which is essentially a form of optimistic concurrency control - the operation will only put the new data if the current values are what the client says they should be. Retrieving the row we just created is accomplished using the Get class, as shown in the next listing. (From this point forward, listings will omit the boilerplate code to create a configuration, instantiate the HTable, and the flush and close calls.) Get get = new Get(Bytes.toBytes("doe-john-m-12345")); get.addFamily(Bytes.toBytes("personal")); get.setMaxVersions(3); Result result = table.get(get); The code in the previous listing instantiates a Get instance supplying the row key we want to find. Next we use addFamily to instruct HBase that we only need data from the personal column family, which also cuts down the amount of work HBase must do when reading information from disk. We also specify that we'd like up to three versions of each column in our result, perhaps so we can list historical values of each column. Finally, calling get returns a Result instance which can then be used to inspect all the column values returned. In many cases you need to find more than one row. HBase lets you do this by scanning rows, as shown in the second part which showed using a scan in the HBase shell session. The corresponding class is the Scan class. You can specify various options, such as the start and ending row key to scan, which columns and column families to include and the maximum versions to retrieve. You can also add filters, which allow you to implement custom filtering logic to further restrict which rows and columns are returned. A common use case for filters is pagination. For example, we might want to scan through all people whose last name is Smith one page (e.g. 25 people) at a time. The next listing shows how to perform a basic scan. Scan scan = new Scan(Bytes.toBytes("smith-")); scan.addColumn(Bytes.toBytes("personal"), Bytes.toBytes("givenName")); scan.addColumn(Bytes.toBytes("contactinfo"), Bytes.toBytes("email")); scan.setFilter(new PageFilter(25)); ResultScanner scanner = table.getScanner(scan); for (Result result : scanner) { // ... } In the above listing we create a new Scan that starts from the row key smith- and we then use addColumn to restrict the columns returned (thus reducing the amount of disk transfer HBase must perform) to personal:givenName and contactinfo:email. A PageFilter is set on the scan to limit the number of rows scanned to 25. (An alternative to using the page filter would be to specify a stop row key when constructing the Scan.) We then get a ResultScanner for the Scan just created, and loop through the results performing whatever actions are necessary. Since the only method in HBase to retrieve multiple rows of data is scanning by sorted row keys, how you design the row key values is very important. We'll come back to this topic later. You can also delete data in HBase using the Delete class, analogous to the Put class to delete all columns in a row (thus deleting the row itself), delete column families, delete columns, or some combination of those. Connection Handling In the above examples not much attention was paid to connection handling and RPCs (remote procedure calls). HBase provides the HConnection class which provides functionality similar to connection pool classes to share connections, for example you use the getTable() method to get a reference to an HTable instance. There is also an HConnectionManager class which is how you get instances of HConnection. Similar to avoiding network round trips in web applications, effectively managing the number of RPCs and amount of data returned when using HBase is important, and something to consider when writing HBase applications. Conclusion to Part 4 In this part we used the HBase Java API to create a people table, insert a new person, and find the newly inserted person information. We also used the Scan class to scan the people table for people with last name "Smith" and showed how to restrict the data retrieved and finally how to use a filter to limit the number of results. In the next part, we'll learn how to deal with the absence of SQL and relations when modeling schemas in HBase. References HBase web site, http://hbase.apache.org/ HBase wiki, http://wiki.apache.org/hadoop/Hbase HBase Reference Guide http://hbase.apache.org/book/book.html HBase: The Definitive Guide, http://bit.ly/hbase-definitive-guide Google Bigtable Paper, http://labs.google.com/papers/bigtable.html Hadoop web site, http://hadoop.apache.org/ Hadoop: The Definitive Guide, http://bit.ly/hadoop-definitive-guide Fallacies of Distributed Computing, http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing HBase lightning talk slides, http://www.slideshare.net/scottleber/hbase-lightningtalk Sample code, https://github.com/sleberknight/basic-hbase-examples

In Lucene's facet module we recently added support for dynamic range faceting, to show how many hits match each of a dynamic set of ranges. For example, the Updated drill-down in the Lucene/Solr issue search application uses range facets. Another example is distance facets (< 1 km, < 2 km, etc.), where the distance is dynamically computed based on the user's current location. Price faceting might also use range facets, if the ranges cannot be established during indexing. To implement range faceting, for each hit, we first calculate the value (the distance, the age, the price) to be aggregated, and then lookup which ranges match that value and increment its counts. Today we use a simple linear search through all ranges, which has O(N) cost, where N is the number of ranges. But this is inefficient! Segment Trees There are fun data structures like segment trees and interval trees with O(log(N) + M) cost per lookup, where M is the number of ranges that match the given value. I chose to explore segment trees, as Lucene only requires looking up by a single value (interval trees can also efficiently look up all ranges overlapping a provided range) and also because all the ranges are known up front (interval trees also support dynamically adding or removing ranges). If the ranges will never overlap, you can use a simple binary search; Guava's ImmutableRangeSet takes this approach. However, Lucene's range faceting allows overlapping ranges so we can't do that. Segment trees are simple to visualize: you "project" all ranges down on top of one another, creating a one-dimensional Venn diagram, to define the elementary intervals. This classifies the entire range of numbers into a minimal number of distinct ranges, each elementary interval, such that all points in each elementary interval always match the same set of input ranges. The lookup process is then a binary search to determine which elementary interval a point belongs to, recording the matched ranges as you recurse down the tree. Consider these ranges; the lower number is inclusive and the upper number is exclusive: 0: 0 – 10 1: 0 – 20 2: 10 – 30 3: 15 – 50 4: 40 – 70 The elementary intervals (think Venn diagram!) are: -∞ – 0 0 – 10 10 – 15 15 – 20 20 – 30 30 – 40 40 – 50 50 – 70 70 – ∞ Finally, you build a binary tree on top of the elementary ranges, and then add output range indices to both internal nodes and the leaves of that tree, necessary to prevent adversarial cases that would require too much (O(N^2)) space. During lookup, as you walk down the tree, you gather up the output ranges (indices) you encounter; for our example, each elementary range is assigned the follow range indices as outputs: -∞ – 0 → 0 – 10 → 0 10 – 15 → 1, 2 15 – 20 → 1, 2, 3 20 – 30 → 2, 3 30 – 40 → 3 40 – 50 → 3, 4 50 – 70 → 4 70 – ∞ → Some ranges correspond to 1 elementary interval, while other ranges correspond to 2 or 3 or more, in general. Some, 2 in this example, may have no matching input ranges. Looking Up Matched Ranges I've pushed all sources described below to new Google code project; the code is still somewhat rough and exploratory, so there are likely exciting bugs lurking, but it does seem to work: it includes (passing!) tests and simple micro-benchmarks. I started with a basic segment tree implementation as described on the Wikipedia page, for long values, called SimpleLongRangeMultiSet; here's the recursive lookup method: private int lookup(Node node, long v, int[] answers, int upto) { if (node.outputs != null) { for(int range : node.outputs) { answers[upto++] = range; } } if (node.left != null) { if (v <= node.left.end) { upto = lookup(node.left, v, answers, upto); } else { upto = lookup(node.right, v, answers, upto); } } return upto; } This worked correctly, but I realized there must be non-trivial overhead for the recursion, checking for nulls, the for loop over the output values, etc. Next, I tried switching to parallel arrays to hold the binary tree (ArrayLongRangeMultiSet), where the left child of node N is at 2*N and the right child is at 2*N+1, but this turned out to be slower. After that I tested a code specializing implementation, first by creating dynamic Java source code from the binary tree. This eliminates the recursion and creates a single simple method that uses a series of if statements, specialized to the specific ranges, to do the binary search and record the range indices. Here's the resulting specialized code, compiled from the above ranges: void lookup(long v, int[] answers) { int upto = 0; if (v <= 19) { if (v <= 9) { if (v >= 0) { answers[upto++] = 0; answers[upto++] = 1; } } else { answers[upto++] = 1; answers[upto++] = 2; if (v >= 15) { answers[upto++] = 3; } } } else { if (v <= 39) { answers[upto++] = 3; if (v <= 29) { answers[upto++] = 2; } } else { if (v <= 49) { answers[upto++] = 3; answers[upto++] = 4; } else { if (v <= 69) { answers[upto++] = 4; } } } } } Finally, using the ASM library, I compiled the tree directly to specialized Java bytecode, and this proved to be fastest (up to 2.5X faster in some cases). As a baseline, I also added the simple linear search method, LinearLongRangeMultiSet; as long as you don't have too many ranges (say 10 or less), its performance is often better than the Java segment tree. The implementation also allows you to specify the allowed range of input values (for example, maybe all values are >=0 in your usage), which can save an if statement or two in the lookup method. Counting All Matched Ranges While the segment tree allows you to quickly look up all matching ranges for a given value, after a nice tip from fellow Lucene committee Robert Muir, we realized Lucene's range faceting does not need to know the ranges for each value; instead, it only requires the aggregated counts for each range in the end, after seeing many values. This leads to an optimization: compute counts for each elementary interval and then in the end, roll up those counts to get the count for each range. This will only work for single-valued fields, since for a multi-valued field you'd need to carefully never increment the same range more than once per hit. So based on that approach, I created a new LongRangeCounter abstract base class, and the SimpleLongRangeCounter Java implementation, and also the ASM specialized version, and the results are indeed faster (~20 to 50%) than using the lookup method to count; I'll use this approach with Lucene. Segment trees are normally always "perfectly" balanced but one final twist I explored was to use a training set of values to bias the order of the if statements. For example, if your ranges cover a tiny portion of the search space, as is the case for the Updated drill-down, then it should be faster to use a slightly unbalanced tree, by first checking if the value is less than the maximum range. However, in testing, while there are some cases where this "training" is a bit faster, often it's slower; I'm not sure why. Lucene I haven't folded this into Lucene yet, but I plan to; all the exploratory code lives in the segment-trees Google code project for now. Results on the micro-benchmarks can be entirely different once the implementations are folded into a "real" search application. While ASM is a powerful way to generate specialized code, and it gives sizable performance gains at least in the micro-benchmarks, it is an added dependency and complexity for ongoing development and many more developers know Java than ASM. It may also confuse hotspot, causing deoptimizations when there are multiple implementations for an abstract base class. Furthermore, if there are many ranges, the resulting specialized bytecode can be become quite large (but, still O(N*log(N)) in size), which may cause other problems. On balance I'm not sure the sizable performance gains (on a micro-benchmark) warrant using ASM in Lucene's range faceting.

To serialize a Business object marked with [DataContract] and properties marked with [DataMember] attributes… public static string DataContractSerializeObject(T objectToSerialize) { using (var output = new StringWriter()) { using (var writer = new XmlTextWriter(output) { Formatting = Formatting.Indented }) { var dataContractSerializer = new DataContractSerializer(typeof(T), EntityUtilities.GetAllKnownTypes(), int.MaxValue, true, true, null); dataContractSerializer.WriteObject(writer, objectToSerialize); return output.GetStringBuilder().ToString(); } } } Then to deserialize back to your DataContract type, use this logic… public static T Deserialize(string xml) { using (Stream stream = new MemoryStream()) { byte[] data = System.Text.Encoding.UTF8.GetBytes(xml); stream.Write(data, 0, data.Length); stream.Position = 0; var dataContractSerializer = new DataContractSerializer(typeof(T), EntityUtilities.GetAllKnownTypes(), int.MaxValue, true, true, null); return (T)dataContractSerializer.ReadObject(stream); } }

A typical Maven and Spring web application has a fair amount of XML and verbosity to it. Add in Jersey and Spring Security and you can have hundreds of lines of XML before you even start to write your Java code. As part of a recent project, I was tasked with upgrading a webapp like this to use Spring 4 and Spring Boot. I also figured I'd try to minimize the XML. This is my story on how I upgraded to Spring 4, Jersey 2, Java 8 and Spring Boot 0.5.0 M6. When I started, the app was using Spring 3.2.5, Spring Security 3.1.4 and Jersey 1.18. The pom.xml had four Jersey dependencies, three Spring dependencies and three Spring Security dependencies, along with a number of exclusions for "jersey-spring". Upgrading to Spring 4 Upgrading to Spring 4 was easy, I changed the version property to 4.0.0.RC2 and added the new Spring bill of materials to my pom.xml. I also add the Spring milestone repo since Spring 4 won't be released to Maven central until tomorrow. org.springframework spring-framework-bom ${spring.framework.version} pom import spring-milestones http://repo.spring.io/milestone true Next, I removed all the references to ${spring.framework.version} in dependencies since it'd be controlled by Maven's dependency management feature. org.springframework spring-web - ${spring.framework.version} I also changed to use Maven 3's wildcard syntax to exclude multiple dependencies. com.sun.jersey.contribs jersey-spring org.springframework - spring - - - org.springframework - spring-core - - - org.springframework - spring-web - - - org.springframework - spring-beans - - - org.springframework - spring-context + * I confirmed the upgrade worked by running "mvn dependency:tree | grep spring", followed by "mvn jetty:run" and viewing the app in my browser. Upgrading to Jersey 2 The next item I tackled was upgrading to Jersey 2.4.1. I changed the version number in my pom.xml, then added the Jersey BOM. org.glassfish.jersey jersey-bom ${jersey.version} pom import You might ask "why Jersey?" if we already have Spring MVC and its REST support? You might also ask why not Play or Grails instead of a Java + Spring stack? For this particular project, I recommended technology options, and these were certainly among them. However, the team chose differently and I support their decision. The project is creating an iOS app, as well as a responsive HTML5 mobile/desktop app. We figured we had enough risk with new technologies on the front-end that we should play it a bit safer on the backend. To make the backend work a bit sexier, we've decided to allow Spring 4, Java 8 and possibly some reactive principles. Next, I changed from the old com.sun.jersey dependencies to org.glassfish.jersey and removed jersey-spring. org.glassfish.jersey.containers jersey-container-servlet org.glassfish.jersey.media jersey-media-json-jackson The last thing I needed to do was change the servlet-class and param-name in web.xml: jersey-servlet org.glassfish.jersey.servlet.ServletContainer jersey.config.server.provider.packages com.raibledesigns.boot.service 1 Requiring Java 8 Requiring Java 8 to compile was easy enough. I added the maven-compiler-plugin to enforce a minimum version. maven-compiler-plugin 3.1 1.8 1.8 I downloaded the latest Java 8 SDK and installed it. Then I set my JAVA_HOME to use it. export JAVA_HOME=`/usr/libexec/java_home -v 1.8` Integrating Spring Boot I learned about Spring Boot a few weeks ago at Devoxx. Josh Long gave me a 3-minute demo at the speaker's dinner and showed me enough to peak my interest. To integrate it into my project, I started with the Quick Start. I added the boot-parent, dependencies for web, security and actuator (logging, metrics, etc.) and the Maven plugin. I removed all the Spring and Spring Security dependencies. org.springframework.boot spring-boot-starter-parent 0.5.0.M6 ... spring-milestones http://repo.spring.io/milestone ... org.springframework.boot spring-boot-starter-web org.springframework.boot spring-boot-starter-security org.springframework.boot spring-boot-starter-actuator ... org.springframework.boot spring-boot-maven-plugin Upon restarting my app, I got an error about spring-security.xml using a 3.1 XSD. I fixed it by changing to 3.2. Next, I wanted to eliminate web.xml. First of all, I created an ApplicationInitializer so the WAR could be started from the command line. package com.raibledesigns.boot.config; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.boot.builder.SpringApplicationBuilder; import org.springframework.boot.web.SpringBootServletInitializer; import org.springframework.context.annotation.ComponentScan; import org.springframework.context.annotation.Configuration; @Configuration @EnableAutoConfiguration @ComponentScan public class ApplicationInitializer extends SpringBootServletInitializer { @Override protected SpringApplicationBuilder configure(SpringApplicationBuilder application) { return application.sources(ApplicationInitializer.class); } public static void main(String[] args) { SpringApplication.run(ApplicationInitializer.class, args); } } However, after adding this, I received the following error on startup: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.boot.context.properties.ConfigurationPropertiesBindingPostProcessor': Invocation of init method failed; nested exception is java.lang.AbstractMethodError: org.hibernate.validator.internal.engine.ConfigurationImpl .getDefaultParameterNameProvider()Ljavax/validation/ParameterNameProvider; Adding hibernate-validator as a dependency solved this problem: org.hibernate hibernate-validator To configure Spring Security without web.xml and spring-security.xml, I created WebSecurityConfig.java: package com.raibledesigns.boot.config; import org.springframework.context.annotation.Configuration; import org.springframework.core.Ordered; import org.springframework.core.annotation.Order; import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder; import org.springframework.security.config.annotation.web.builders.HttpSecurity; import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity; import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter; @Configuration @EnableWebSecurity @Order(Ordered.LOWEST_PRECEDENCE - 6) public class WebSecurityConfig extends WebSecurityConfigurerAdapter { @Override protected void configure(HttpSecurity http) throws Exception { http.authorizeRequests() .antMatchers("/", "/home").permitAll() .antMatchers("/v1.0/**").hasRole("USER") .anyRequest().authenticated(); http.httpBasic().realmName("My API"); } @Override protected void configure(AuthenticationManagerBuilder authManagerBuilder) throws Exception { authManagerBuilder.inMemoryAuthentication() .withUser("test").password("test123").roles("USER"); } } To configure Jersey without web.xml, I created a JerseyConfig class: package com.raibledesigns.boot.config; import org.glassfish.jersey.filter.LoggingFilter; import org.glassfish.jersey.jackson.JacksonFeature; import org.glassfish.jersey.server.ResourceConfig; import org.glassfish.jersey.server.ServerProperties; import javax.ws.rs.ApplicationPath; @ApplicationPath("/v1.0") public class JerseyConfig extends ResourceConfig { public JerseyConfig() { packages("com.raibledesigns.boot.service"); property(ServerProperties.BV_SEND_ERROR_IN_RESPONSE, true); property(ServerProperties.JSON_PROCESSING_FEATURE_DISABLE, false); property(ServerProperties.MOXY_JSON_FEATURE_DISABLE, true); property(ServerProperties.WADL_FEATURE_DISABLE, true); register(LoggingFilter.class); register(JacksonFeature.class); } } Finally, I created MvcConfig.java to set the welcome page. package com.raibledesigns.boot.config; import org.springframework.context.annotation.Configuration; import org.springframework.web.servlet.config.annotation.ViewControllerRegistry; import org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter; @Configuration public class MvcConfig extends WebMvcConfigurerAdapter { @Override public void addViewControllers(ViewControllerRegistry registry) { registry.addViewController("/").setViewName("index"); } } To cleanup, I deleted src/main/webapp/WEB-INF and created src/main/resources/logback.xml: Since Boot doesn't support JSP out-of-the-box, I renamed my index.jsp file to index.html and changed the URL in it to point to "/v1.0/hello". I was pleased to see that everything worked nicely. I learned shortly after that I could remove the Spring BOM since Spring Boot uses a property to control its Spring version. The only issue I found is when started the app with "mvn package && java -jar target/app.war", it failed to initialize Jersey. I tried adding a @Bean for the servlet: @Bean public ServletRegistrationBean jerseyServlet() { ServletRegistrationBean registration = new ServletRegistrationBean(new ServletContainer(), "/v1.0/*"); registration.addInitParameter(ServletProperties.JAXRS_APPLICATION_CLASS, JerseyConfig.class.getName()); return registration; } Unfortunately, when running it using "java -jar", I get the following error: org.glassfish.hk2.api.MultiException: A MultiException has 1 exceptions. They are: 1. org.glassfish.jersey.server.internal.scanning.ResourceFinderException: java.io.FileNotFoundException: /.../target/app.war!/WEB-INF/classes (No such file or directory) at org.jvnet.hk2.internal.Utilities.justCreate(Utilities.java:869) at org.jvnet.hk2.internal.ServiceLocatorImpl.create(ServiceLocatorImpl.java:814) at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:906) at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:898) at org.glassfish.jersey.server.ApplicationHandler.createApplication(ApplicationHandler.java:300) at org.glassfish.jersey.server.ApplicationHandler.(ApplicationHandler.java:279) at org.glassfish.jersey.servlet.WebComponent.(WebComponent.java:302) This seems strange since there is a WEB-INF/classes in my WAR. Regardless, this is not a Boot problem per se, but more of a Jersey issue. From one of the Boot developers: The whole idea with Boot is that servlets are just a transport - they are a means to an end, and hopefully not the only one - the "container" is Spring, not the servlet container. We probably could add some form of support for SCI but only by hacking the containers since the spec really doesn't allow for much control of their lifecycle. It hasn't been a priority so far. Summary I hope this article is useful to see how you to upgrade your Java webapps to use Spring 4 and Spring Boot. I've created a boot-makeover project on GitHub with all the code mentioned. You can also view the commits for each step.

CMS, or content management systems, are platforms for managing and administering website content. There is no denying that CMSes are important in today's web ecosystem. These content management systems not only provide an easy way to build and maintain websites, but they also lend a helping hand in updating and editing website content without the need to spend hours or days writing and altering codes and scripts. Some of the leading CMSes are PHP-based, Ruby on Rails-based, ASP.NET-based, and Java-based. Among these, due to scalability, modernized architecture and open-source standards of a few, Java-based CMSs are getting quite a lot of attention lately, especially for enterprise websites, because of the scalable, modern, open source technology behind most of them. There are plenty of CMS tools based on Java to help developers create multi-lingual and multi-channel websites. But how do we decide on the best one for our use case? In this article, we’re going to explore the top 24 content management systems based on Java. Let’s have a look at each of them in detail: 1. Alfresco : Alfresco is one of the top open-source content management systems of Java. It comes with enterprise repository and portlet capabilities along with document management, collaboration, records management, knowledge management, web content management, imaging, and a lot more. Alfresco has a modular architecture and enables end users to efficiently manage websites across the cloud, mobile, hybrid and on-premise environments using open source Java technologies, such as Spring, Hibernate, Lucene and JSF. 2. Magnolia : Magnolia is a well-documented, easy to use, enterprise-grade open source CMS based on the Java Content Repository Standard. It is a highly popular CMS due to its out-of-the-box functionality and ease of use under an open source license. Moreover, Magnolia supports unique content delivery capabilities in a search-engine optimized manner and also follows W3C standards. Magnolia CMS has been deployed by enterprises and governments in more than 100 countries across the world. Here's a case study on Magnolia-based website development 3. LogicalDOC : Though less known than other software such as Alfresco, LogicalDOC is emerging as a powerful and more affordable alternative. With primary focus on Document Management, it offers very interesting content management, knowledge management and collaboration features, and all this in a really efficient way. A peculiar aspect of the interface is the use of Google GWT , this makes the user interface very responsive while the data transfer with the server is minimum. Also the availability of Free Apps for Android and Apple devices (iPhone and iPad) is an interesting feature. 4. Asbru: Asbru is another powerful, fully-featured, easy to use content management system with database-driven capabilities. It is built on the Spring framework with integrated community, databases, eCommerce and statistics modules, which helps developers to create, publish and manage rich and user-friendly internet, extranet and intranet websites on the go. Available in various editions, Asbru provides users with a simple, user-friendly platform to manage websites along with a host of other benefits and features such as custom templates and data, password protected content, multi-lingual content, communities, eCommerce and website analytics, a cutting-edge WYSIWYG content editor and a lot more. 5. OpenCMS : OpenCMS is based on Java and XML technology that allows you to build highly customizable and interactive websites and portals. It comes integrated with a WYSIWYG editor and fully-featured Template Engine which is fully compliant with W3C standards. OpenCMS can be deployed both in an open-source environment (Linux, Apache, Tomcat, MySQL) as well as a commercial environment (Windows NT, IIS, BEA Weblogic, Oracle) 6. Walrus: Walrus is yet another Spring-based CMS that provides unique and effective content management capabilities with a smart administrative interface and drag-and-drop facilities. Easy-to-setup and undo/redo features make Walrus a highly preferred and suitable CMS for government and non-profit enterprises. 7. Pulse : Pulse is a Java-based framework and portal solution that offers easy-to-use and extensible patterns for creating rich browser web applications and responsive websites. It brings a bunch of innovative and powerful components including content management, web shops, user management and more. A few of its key features include a WebDAV based virtual file system for digital asset management, mature user and role management, built-in internationalization, and more. 8. MeshCMS: MeshCMS is an easy to use online editing system written in Java. It comes with a host of features that you will find in any ideal content management system however, it uses a conventional approach in managing and editing website content. It is considered one of the fastest CMSes for editing files online, managing files, and building some very common components like menus, breadcrumbs, mail forms and so on. MeshCMS is accompanied by cross-browser capabilities, a WYSIWYG editor, hot-linking prevention, and tag-library that makes content management an interesting affair. 9. Liferay: Liferay is one of the most popular CMSes based on Java, and is recommended by many industry experts. It comes with awesome features that can make your content management tasks simple. Liferay is a very popular for developing personal as well as professional websites with ease. 10. DotCMS: DotCMS is a next-gen enterprise CMS that wears an open-source hat. It is highly popular and widely used CMS due to its open APIs, extensible and scalable architecture that it used to create personalized and engaging websites, intranets, extranets and applications with ease. 11. Jease: Jease highly known as ‘Java with ease’ is another open source content management system that is built on popular Java technologies like db40, Perst, Lucence and ZK. It is an extremely lightweight CMS with excellent Ajax interface. Due to its intuitive and interactive interface, it is highly simple and easy to customize and deploy websites in Jease even for inexperienced Java developers. 12. Hippo: Hippo is again a powerful open-source CMS made in Java that features enterprise level capabilities that helps in delivering personalized websites and channels. Hippo outlines its competitor by delivering outstanding customer experience through innovative solutions. Hippo has come a long way since 1999 serving medium to large organizations by offering a personalized multichannel content distribution platform including website, mobile, tablet, extranets and intranets. Its major version update was in December 2012 and since then it is seeing minor updates every couple of months. 13. Apache Lenya: Apache Lenya is another open-source Java CMS that features revision control , multisite management, scheduling, search, WYSIWYG editors, and workflow which makes website development and management quite interesting and easy for developers. Available in a variety of languages, Apache Lenya is highly preferred CMS among enterprises that desire to develop multi-lingual websites. 14. Contelligent: Conteligent is another smart CMS solution offered under Java technology stack. It is fully compliant with J2EE and offers great solution for creating and managing personalized websites. 15. InfoGlue: InfoGlue again is a Java-based CMS that is known for its advanced, scalable and robust open-source architecture. It is a highly flexible CMS built on JSR-168 and comes with full multi-language support, excellent information reuse and high integration capabilities. 16. OpenEdit: OpenEdit CMS is a dynamic tool for managing website content with online editing capabilities. Built in open-source architecture, OpenEdit provides facilities like user manager, file manager, version control and notification tools for managing media-rich websites. OpenCMS features enterprise grade plugins such as eCommerce, Content Management, Blog, Events Calendar, Social Networking Tools and more. 17. AtLeap: Atleap is a multi-lingual CMS based on Java which offers amazing content delivery assistance with SEO and full text search functionalities. AtLeap, a product of Blandware, is not only a CMS but a highly robust framework for developing website and web applications 18. Weceem: Weceem is yet another open source content management system, unlikely other CMS it is built upon well-known Java framework grails, spring and Java itself. Weceem has garner positive reviews and is an ideal CMS when it comes to grails, but faces tough competition in best Java CMS category. I came across a LinkedIn discussion which was enough for me to put this CMS in the Best Java CMS list. 19. Nuxeo: Nuxeno is a powerful open source CMS built on Java-based architecture. It offers solutions related to document management, case management and digital asset management. It is free from licensing free but do costs you when reach out for support and maintenance help. It has strong groups of customers including Electronic Arts, U.S. Navy and as stated on the company website, it’s been used in over 145 countries across thousands of organizations. 20. XperienCentral: Xperien central is currently the only CMS that offers unique content to a visitor as per his earlier journey, so you can tailor the content to increase the conversion. It offers multi-channel content delivery across website, mobile social media channels and applications. It is built on Java and hence it is extremely scalable and agile. 21 Atex: Atex is a web CMS that uses polopoly technology to deliver content. As per claims, it is the only industry leading CMS with built in paywall. Atex again is one of the premium CMS that offers amazing solutions for managing websites and helps marketers deliver the right content to relevant audiences. It has rich set of clientele. 22 Escenic: Customers of escenic include News of the World, The Sun, The Times, the Independent titles. It’s a closed source Java framework. Both Atex and Escenic are found to be highly popular in Sweden. Some of the biggest sites in Sweden use both these CMS. idg.se uses Atex and Aftonbladet.se uses Escenic 23. Adobe Experience Manager/ CQ5 : Best CMS list cannot be completed without including adobe experience manager. It is an all-round CMS which offers all kinds of agility and flexibility an organization may want. It helps deliver unique customer experience by delivering different content on different channels. Adobe Experience Manager was recently named a leader in web content management by Garnters magic quadrant. Earlier it was known as CQ5 but later was acquire by Adobe in 2010 24. SDL Tridion Again a well known CMS and highly recommended by industry experts. Its simple intuitive UI makes it simple to manage the content and deliver it uniformly across all channels. It recently received top score in overall content management experience according to an independent research firm - Forrester Research, Inc., This completes the list 23 top Java-based content management system. Hope after reading about all the CMS, you have got enough inferences and insight as to which CMS would be best for your website development project.

European countries are moving towards Single Euro Payments Area (SEPA) initiative and therefore a lot of companies are implementing International Bank Account Numbers (IBAN) or replacing existing bank account details with IBAN. Recently I was dealing with International Bank Account Numbers and couldn't find any open source library neither to generate them nor to validate them. To fill the gap I implemented iban4j Java library for International Bank Account Number (IBAN) generation and validation. Using iban4j is as easy as: Iban iban = new Iban.Builder() .countryCode(CountryCode.AT) .bankCode("19043") .accountNumber("00234573201") .build(); Library is available in maven central: org.iban4j iban4j 1.0.0 For more details check out github page

Jetty 9.1 is finally released, bringing Java WebSockets (JSR-356) to non-EE environments. It's awesome news and today's post will be about using this great new API along with Spring Framework. JSR-356 defines concise, annotation-based model to allow modern Java web applications easily create bidirectional communication channels using WebSockets API. It covers not only server-side, but client-side as well, making this API really simple to use everywhere. Let's get started! Our goal would be to build a WebSockets server which accepts messages from the clients and broadcasts them to all other clients currently connected. To begin with, let's define the message format, which server and client will be exchanging, as this simple Message class. We can limit ourselves to something like a String, but I would like to introduce to you the power of another new API - Java API for JSON Processing (JSR-353). package com.example.services; public class Message { private String username; private String message; public Message() { } public Message( final String username, final String message ) { this.username = username; this.message = message; } public String getMessage() { return message; } public String getUsername() { return username; } public void setMessage( final String message ) { this.message = message; } public void setUsername( final String username ) { this.username = username; } } To separate the declarations related to the server and the client, JSR-356 defines two basic annotations:@ServerEndpoint and @ClientEndpoit respectively. Our client endpoint, let's call itBroadcastClientEndpoint, will simply listen for messages server sends: package com.example.services; import java.io.IOException; import java.util.logging.Logger; import javax.websocket.ClientEndpoint; import javax.websocket.EncodeException; import javax.websocket.OnMessage; import javax.websocket.OnOpen; import javax.websocket.Session; @ClientEndpoint public class BroadcastClientEndpoint { private static final Logger log = Logger.getLogger( BroadcastClientEndpoint.class.getName() ); @OnOpen public void onOpen( final Session session ) throws IOException, EncodeException { session.getBasicRemote().sendObject( new Message( "Client", "Hello!" ) ); } @OnMessage public void onMessage( final Message message ) { log.info( String.format( "Received message '%s' from '%s'", message.getMessage(), message.getUsername() ) ); } } That's literally it! Very clean, self-explanatory piece of code: @OnOpen is being called when client got connected to the server and @OnMessage is being called every time server sends a message to the client. Yes, it's very simple but there is a caveat: JSR-356 implementation can handle any simple objects but not the complex ones like Message is. To manage that, JSR-356 introduces concept of encoders and decoders. We all love JSON, so why don't we define our own JSON encoder and decoder? It's an easy task which Java API for JSON Processing (JSR-353) can handle for us. To create an encoder, you only need to implementEncoder.Text< Message > and basically serialize your object to some string, in our case to JSON string, using JsonObjectBuilder. package com.example.services; import javax.json.Json; import javax.json.JsonReaderFactory; import javax.websocket.EncodeException; import javax.websocket.Encoder; import javax.websocket.EndpointConfig; public class Message { public static class MessageEncoder implements Encoder.Text< Message > { @Override public void init( final EndpointConfig config ) { } @Override public String encode( final Message message ) throws EncodeException { return Json.createObjectBuilder() .add( "username", message.getUsername() ) .add( "message", message.getMessage() ) .build() .toString(); } @Override public void destroy() { } } } For decoder part, everything looks very similar, we have to implement Decoder.Text< Message > and deserialize our object from string, this time using JsonReader. package com.example.services; import javax.json.JsonObject; import javax.json.JsonReader; import javax.json.JsonReaderFactory; import javax.websocket.DecodeException; import javax.websocket.Decoder; public class Message { public static class MessageDecoder implements Decoder.Text< Message > { private JsonReaderFactory factory = Json.createReaderFactory( Collections.< String, Object >emptyMap() ); @Override public void init( final EndpointConfig config ) { } @Override public Message decode( final String str ) throws DecodeException { final Message message = new Message(); try( final JsonReader reader = factory.createReader( new StringReader( str ) ) ) { final JsonObject json = reader.readObject(); message.setUsername( json.getString( "username" ) ); message.setMessage( json.getString( "message" ) ); } return message; } @Override public boolean willDecode( final String str ) { return true; } @Override public void destroy() { } } } And as a final step, we need to tell the client (and the server, they share same decoders and encoders) that we have encoder and decoder for our messages. The easiest thing to do that is just by declaring them as part of @ServerEndpoint and @ClientEndpoit annotations. import com.example.services.Message.MessageDecoder; import com.example.services.Message.MessageEncoder; @ClientEndpoint( encoders = { MessageEncoder.class }, decoders = { MessageDecoder.class } ) public class BroadcastClientEndpoint { } To make client's example complete, we need some way to connect to the server usingBroadcastClientEndpoint and basically exchange messages. The ClientStarter class finalizes the picture: package com.example.ws; import java.net.URI; import java.util.UUID; import javax.websocket.ContainerProvider; import javax.websocket.Session; import javax.websocket.WebSocketContainer; import org.eclipse.jetty.websocket.jsr356.ClientContainer; import com.example.services.BroadcastClientEndpoint; import com.example.services.Message; public class ClientStarter { public static void main( final String[] args ) throws Exception { final String client = UUID.randomUUID().toString().substring( 0, 8 ); final WebSocketContainer container = ContainerProvider.getWebSocketContainer(); final String uri = "ws://localhost:8080/broadcast"; try( Session session = container.connectToServer( BroadcastClientEndpoint.class, URI.create( uri ) ) ) { for( int i = 1; i <= 10; ++i ) { session.getBasicRemote().sendObject( new Message( client, "Message #" + i ) ); Thread.sleep( 1000 ); } } // Application doesn't exit if container's threads are still running ( ( ClientContainer )container ).stop(); } } Just couple of comments what this code does: we are connecting to WebSockets endpoint atws://localhost:8080/broadcast, randomly picking some client name (from UUID) and generating 10 messages, every with 1 second delay (just to be sure we have time to receive them all back). Server part doesn't look very different and at this point could be understood without any additional comments (except may be the fact that server just broadcasts every message it receives to all connected clients). Important to mention here: new instance of the server endpoint is created every time new client connects to it (that's why peers collection is static), it's a default behavior and could be easily changed. package com.example.services; import java.io.IOException; import java.util.Collections; import java.util.HashSet; import java.util.Set; import javax.websocket.EncodeException; import javax.websocket.OnClose; import javax.websocket.OnMessage; import javax.websocket.OnOpen; import javax.websocket.Session; import javax.websocket.server.ServerEndpoint; import com.example.services.Message.MessageDecoder; import com.example.services.Message.MessageEncoder; @ServerEndpoint( value = "/broadcast", encoders = { MessageEncoder.class }, decoders = { MessageDecoder.class } ) public class BroadcastServerEndpoint { private static final Set< Session > sessions = Collections.synchronizedSet( new HashSet< Session >() ); @OnOpen public void onOpen( final Session session ) { sessions.add( session ); } @OnClose public void onClose( final Session session ) { sessions.remove( session ); } @OnMessage public void onMessage( final Message message, final Session client ) throws IOException, EncodeException { for( final Session session: sessions ) { session.getBasicRemote().sendObject( message ); } } } In order this endpoint to be available for connection, we should start the WebSockets container and register this endpoint inside it. As always, Jetty 9.1 is runnable in embedded mode effortlessly: package com.example.ws; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.servlet.DefaultServlet; import org.eclipse.jetty.servlet.ServletContextHandler; import org.eclipse.jetty.servlet.ServletHolder; import org.eclipse.jetty.websocket.jsr356.server.deploy.WebSocketServerContainerInitializer; import org.springframework.web.context.ContextLoaderListener; import org.springframework.web.context.support.AnnotationConfigWebApplicationContext; import com.example.config.AppConfig; public class ServerStarter { public static void main( String[] args ) throws Exception { Server server = new Server( 8080 ); // Create the 'root' Spring application context final ServletHolder servletHolder = new ServletHolder( new DefaultServlet() ); final ServletContextHandler context = new ServletContextHandler(); context.setContextPath( "/" ); context.addServlet( servletHolder, "/*" ); context.addEventListener( new ContextLoaderListener() ); context.setInitParameter( "contextClass", AnnotationConfigWebApplicationContext.class.getName() ); context.setInitParameter( "contextConfigLocation", AppConfig.class.getName() ); server.setHandler( context ); WebSocketServerContainerInitializer.configureContext( context ); server.start(); server.join(); } } The most important part of the snippet above is WebSocketServerContainerInitializer.configureContext: it's actually creates the instance of WebSockets container. Because we haven't added any endpoints yet, the container basically sits here and does nothing. Spring Framework and AppConfig configuration class will do this last wiring for us. package com.example.config; import javax.annotation.PostConstruct; import javax.inject.Inject; import javax.websocket.DeploymentException; import javax.websocket.server.ServerContainer; import javax.websocket.server.ServerEndpoint; import javax.websocket.server.ServerEndpointConfig; import org.eclipse.jetty.websocket.jsr356.server.AnnotatedServerEndpointConfig; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.web.context.WebApplicationContext; import com.example.services.BroadcastServerEndpoint; @Configuration public class AppConfig { @Inject private WebApplicationContext context; private ServerContainer container; public class SpringServerEndpointConfigurator extends ServerEndpointConfig.Configurator { @Override public < T > T getEndpointInstance( Class< T > endpointClass ) throws InstantiationException { return context.getAutowireCapableBeanFactory().createBean( endpointClass ); } } @Bean public ServerEndpointConfig.Configurator configurator() { return new SpringServerEndpointConfigurator(); } @PostConstruct public void init() throws DeploymentException { container = ( ServerContainer )context.getServletContext(). getAttribute( javax.websocket.server.ServerContainer.class.getName() ); container.addEndpoint( new AnnotatedServerEndpointConfig( BroadcastServerEndpoint.class, BroadcastServerEndpoint.class.getAnnotation( ServerEndpoint.class ) ) { @Override public Configurator getConfigurator() { return configurator(); } } ); } } As we mentioned earlier, by default container will create new instance of server endpoint every time new client connects, and it does so by calling constructor, in our caseBroadcastServerEndpoint.class.newInstance(). It might be a desired behavior but because we are usingSpring Framework and dependency injection, such new objects are basically unmanaged beans. Thanks to very well-thought (in my opinion) design of JSR-356, it's actually quite easy to provide your own way of creating endpoint instances by implementing ServerEndpointConfig.Configurator. TheSpringServerEndpointConfigurator is an example of such implementation: it's creates new managed bean every time new endpoint instance is being asked (if you want single instance, you can create singleton of the endpoint as a bean in AppConfig and return it all the time). The way we retrieve the WebSockets container is Jetty-specific: from the attribute of the context with name"javax.websocket.server.ServerContainer" (it probably might change in the future). Once container is there, we are just adding new (managed!) endpoint by providing our own ServerEndpointConfig (based onAnnotatedServerEndpointConfig which Jetty kindly provides already). To build and run our server and clients, we need just do that: mvn clean package java -jar target\jetty-web-sockets-jsr356-0.0.1-SNAPSHOT-server.jar // run server java -jar target/jetty-web-sockets-jsr356-0.0.1-SNAPSHOT-client.jar // run yet another client As an example, by running server and couple of clients (I run 4 of them, '392f68ef', '8e3a869d', 'ca3a06d0', '6cb82119') you might see by the output in the console that each client receives all the messages from all other clients (including its own messages): Nov 29, 2013 9:21:29 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Hello!' from 'Client' Nov 29, 2013 9:21:29 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #1' from '392f68ef' Nov 29, 2013 9:21:29 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #2' from '8e3a869d' Nov 29, 2013 9:21:29 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #7' from 'ca3a06d0' Nov 29, 2013 9:21:30 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #4' from '6cb82119' Nov 29, 2013 9:21:30 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #2' from '392f68ef' Nov 29, 2013 9:21:30 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #3' from '8e3a869d' Nov 29, 2013 9:21:30 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #8' from 'ca3a06d0' Nov 29, 2013 9:21:31 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #5' from '6cb82119' Nov 29, 2013 9:21:31 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #3' from '392f68ef' Nov 29, 2013 9:21:31 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #4' from '8e3a869d' Nov 29, 2013 9:21:31 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #9' from 'ca3a06d0' Nov 29, 2013 9:21:32 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #6' from '6cb82119' Nov 29, 2013 9:21:32 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #4' from '392f68ef' Nov 29, 2013 9:21:32 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #5' from '8e3a869d' Nov 29, 2013 9:21:32 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #10' from 'ca3a06d0' Nov 29, 2013 9:21:33 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #7' from '6cb82119' Nov 29, 2013 9:21:33 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #5' from '392f68ef' Nov 29, 2013 9:21:33 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #6' from '8e3a869d' Nov 29, 2013 9:21:34 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #8' from '6cb82119' Nov 29, 2013 9:21:34 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #6' from '392f68ef' Nov 29, 2013 9:21:34 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #7' from '8e3a869d' Nov 29, 2013 9:21:35 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #9' from '6cb82119' Nov 29, 2013 9:21:35 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #7' from '392f68ef' Nov 29, 2013 9:21:35 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #8' from '8e3a869d' Nov 29, 2013 9:21:36 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #10' from '6cb82119' Nov 29, 2013 9:21:36 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #8' from '392f68ef' Nov 29, 2013 9:21:36 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #9' from '8e3a869d' Nov 29, 2013 9:21:37 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #9' from '392f68ef' Nov 29, 2013 9:21:37 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #10' from '8e3a869d' Nov 29, 2013 9:21:38 PM com.example.services.BroadcastClientEndpoint onMessage INFO: Received message 'Message #10' from '392f68ef' 2013-11-29 21:21:39.260:INFO:oejwc.WebSocketClient:main: Stopped org.eclipse.jetty.websocket.client.WebSocketClient@3af5f6dc Awesome! I hope this introductory blog post shows how easy it became to use modern web communication protocols in Java, thanks to Java WebSockets (JSR-356), Java API for JSON Processing (JSR-353) and great projects such as Jetty 9.1! As always, complete project is available on GitHub.