In this example I will show how to test Spring Integration flow using Mock SftpServer.

for people in hurry, get the latest code and the steps in github . to run the junit test, run “mvn test” and understand the test flow. introduction: fakeftpserver in this spring integration fakeftpserver example, i will demonstrate using spring fakeftpserver to junit test a spring integration flow. this is an interesting topic, and there are few articles on unit testing file transfers , which gives some insight on this topic. in this blog, we will test a spring integration flow which checks for a list of files, apply a splitter to separate each file and start downloading them into a local location. once the download is complete, it will delete the files on the ftp server. in my next blog, i will show how to do junit testing of spring integration flow with sftp server. spring integration flow spring integration fakeftpserver example in order to use fakeftpserver we need to have maven dependency as below, org.mockftpserver mockftpserver 2.3 test the first step to this is to create a fakeftpserver before every test runs as below, @before public void setup() throws exception { fakeftpserver = new fakeftpserver(); fakeftpserver.setservercontrolport(9999); // use any free port filesystem filesystem = new unixfakefilesystem(); filesystem.add(new fileentry(file, contents)); fakeftpserver.setfilesystem(filesystem); useraccount useraccount = new useraccount("user", "password", home_dir); fakeftpserver.adduseraccount(useraccount); fakeftpserver.start(); } @after public void teardown() throws exception { fakeftpserver.stop(); } finally run the junit test case as seen below, @autowired private filedownloadutil downloadutil; @test public void testftpdownload() throws exception { file file = new file("src/test/resources/output"); delete(file); ftpclient client = new ftpclient(); client.connect("localhost", 9999); client.login("user", "password"); string files[] = client.listnames("/dir"); client.help(); logger.debug("before delete" + files[0]); assertequals(1, files.length); downloadutil.downloadfilesfromremotedirectory(); logger.debug("after delete"); files = client.listnames("/dir"); client.help(); assertequals(0, files.length); assertequals(1, file.list().length); } i hope this blog helped.

What is a Theory? Functionally, a theory is an alternative to JUnit's parameterized tests. Semantically, a theory encapsulates the tester's understanding of an object's universal behavior. That is, whatever it is that a theory asserts, it is expected to be true for all data. Theories should be especially useful for finding bugs in edge cases. Contrast this with a typical unit test, which asserts that a specific data point will have a specific outcome, and only asserts that. (For this reason, typical unit tests are sometimes called example-based tests to contrast them with theories.) This is very nice in theory, but... A @Theory needs a special JUnit runner (Theories.class). So if you want to use Spring and/or Mockito together with theories, you have a problem. All of these features need a different runner and you can only use one on each test class. The solution For Mockito is easy. Instead of using the @Mock annotiation, you can use the static createMock method. One problem solved. For Spring is a little bit trickier. First of all, you have to use @ContextConfiguration to declare the XML with the bean definitions that you need. But the trickiest part is that you have to tell Spring how to do the autowiring without using its own runner. This can be accomplish adding this line to the @Before method: new TestContextManager(getClass()).prepareTestInstance(this); Basic Usage Example package org.mackenzine.theories; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotNull; import static org.mockito.Mockito.when; import java.util.Date; import org.junit.Before; import org.junit.experimental.theories.DataPoints; import org.junit.experimental.theories.Theories; import org.junit.experimental.theories.Theory; import org.junit.runner.RunWith; import org.mockito.Mockito; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.TestContextManager; @RunWith(Theories.class) @ContextConfiguration("classpath:parser.xml") public class QuoteTheoriesTest { private static String deleteMessage = "deleteMessage"; private static String updateMessage = "updateMessage"; private QuoteFactory factory; private final Event event = Mockito.mock(Event.class); private final Contract contract = Mockito.mock(Contract.class); private final Commodity commodity = Mockito.mock(Commodity.class); @Autowired private Parser parser; @Before public void setUp() throws Exception { factory = new QuoteFactory(); new TestContextManager(getClass()).prepareTestInstance(this); } @DataPoints public static String[] getEventTypes() { return new String[] { updateMessage, deleteMessage }; } @Theory public void shouldCreateQuote(final String message) throws Exception { Date now = new Date(); when(event.getParsedMessage()).thenReturn(parser.parse(message)); when(event.getContract()).thenReturn(contract); when(event.getTradeDate()).thenReturn(now); when(contract.getExternalCode()).thenReturn("externalCode"); when(contract.getCommodity()).thenReturn(commodity); when(commodity.getCommodityCode()).thenReturn("code"); Quote quote = factory.createQuote(event); assertNotNull(quote); assertEquals("code", quote.getCommodityCode()); assertEquals(now, quote.getTradeDate()); } } Sources Definition of Theories: https://blogs.oracle.com/jacobc/entry/junit_theories Original Idea for Parameterized Tests: http://stackoverflow.com/questions/8974977/spring-parameterized-theories-junit-tests Thread on SpringSource: http://forum.springsource.org/showthread.php?78929-Is-Theory-supported Open Issue in SpringSource for Parameterized Tests (not for Theories): https://jira.springsource.org/browse/SPR-5292

YAML is well-known format within Ruby community, quite widely used for a long time now. But we as Java developers mostly deal with property files and XMLs in case we need some configuration for our apps. How many times we needed to express complicated configuration by inventing our own XML schema or imposing property names convention? Though JSON is becoming a popular format for web applications, using JSON files to describe the configuration is a bit cumbersome and, in my opinion, is not as expressive as YAML. Let's see what YAML can do for us to make our life easier. For sure, let's start with the problem. In order for our application to function properly, we need to feed it following data somehow: version and release date database connection parameters list of supported protocols list of users with their passwords This list of parameters sounds a bit weird, but the purpose is to demonstrate different data types in work: strings, numbers, dates, lists and maps. The Java model consists of two simple classes: Connection package com.example.yaml; public final class Connection { private String url; private int poolSize; public String getUrl() { return url; } public void setUrl(String url) { this.url = url; } public int getPoolSize() { return poolSize; } public void setPoolSize(int poolSize) { this.poolSize = poolSize; } @Override public String toString() { return String.format( "'%s' with pool of %d", getUrl(), getPoolSize() ); } } and Configuration, both are typical Java POJOs, verbose because of property setters and getters (we get used to it, right?). package com.example.yaml; import static java.lang.String.format; import java.util.Date; import java.util.List; import java.util.Map; public final class Configuration { private Date released; private String version; private Connection connection; private List< String > protocols; private Map< String, String > users; public Date getReleased() { return released; } public String getVersion() { return version; } public void setReleased(Date released) { this.released = released; } public void setVersion(String version) { this.version = version; } public Connection getConnection() { return connection; } public void setConnection(Connection connection) { this.connection = connection; } public List< String > getProtocols() { return protocols; } public void setProtocols(List< String > protocols) { this.protocols = protocols; } public Map< String, String > getUsers() { return users; } public void setUsers(Map< String, String > users) { this.users = users; } @Override public String toString() { return new StringBuilder() .append( format( "Version: %s\n", version ) ) .append( format( "Released: %s\n", released ) ) .append( format( "Connecting to database: %s\n", connection ) ) .append( format( "Supported protocols: %s\n", protocols ) ) .append( format( "Users: %s\n", users ) ) .toString(); } } ow, as model is quite clear, let us try to express it as the human being normally does it. Looking back to our list of required configuration, let's try to write it down one by one. 1. version and release date version: 1.0 released: 2012-11-30 2. database connection parameters connection: url: jdbc:mysql://localhost:3306/db poolSize: 5 3. list of supported protocols protocols: - http - https 4. list of users with their passwords users: tom: passwd bob: passwd And this is it, our configuration expressed in YAML syntax is completed! The whole file sample.yml looks like this: version: 1.0 released: 2012-11-30 # Connection parameters connection: url: jdbc:mysql://localhost:3306/db poolSize: 5 # Protocols protocols: - http - https # Users users: tom: passwd bob: passwd To make it work in Java, we just need to use the awesome library called snakeyml, respectively the Maven POM file is quite simple: 4.0.0 com.example yaml 0.0.1-SNAPSHOT jar UTF-8 org.yaml snakeyaml 1.11 org.apache.maven.plugins maven-compiler-plugin 2.3.1 1.7 1.7 Please notice the usage of Java 1.7, the language extensions and additional libraries simplify a lot of regular tasks as we could see looking into YamlConfigRunner: package com.example.yaml; import java.io.IOException; import java.io.InputStream; import java.nio.file.Files; import java.nio.file.Paths; import org.yaml.snakeyaml.Yaml; public class YamlConfigRunner { public static void main(String[] args) throws IOException { if( args.length != 1 ) { System.out.println( "Usage: " ); return; } Yaml yaml = new Yaml(); try( InputStream in = Files.newInputStream( Paths.get( args[ 0 ] ) ) ) { Configuration config = yaml.loadAs( in, Configuration.class ); System.out.println( config.toString() ); } } } The code snippet here loads the configuration from file (args[ 0 ]), tries to parse it and fill up the Configuration class with meaningful data using JavaBeans conventions, converting to the declared types where possible. Running this class with sample.yml as an argument generates the following output: Version: 1.0 Released: Thu Nov 29 19:00:00 EST 2012 Connecting to database: 'jdbc:mysql://localhost:3306/db' with pool of 5 Supported protocols: [http, https] Users: {tom=passwd, bob=passwd} Totally identical to the values we have configured!

When running a Java program, verbose options can be used to tell the JVM which kind of information to see. JVM suports three verbose options out of the box. As the name suggests, verbose is for displaying the work done by JVM. Mostly the information provided by these parameters is used for debugging purposes. Since it is used for debugging, its use is in development. One would never have to use verbose parameters in production enviornment. The three verbose options supported by JVM are: -verbose:class -verbose:gc -verbose:jni -verbose:class is used to display the information about classes being loaded by JVM. This is useful when using class loaders for loading classes dynamically or for analysing what all classes are getting loaded in a particular scenario. A very simple program which does nothing also loads so many classes as shown below: package com.example; public class Test { public static void main(String args[]) { } } Output: [Opened C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.lang.Object from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.io.Serializable from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.lang.Comparable from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.lang.CharSequence from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] .............................................................................. .............................................................................. .............................................................................. [Loaded java.lang.Void from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.lang.Shutdown from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.lang.Shutdown$Lock from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] I am showing only selected output from Eclipse console. Classes from java.lang, java.io and java.util packages are loaded into memory by default. As stated earlier, one would use the verbose:class option when checking for what all classes are getting loaded which is usually a requirement when working with class loaders in Java. -verbose:gc is used to check garbage collection event information. When used as command line argument for running Java program, the details of garbage collection are printed on the console. The following program demonstrates the usage: package com.example; public class Test { public static void main(String args[]) throws InterruptedException { Test t1 = new Test(); t1=null; System.gc(); } } Output: [GC 318K->304K(61056K), 0.0081277 secs] [Full GC 304K->225K(61056K), 0.0054004 secs] -verbose:jni is used for printing the native methods as and when they are registered in the application. These methods include JDK as well as custom native methods. Note that jni stands for Java Native Interface. The sourcecode and output demonstrating the usage of this option is shown below: package com.example; public class Test { public static void main(String args[]) throws InterruptedException { } } Output: [Dynamic-linking native method java.lang.Object.registerNatives ... JNI] [Registering JNI native method java.lang.Object.hashCode] [Registering JNI native method java.lang.Object.wait] [Registering JNI native method java.lang.Object.notify] [Registering JNI native method java.lang.Object.notifyAll] [Registering JNI native method java.lang.Object.clone] [Dynamic-linking native method java.lang.System.registerNatives ... JNI] ............................................................... ............................................................... ............................................................... [Registering JNI native method sun.misc.Perf.highResCounter] [Registering JNI native method sun.misc.Perf.highResFrequency] [Dynamic-linking native method java.lang.ClassLoader.defineClass1 ... JNI] [Dynamic-linking native method java.lang.Runtime.gc ... JNI] [Dynamic-linking native method java.lang.ref.Finalizer.invokeFinalizeMethod ... JNI]

Groovy has an extension to its HTTPBuilder class called RESTClient which makes it fairly easy to test a RESTful web service.

If you're writing a desktop or Java Web Start program in Java using Swing, you might feel the need to run some stuff in the background by creating your own threads. There's nothing stopping you from using standard multi-threading techniques in Swing, and the usual considerations apply. If you have multiple threads accessing the same variables, you'll need to use synchronized methods or code blocks (or thread-safe classes like AtomicInteger or ArrayBlockingQueue). However, there is a pitfall for the unwary. As with most user interface APIs, you can't update the user interface from threads you've created yourself. Well, as every Java undergraduate knows, you often can, but you shouldn't. If you do this, sometimes your program will work and other times it won't. You can get around this problem by using the specialised SwingWorker class. In this article, I'll show you how you can get your programs working even if you're using the Thread class, and then we'll go on to look at the SwingWorker solution. For demonstration purposes, I've created a little Swing program. As you can see, it consists of two labels and a start button. At the moment, clicking the start button invokes a handler method which does nothing. Here's the Java code: import java.awt.Font; import java.awt.GridBagConstraints; import java.awt.GridBagLayout; import java.awt.event.ActionEvent; import java.awt.event.ActionListener; import java.util.List; import java.util.concurrent.ExecutionException; import javax.swing.JButton; import javax.swing.JFrame; import javax.swing.JLabel; import javax.swing.SwingUtilities; import javax.swing.SwingWorker; public class MainFrame extends JFrame { private JLabel countLabel1 = new JLabel("0"); private JLabel statusLabel = new JLabel("Task not completed."); private JButton startButton = new JButton("Start"); public MainFrame(String title) { super(title); setLayout(new GridBagLayout()); countLabel1.setFont(new Font("serif", Font.BOLD, 28)); GridBagConstraints gc = new GridBagConstraints(); gc.fill = GridBagConstraints.NONE; gc.gridx = 0; gc.gridy = 0; gc.weightx = 1; gc.weighty = 1; add(countLabel1, gc); gc.gridx = 0; gc.gridy = 1; gc.weightx = 1; gc.weighty = 1; add(statusLabel, gc); gc.gridx = 0; gc.gridy = 2; gc.weightx = 1; gc.weighty = 1; add(startButton, gc); startButton.addActionListener(new ActionListener() { public void actionPerformed(ActionEvent arg0) { start(); } }); setSize(200, 400); setDefaultCloseOperation(EXIT_ON_CLOSE); setVisible(true); } private void start() { } public static void main(String[] args) { SwingUtilities.invokeLater(new Runnable() { @Override public void run() { new MainFrame("SwingWorker Demo"); } }); } } We're going to add some code into the start() method which is called in response to the start button being clicked. First let's try a normal thread. private void start() { Thread worker = new Thread() { public void run() { // Simulate doing something useful. for(int i=0; i<=10; i++) { // Bad practice countLabel1.setText(Integer.toString(i)); try { Thread.sleep(1000); } catch (InterruptedException e) { } } // Bad practice statusLabel.setText("Completed."); } }; worker.start(); } As a matter of fact, this code seems to work (at least for me anyway). The program ends up looking like this: This isn't recommended practice, however. We're updating the GUI from our own thread, and under some circumstances that will certainly cause exceptions to be thrown. If we want to update the GUI from another thread, we should use SwingUtilities to schedule our update code to run on the event dispatch thread. The following code is fine, but ugly as the devil himself. private void start() { Thread worker = new Thread() { public void run() { // Simulate doing something useful. for(int i=0; i<=10; i++) { final int count = i; SwingUtilities.invokeLater(new Runnable() { public void run() { countLabel1.setText(Integer.toString(count)); } }); try { Thread.sleep(1000); } catch (InterruptedException e) { } } SwingUtilities.invokeLater(new Runnable() { public void run() { statusLabel.setText("Completed."); } }); } }; worker.start(); } Surely there must be something we can do to make our code more elegant? The SwingWorker Class SwingWorker is an alternative to using the Thread class, specifically designed for Swing. It's an abstract class and it takes two template parameters, which make it look highly ferocious and puts most people off using it. But in fact it's not as complex as it seems. Let's take a look at some code that just runs a background thread. For this first example, we won't be using either of the template parameters, so we'll set them both to Void, Java's class equivalent of the primitive void type (with a lower-case 'v'). Running a Background Task We can run a task in the background by implementing the doInBackground method and calling execute to run our code. SwingWorker worker = new SwingWorker() { @Override protected Void doInBackground() throws Exception { // Simulate doing something useful. for (int i = 0; i <= 10; i++) { Thread.sleep(1000); System.out.println("Running " + i); } return null; } }; worker.execute(); Note that SwingWorker is a one-shot affair, so if we want to run the code again, we'd need to create another SwingWorker; you can't restart the same one. Pretty simple, hey? But what if we want to update the GUI with some kind of status after running our code? You cannot update the GUI from doInBackground, because it's not running in the main event dispatch thread. But there is a solution. We need to make use of the first template parameter. Updating the GUI After the Thread Completes We can update the GUI by returning a value from doInBackground() and then over-riding done(), which can safely update the GUI. We use the get() method to retrieve the value returned from doInBackground() So the first template parameter determines the return type of both doInBackground() and get(). SwingWorker worker = new SwingWorker() { @Override protected Boolean doInBackground() throws Exception { // Simulate doing something useful. for (int i = 0; i <= 10; i++) { Thread.sleep(1000); System.out.println("Running " + i); } // Here we can return some object of whatever type // we specified for the first template parameter. // (in this case we're auto-boxing 'true'). return true; } // Can safely update the GUI from this method. protected void done() { boolean status; try { // Retrieve the return value of doInBackground. status = get(); statusLabel.setText("Completed with status: " + status); } catch (InterruptedException e) { // This is thrown if the thread's interrupted. } catch (ExecutionException e) { // This is thrown if we throw an exception // from doInBackground. } } }; worker.execute(); What if we want to update the GUI as we're going along? That's what the second template parameter is for. Updating the GUI from a Running Thread To update the GUI from a running thread, we use the second template parameter. We call the publish() method to 'publish' the values with which we want to update the user interface (which can be of whatever type the second template parameter specifies). Then we override the process() method, which receives the values that we publish. Actually process() receives lists of published values, because several values may get published before process() is actually called. In this example we just publish the latest value to the user interface. SwingWorker worker = new SwingWorker() { @Override protected Boolean doInBackground() throws Exception { // Simulate doing something useful. for (int i = 0; i <= 10; i++) { Thread.sleep(1000); // The type we pass to publish() is determined // by the second template parameter. publish(i); } // Here we can return some object of whatever type // we specified for the first template parameter. // (in this case we're auto-boxing 'true'). return true; } // Can safely update the GUI from this method. protected void done() { boolean status; try { // Retrieve the return value of doInBackground. status = get(); statusLabel.setText("Completed with status: " + status); } catch (InterruptedException e) { // This is thrown if the thread's interrupted. } catch (ExecutionException e) { // This is thrown if we throw an exception // from doInBackground. } } @Override // Can safely update the GUI from this method. protected void process(List chunks) { // Here we receive the values that we publish(). // They may come grouped in chunks. int mostRecentValue = chunks.get(chunks.size()-1); countLabel1.setText(Integer.toString(mostRecentValue)); } }; worker.execute(); More .... ? You Want More .... ? I hope you enjoyed this introduction to the highly-useful SwingWorker class. You can find more tutorials, including a complete free video course on multi-threading and courses on Swing, Android and Servlets, on my site Cave of Programming. Until next time .... happy coding. - John Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

In an ideal continuous integration pipeline different levels of testing are involved. Individual software modules are typically validated through unit tests, whereas aggregates of software modules are validated through integration tests. When a continuous integration build tool like Jenkins is used it is natural to define different build steps, each step returning feedback and generating test reports and trend charts for a specific level of testing. FitNesse is a lightweight testing framework that is meant to implement integration testing in a highly collaborative way, which makes it very suitable to be used within agile software projects. With Jenkins and Maven it is quite easy to trigger the execution of FitNesse integration tests automatically. When properly configured and bootstrapped, Jenkins can treat the FitNesse test results in a very similar way as it treats regular JUnit test results. Now lets suppose within a Maven project we have a FitNesse suite that contains the integration tests we want to be executed by a Jenkins job. With the Maven Failsafe Plugin and the help of some convenient FitNesse built-in JUnit utility classes this can be accomplished really easily. First of all we need to create a JUnit integration test class that will actually bootstrap the FitNesse tests. Lets says this class is named FitNesseIT. Within this class we need to instantiate a JUnitXMLTestListener and a JUnitHelper in such a way that Jenkins will automatically recognize the test results as regular JUnit test results: import fitnesse.junit.*; resultListener = new JUnitXMLTestListener("target/failsafe-reports"); jUnitHelper = new JUnitHelper(".", "target/fitnesse-reports", resultListener); The port property of the JUnitHelper does not need to be set when using the SLIM test system. However, if the FIT test system is used, this port must be set to an appropriate value as it specifies the port number of the FitServer that will be launched to execute the FIT tests. It is recommended to assign a random free available port, as it is considered a good practice to avoid using any fixed port on the executing Jenkins node: // if test system == FIT socket = new ServerSocket(0); jUnitHelper.setPort(socket.getLocalPort()); socket.close(); The debugMode property of the JUnitHelper should not be changed. It is set to true by default, which means that the SlimService or FitServer will efficiently run within the same Java process that is created by the Maven Failsafe Plugin to run the integration test. The JUnitHelper will be used to kick off the execution of the actual FitNesse tests: @Test public void assertSuitePasses() throws Exception { jUnitHelper.assertSuitePasses(suiteName); } The execution of the FitNesseIT test class itself can be triggered through the use of the Maven Failsafe Plugin. In this way the FitNesse suite will be executed automatically as part of the Maven lifecycle integration-test build phase. The FitNesseIT test class can also be executed from your IDE, which makes it really easy to actually debug the FitNesse tests by stepping through the fixture classes. Instead of instantiating a JUnitHelper ourself, we could have used the JUnit runner class FitNesseSuite and specified by annotation the actual FitNesse suite that needs to be executed as a JUnit test. However this runner class does not create the JUnit XML report files that need to be processed by Jenkins. As the JUnitXMLTestListener will already create report files for all individual FitNesse tests, there is no need to have a separate report file for the bootstrapping FitNesseIT test class itself. Therefore, the disableXmlReport configuration property of the Maven Failsafe Plugin need to be enabled. In this way the Jenkins job will only take the results of the individual FitNesse tests into account when generating its test report and trend chart. Furthermore, the system property variables TEST_SYSTEM and SLIM_PORT need to be configured appropriately: org.apache.maven.plugins maven-failsafe-plugin integration-test true slim 0 By setting the SLIM_PORT to 0, the SLIM executor will run on a random free available port, so no fixed port will be used on the executing Jenkins node. Obviously, when using FIT the TEST_SYSTEM variable must be set to fit instead of slim and the SLIM_PORT variable is not needed. Alternatively, the TEST_SYSTEM and SLIM_PORT variables can be defined with the Fitnesse define keyword: !define TEST_SYSTEM {slim} !define SLIM_PORT {0} As Jenkins automatically scans the failsafe-reports directories “**/target/failsafe-reports”, the FitNesse test results will be processed out of the box. No additional Jenkins plugins are required. The JUnitHelper also creates a nice HTML report that consist of a summary including some useful statistics as well as detailed test result pages for all executed tests. This report can be found in the “target/fitnesse-reports” directory and can be published by a post-build action with the HTML Publisher Plugin. In a continuous integration pipeline it makes sense to trigger the execution of the integration tests in an individual build step. This can be accomplished typically by activating the Maven Failsafe Plugin using a Maven profile. In this way the integration test results and unit test results are not mixed into the same reports and trend charts by Jenkins.

Lately I encountered a configuration tweak I was not aware of, the problem: I had a single Java installation on a Linux machine from which I had to start two JVM instances - each using a different set of JCE providers. A reminder: the JVM loads its security configuration, including the JCE providers list, from a master security properties file within the JRE folder (JRE_HOME/lib/security/java.security), the location of that file is fixed in the JVM and cannot be modified. Going over the documentation (not too much helpful, I must admit) and the code (more helpful, look for Security.java, for example here) reveled the secret. security.overridePropertiesFile It all starts within the default java.security file provided with the JVM, looking at it we will find the following (somewhere around the middle of the file) # # Determines whether this properties file can be appended to # or overridden on the command line via -Djava.security.properties # security.overridePropertiesFile=true If the overridePropertiesFile doesn’t equal to true we can stop here - the rest of this article is irrelevant (unless we have the option to change it – but I didn’t have that). Lucky to me by default it does equal to true. java.security.properties Next step, the interesting one, is to override or append configuration to the default java.security file per JVM execution. This is done by setting the 'java.security.properties' system property to point to a properties file as part of the JVM invocation; it is important to notice that referencing to the file can be done in one of two flavors: Overriding the entire file provided by the JVM - if the first character in the java.security.properties' value is the equals sign the default configuration file will be entirely ignored, only the values in the file we are pointing to will be affective Appending and overriding values of the default file - any other first character in the property's value (that is the first character in the alternate configuration file path) means that the alternate file will be loaded and appended to the default one. If the alternate file contains properties which are already in the default configuration file the alternate file will override those properties. Here are two examples # # Completely override the default java.security file content # (notice the *two* equal signs) # java -Djava.security.properties==/etc/sysconfig/jvm1.java.security # # Append or override parts of the default java.security file # (notice the *single* equal sign) # java -Djava.security.properties=/etc/sysconfig/jvm.java.security Be Carefull As an important configuration option as it is we must not forget its security implications. We should always make sure that no one can tamper the value of the property and that no one can tamper the alternate file content if he shouldn't be allowed to.

This code reads the port number defined in Server.XML of Tomcat, the code source is http://www.asjava.com/tomcat/how-to-get-tomcat-port-number-in-java/ public static Integer getTomcatPortFromConfigXml(File serverXml) { Integer port; try { DocumentBuilderFactory domFactory = DocumentBuilderFactory.newInstance(); domFactory.setNamespaceAware(true); // never forget this! DocumentBuilder builder = domFactory.newDocumentBuilder(); Document doc = builder.parse(serverXml); XPathFactory factory = XPathFactory.newInstance(); XPath xpath = factory.newXPath(); XPathExpression expr = xpath.compile("/Server/Service[@name='Catalina']/Connector[count(@scheme)=0]/@port[1]"); String result = (String) expr.evaluate(doc, XPathConstants.STRING); port = result != null && result.length() > 0 ? Integer.valueOf(result) : null; } catch (Exception e) { port = null; } return port; }

To get JDK 7 up, I downloaded the JDK from Oracle. They have a nice dmg file, which makes it easy to install. After reading their installation instructions and running /usr/libexec/java_home (which I didn't even know about), it still wasn't defaulting to JDK 7. Surgery required. So, I headed over to: /System/Library/Frameworks/JavaVM.framework/Versions This is where the system jvm's are stored. You'll notice a symbolic link for CurrentJDK. It probably points to: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents You're going to want to point that to the new JDK, which java_home tells us is located in: bone@zen:/usr/libexec$ /usr/libexec/java_home /Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home So, the magic commands you need are: bone@zen:/System/Library/Frameworks/JavaVM.framework/Versions$ sudo rm CurrentJDK bone@zen:/System/Library/Frameworks/JavaVM.framework/Versions$ sudo ln -s /Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/ CurrentJDK Then, you should be good: bone@zen:/System/Library/Frameworks/JavaVM.framework/Versions$ java -version java version "1.7.0_09" Java(TM) SE Runtime Environment (build 1.7.0_09-b05) Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)

This article will provide you with a quick reference guide on how to calculate the Java process size memory footprint for Java VM processes running on IBM AIX 5.3+ OS. This is a complementary post to my original article on this subject: how to monitor the Java native memory on AIX. I highly recommend this article to any individual involved in production support or development of Java applications deployed on AIX. Why is this knowledge important? From my perspective, basic knowledge on how the OS is managing the memory allocation of your JVM processes is very important. We often overlook this monitoring aspect and only focus on the Java heap itself. From my experience, most Java memory related problems are observed from the Java heap itself such as garbage collection problems, leaks etc. However, I’m confident that you will face situations in the future involving native memory problems or OS memory challenges. Proper knowledge of your OS and virtual memory management is crucial for proper root causes analysis, recommendations and solutions. AIX memory vs. pages As you may have seen from my earlier post, the AIX Virtual Memory Manager (VMM) is responsible to manage memory requests from the system and its applications. The actual physical memory is converted and partitioned in units called pages; allocated either in physical RAM or stored on disk until it is needed. Each page can have a size of 4 KB (small page), 64 KB (medium page) or 16 MB (large page). Typically for a 64-bit Java process you will see a mix of all of the above. What about the topas command? The typical reflex when supporting applications on AIX is to run the topas command, similar to Solaris top. Find below an example of output from AIX 5.3: As you can see, the topas command is not very helpful to get a clear view on the memory utilization since it is not providing the breakdown view that we need for our analysis. It is still useful to get a rough idea of the paging space utilization which can give you a quick idea of your top "paging space" consumer processes. Same can be achieved via the ps aux command. AIX OS command to the rescue: svmon The AIX svmon command is by far my preferred command to deep dive into the Java process memory utilization. This is a very powerful command, similar to Solaris pmap. It allows you to monitor the current memory “pages” allocation along with each segment e.g. Java Heap vs. native heap segments. Analyzing the svmon output will allow you to calculate the memory footprint for each page type (4 KB, 64 KB, and 16 MB). Now find below a real example which will allow you to understand how the calculation is done: # 64-bit JVM with -Xms2048m & -Xmx2048m (2 GB Java Heap) # Command: svmon –P As you can see, the total footprint of our Java process size was found at 2.2 GB which is aligned with current Java heap settings. You should be able to easily perform the same memory footprint analysis from your AIX environment. I hope this article has helped you to understand how to calculate the Java process size on AIX OS. Please feel free to post any comment or question.

A frequent issue I come across writing integration applications with Mule is deciding how to communicate back and forth between my front end application, typically a web or mobile application, and a flow hosted on Mule. I could use web services and do something like annotate a component with JAX-RS and expose this out over HTTP. This is potentially overkill, particularly if I only want to host a few methods, the methods are asynchronous or I don’t want to deal with the overhead of HTTP. It also could be a lot of extra effort if the only consumers of the API, at least initially, are internal facing applications. Another choice is to use “synchronous” JMS with temporary reply queues. While Mule makes this easy to do, particularly with MuleClient, I now have to deal with the overhead of spinning up a JMS infrastructure. I could also be limited to Java only clients, depending on which JMS broker I choose. The latter is particularly signifcant, as Java probably isn’t the technology of choice on the web or mobile layer. ØMQ for RPC ØMQ, or ZeroMQ, is a networking library designed from the ground up to ease integration between distributed applications. In addition to supporting a variety of messaging patterns, which are enumerated in the extremely well written guide, the library is written in platform agnostic C with wrappers for different languages like Java, Python and Ruby. These features make it a good candidate to solve the challenges I introduced above, particularly since a community contributed module for ØMQ was released recently. Let’s consider a simple service that accepts a request for a range of stock quotes and returns the results and see how we can host this service with Mule and expose it out with the ØMQ Module. Data Serialization with Protocol Buffers Data is transported back and forth over ØMQ as byte arrays. We, as such, need to decide on a way to serialize our stock quote request and responses “on the wire.” Before we do that, however, let’s take a look at the Java canonical data model we’re using on the client and server side. The following Gists show the important bits of the StockQuote and StockQuoteResponse classes. public class StockQuote implements Serializable { String symbol; Date date; Double open; Double high; Double low; Double close; Long volume; Double adjustedClose; public class StockQuoteRequest implements Serializable { String symbol; Date startDate; Date endDate; public interface StockDataService { public List getQuote(StockQuoteRequest request); } We could use Java serialization to get the objects into byte arrays. Ignoring the other deficiencies of default Java serialization, the main drawback is that it limits our clients to one’s running on a JVM. XML or JSON provide better alternatives, but for the purposes of this example we’ll assume we want a more compact representation of the data (this isn’t totally unrealistic, stock quote data can be extremely time sensitive and we probably want to minimize serialization and deserialization overhead.) Protocol Buffers provide a good middle ground and also boast a Mule Module to provide the necessary transformers we need to move back and forth from the byte array representations. Let’s define two .proto files to define the wire format and generate the intermediary stubs for serialization. package com.acmesoft.zeromq; option java_package = "com.acmesoft.stock.model.serialization.protobuf"; option optimize_for = SPEED;package com.acmesoft.zeromq; option java_package = "com.acmesoft.stock.model.serialization.protobuf"; option optimize_for = SPEED; option java_multiple_files = true; message StockQuoteResponseBuffer { repeated StockQuoteBuffer result = 1; } message StockQuoteBuffer { required string symbol = 1; required int64 date = 2; required double open = 3; required double high = 4; required double low = 5; required double close = 6; required int64 volume = 7; required double adjustedClose = 8; } option java_multiple_files = true; message StockQuoteRequestBuffer { required string symbol = 1; required int64 start = 2; required int64 end = 3; } You typically would use the “protoc” compiler to generate the Java stubs. This is tedious, however, so we’ll instead modify the pom.xml of our project to compile the protoc files during the compile goals: com.google.protobuf.tools maven-protoc-plugin /usr/local/bin/protoc compile testCompile Since we already have a domain model we’ll add some helper classes to simplify the serialization tasks on the client side. public byte[] toProtocolBufferAsBytes() { return StockQuoteRequestBuffer.newBuilder() .setSymbol(symbol) .setStart(startDate.getTime()) .setEnd(endDate.getTime()).build().toByteArray(); } public static StockQuoteRequest fromProtocolBuffer(StockQuoteRequestBuffer buffer) { StockQuoteRequest request = new StockQuoteRequest(); request.setSymbol(buffer.getSymbol()); request.setStartDate(new Date(buffer.getStart())); request.setEndDate(new Date(buffer.getEnd())); return request; } public static StockQuoteResponseBuffer toProtocolBuffer(List quotes) { StockQuoteResponseBuffer.Builder responseBuilder = StockQuoteResponseBuffer.newBuilder(); for (StockQuote quote : quotes) { responseBuilder.addResult(StockQuoteBuffer.newBuilder() .setAdjustedClose(quote.getAdjustedClose()) .setClose(quote.getClose()) .setDate(quote.getDate().getTime()) .setHigh(quote.getHigh()) .setLow(quote.getLow()) .setOpen(quote.getOpen()) .setSymbol(quote.getSymbol()) .setVolume(quote.getVolume()).build()); } return responseBuilder.build(); } public static List listOfStockQuotesFromBytes(byte[] bytes) { List buffer; try { buffer = StockQuoteResponseBuffer.parseFrom(bytes).getResultList(); } catch (InvalidProtocolBufferException e) { throw new SerializationException(e); } List quotes = new ArrayList(); for (StockQuoteBuffer stockQuoteBuffer : buffer) { StockQuote stockQuote = new StockQuote(); stockQuote.setClose(stockQuoteBuffer.getClose()); stockQuote.setDate(new Date(stockQuoteBuffer.getDate())); stockQuote.setHigh(stockQuoteBuffer.getHigh()); stockQuote.setOpen(stockQuoteBuffer.getOpen()); stockQuote.setSymbol(stockQuoteBuffer.getSymbol()); stockQuote.setVolume(stockQuoteBuffer.getVolume()); stockQuote.setAdjustedClose(stockQuoteBuffer.getAdjustedClose()); stockQuote.setLow(stockQuoteBuffer.getLow()); quotes.add(stockQuote); } return quotes; } Configuring StockDataService Now that we have a canonical data model and a wire format defined we’re ready to wire up a Mule flow to expose the service out. Note that for this to work you need to have jzmq installed locally on your system. The following dependency needs to be added to your pom.xml once its installed: org.zeromq zmq 2.2.0 /usr/local/lib/zmq.jar system Where systemPath is the location of the zmq.jar on your filesystem. Once that’s out of the way we can configure the flow, as illustrated below: The ZeroMQ inbound-endpoint will be bound to TCP port 9090 with a request-response exchange pattern. The deserialize MP in the protobuf module will deserialize the byte array to the generated StockQuoteRequestBuffer class. From there we’ll use MEL to invoke the helper method on StockQuoteRequest to transform the intermediary class to the domain model. The List of StockQuotes returned from StockDataService will be transformed by the MEL expression using the “toProtocolBuffer” helper method on the domain model. The Protocol Buffer Module is then smart enough to implicitly transform the intermediary object to a byte array for the response. Consuming the Service from the Client Side Now that the server is ready we can turn our attention to the client side code to invoke the remote service. Let’s take a look at how this works: StockQuoteRequest stockQuoteRequest = new StockQuoteRequest(); stockQuoteRequest.setSymbol("FB"); stockQuoteRequest.setStartDate(new Date( new Date().getTime() - (86400000 * 7))); stockQuoteRequest.setEndDate(new Date()); ZMQ.Socket zmqSocket = zmqContext.socket(ZMQ.REQ); zmqSocket.setReceiveTimeOut(RECEIVE_TIMEOUT); zmqSocket.connect("tcp://localhost:9090"); zmqSocket.send(stockQuoteRequest.toProtocolBufferAsBytes(), 0); List quotes = StockQuote.listOfStockQuotesFromBytes(zmqSocket.recv(0)); We start off by defining the StockQuoteRequest object to give us all the quotes for Facebook stock from the last week. We can then open up a ZMQ socket, set the timeout, connect to the ZMQ socket on the remote Mule instance and send the byte representation of the StockQuoteRequest to it. zmqSocket.recv is then used to receive the bytes back from Mule. From here we can use the listOfStockQuotesFromBytes helper method we wrote above to convert the Protocol Buffer representation to a List of StockQuotes. Despite the fair bit of plumbing we did above, this is a pretty concise bit of client side code to invoke the remote service. Conclusion This blog post only touched on the features of ØMQ and the ØMQ Mule Module. In addition to request-reply, other exchange-patterns are supported, like one-way, push and pull. This effectively gives you the benefits of a reliable, asynchronous messaging layer without a centralized infrastructure. I hope to cover this in a later post. Protocol buffers also seem like a natural fit as a wire format for ØMQ. protobuffers echo ØMQ’s principals of being lightweight, fast and platform agnostic. These are also, not coincidently, principals Mule shares as an integration framework. The project for this example is available on GitHub.

This article is intended for Java beginners currently facing java.lang.ClassNotFoundException challenges. It will provide you with an overview of this common Java exception, a sample Java program to support your learning process and resolution strategies. If you are interested on more advanced class loader related problems, I recommended that you review my article series on java.lang.NoClassDefFoundError since these Java exceptions are closely related. java.lang.ClassNotFoundException: Overview As per the Oracle documentation, ClassNotFoundException is thrown following the failure of a class loading call, using its string name, as per below: The Class.forName method The ClassLoader.findSystemClass method The ClassLoader.loadClass method In other words, it means that one particular Java class was not found or could not be loaded at “runtime” from your application current context class loader. This problem can be particularly confusing for Java beginners. This is why I always recommend to Java developers to learn and refine their knowledge on Java class loaders. Unless you are involved in dynamic class loading and using the Java Reflection API, chances are that the ClassNotFoundException error you are getting is not from your application code but from a referencing API. Another common problem pattern is a wrong packaging of your application code. We will get back to the resolution strategies at the end of the article. java.lang.ClassNotFoundException: Sample Java program Now find below a very simple Java program which simulates the 2 most common ClassNotFoundException scenarios via Class.forName() & ClassLoader.loadClass(). Please simply copy/paste and run the program with the IDE of your choice (Eclipse IDE was used for this example). The Java program allows you to choose between problem scenario #1 or problem scenario #2 as per below. Simply change to 1 or 2 depending of the scenario you want to study. # Class.forName() private static final int PROBLEM_SCENARIO = 1; # ClassLoader.loadClass() private static final int PROBLEM_SCENARIO = 2; # ClassNotFoundExceptionSimulator package org.ph.javaee.training5; /** * ClassNotFoundExceptionSimulator * @author Pierre-Hugues Charbonneau * */ public class ClassNotFoundExceptionSimulator { private static final String CLASS_TO_LOAD = "org.ph.javaee.training5.ClassA"; private static final int PROBLEM_SCENARIO = 1; /** * @param args */ public static void main(String[] args) { System.out.println("java.lang.ClassNotFoundException Simulator - Training 5"); System.out.println("Author: Pierre-Hugues Charbonneau"); System.out.println("http://javaeesupportpatterns.blogspot.com"); switch(PROBLEM_SCENARIO) { // Scenario #1 - Class.forName() case 1: System.out.println("\n** Problem scenario #1: Class.forName() **\n"); try { Class newClass = Class.forName(CLASS_TO_LOAD); System.out.println("Class "+newClass+" found successfully!"); } catch (ClassNotFoundException ex) { ex.printStackTrace(); System.out.println("Class "+CLASS_TO_LOAD+" not found!"); } catch (Throwable any) { System.out.println("Unexpected error! "+any); } break; // Scenario #2 - ClassLoader.loadClass() case 2: System.out.println("\n** Problem scenario #2: ClassLoader.loadClass() **\n"); try { ClassLoader classLoader = Thread.currentThread().getContextClassLoader(); Class callerClass = classLoader.loadClass(CLASS_TO_LOAD); Object newClassAInstance = callerClass.newInstance(); System.out.println("SUCCESS!: "+newClassAInstance); } catch (ClassNotFoundException ex) { ex.printStackTrace(); System.out.println("Class "+CLASS_TO_LOAD+" not found!"); } catch (Throwable any) { System.out.println("Unexpected error! "+any); } break; } System.out.println("\nSimulator done!"); } } # ClassA package org.ph.javaee.training5; /** * ClassA * @author Pierre-Hugues Charbonneau * */ public class ClassA { private final static Class CLAZZ = ClassA.class; static { System.out.println("Class loading of "+CLAZZ+" from ClassLoader '"+CLAZZ.getClassLoader()+"' in progress..."); } public ClassA() { System.out.println("Creating a new instance of "+ClassA.class.getName()+"..."); doSomething(); } private void doSomething() { // Nothing to do... } } f you run the program as is, you will see the output as per below for each scenario: #Scenario 1 output (baseline) java.lang.ClassNotFoundException Simulator - Training 5 Author: Pierre-Hugues Charbonneau http://javaeesupportpatterns.blogspot.com ** Problem scenario #1: Class.forName() ** Class loading of class org.ph.javaee.training5.ClassA from ClassLoader 'sun.misc.Launcher$AppClassLoader@bfbdb0' in progress... Class class org.ph.javaee.training5.ClassA found successfully! Simulator done! #Scenario 2 output (baseline) java.lang.ClassNotFoundException Simulator - Training 5 Author: Pierre-Hugues Charbonneau http://javaeesupportpatterns.blogspot.com ** Problem scenario #2: ClassLoader.loadClass() ** Class loading of class org.ph.javaee.training5.ClassA from ClassLoader 'sun.misc.Launcher$AppClassLoader@2a340e' in progress... Creating a new instance of org.ph.javaee.training5.ClassA... SUCCESS!: org.ph.javaee.training5.ClassA@6eb38a Simulator done! For the “baseline” run, the Java program is able to load ClassA successfully. Now let’s voluntary change the full name of ClassA and re-run the program for each scenario. The following output can be observed: #ClassA changed to ClassB private static final String CLASS_TO_LOAD = "org.ph.javaee.training5.ClassB"; #Scenario 1 output (problem replication) java.lang.ClassNotFoundException Simulator - Training 5 Author: Pierre-Hugues Charbonneau http://javaeesupportpatterns.blogspot.com ** Problem scenario #1: Class.forName() ** java.lang.ClassNotFoundException: org.ph.javaee.training5.ClassB at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:186) at org.ph.javaee.training5.ClassNotFoundExceptionSimulator.main(ClassNotFoundExceptionSimulator.java:29) Class org.ph.javaee.training5.ClassB not found! Simulator done! #Scenario 2 output (problem replication) java.lang.ClassNotFoundException Simulator - Training 5 Author: Pierre-Hugues Charbonneau http://javaeesupportpatterns.blogspot.com ** Problem scenario #2: ClassLoader.loadClass() ** java.lang.ClassNotFoundException: org.ph.javaee.training5.ClassB at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at org.ph.javaee.training5.ClassNotFoundExceptionSimulator.main(ClassNotFoundExceptionSimulator.java:51) Class org.ph.javaee.training5.ClassB not found! Simulator done! What happened? Well since we changed the full class name to org.ph.javaee.training5.ClassB, such class was not found at runtime (does not exist), causing both Class.forName() and ClassLoader.loadClass() calls to fail. You can also replicate this problem by packaging each class of this program to its own JAR file and then omit the jar file containing ClassA.class from the main class path Please try this and see the results for yourself…(hint: NoClassDefFoundError) Now let’s jump to the resolution strategies. java.lang.ClassNotFoundException: Resolution strategies Now that you understand this problem, it is now time to resolve it. Resolution can be fairly simple or very complex depending of the root cause. Don’t jump on complex root causes too quickly, rule out the simplest causes first. First review the java.lang.ClassNotFoundException stack trace as per the above and determine which Java class was not loaded properly at runtime e.g. application code, third party API, Java EE container itself etc. Identify the caller e.g. Java class you see from the stack trace just before the Class.forName() or ClassLoader.loadClass() calls. This will help you understand if your application code is at fault vs. a third party API. Determine if your application code is not packaged properly e.g. missing JAR file(s) from your classpath If the missing Java class is not from your application code, then identify if it belongs to a third party API you are using as per of your Java application. Once you identify it, you will need to add the missing JAR file(s) to your runtime classpath or web application WAR/EAR file. If still struggling after multiple resolution attempts, this could means a more complex class loader hierarchy problem. In this case, please review my NoClassDefFoundError article series for more examples and resolution strategies I hope this article has helped you to understand and revisit this common Java exception. Please feel free to post any comment or question if you are still struggling with your java.lang.ClassNotFoundException problem.

This article will provide you with a quick reference guide on how to calculate the Java process size memory footprint for Java VM processes running on IBM AIX 5.3+ OS. This is a complementary post to my original article on this subject: how to monitor the Java native memory on AIX. I highly recommend this read to any individual involved in production support or development of Java applications deployed on AIX. Why is this knowledge important? From my perspective, basic knowledge on how the OS is managing the memory allocation of your JVM processes is very important. We often overlook this monitoring aspect and only focus on the Java heap itself. From my experience, most Java memory related problems are observed from the Java heap itself such as garbage collection problems, leaks etc. However, I’m confident that you will face situations in the future involving native memory problems or OS memory challenges. Proper knowledge of your OS and virtual memory management is crucial for proper root causes analysis, recommendations and solutions. AIX memory vs. pages As you may have seen from my earlier post, the AIX Virtual Memory Manager (VMM) is responsible to manage memory requests from the system and its applications. The actual physical memory is converted and partitioned in units called pages; allocated either in physical RAM or stored on disk until it is needed. Each page can have a size of 4 KB (small page), 64 KB (medium page) or 16 MB (large page). Typically for a 64-bit Java process you will see a mix of all of the above. What about the topas command? The typical reflex when supporting applications on AIX is to run the topas command, similar to Solaris top. Find below an example of output from AIX 5.3: As you can see, the topas command is not very helpful to get a clear view on the memory utilization since it is not providing the breakdown view that we need for our analysis. It is still useful to get a rough idea of the paging space utilization which can give you a quick idea of your top "paging space" consumer processes. Same can be achieved via the ps aux command. AIX OS command to the rescue: svmon The AIX svmon command is by far my preferred command to deep dive into the Java process memory utilization. This is a very powerful command, similar to Solaris pmap. It allows you to monitor the current memory “pages” allocation along with each segment e.g. Java Heap vs. native heap segments. Analyzing the svmon output will allow you to calculate the memory footprint for each page type (4 KB, 64 KB, and 16 MB). Now find below a real example which will allow you to understand how the calculation is done: # 64-bit JVM with -Xms2048m & -Xmx2048m (2 GB Java Heap) # Command: svmon –P As you can see, the total footprint of our Java process size was found at 2.2 GB which is aligned with current Java heap settings. You should be able to easily perform the same memory footprint analysis from your AIX environment I hope this article has helped you to understand how to calculate the Java process size on AIX OS. Please feel free to post any comment or question.

New Relic (Remember, there's a free Lite version) lets you identify the slow transactions of an application out of the box. This means you can simply download an agent, start your application running with an agent and soon see slow transactions being reported. While identifying slow transactions is key to performance tuning, there are times when I need more information about a specific transaction that’s not necessarily the slowest. Sometimes I just want to know how many times a specific method is being called. At others, I want timing information on a specific method not automatically instrumented. Or maybe I just want to exclude all calls to a recursive method from a transaction trace. New Relic provides these features through custom instrumentation. Custom Instrumentation Using the New Relic Java Agent Since I work on the New Relic Java agent, this post will focus on the three mechanisms it provides to get metric information about certain transactions. If you’re interested in setting up custom instrumentation with any of our other agents, see the links at the end of this post. One way to set up custom instrumentation with the Java Agent is to use the New Relic API. Our API allows you to create metrics, increment counters, notice errors, ignore transactions, and more within your code. To get started: 1. Simply add the newrelic-api.jar to your class path. 2. Call the appropriate static method from the NewRelic API in your code. 3. Recompile and restart your application with the Java agent. If you prefer annotations, this second option might be for you: 1. Set enable_custom_tracing to true in your newrelic.yml file. Be sure to add the flag if it does not already exist. 2. Add the newrelic-api.jar to your class path. 3. Add the Trace annotation to the method you want to monitor. 4. Recompile and restart your application with the Java agent. If modifying the source code is not possible or desired, you should use New Relic’s third mechanism for custom instrumentation. New Relic allows you to create an Extension XML file specifying the class and method combinations that you want to monitor. New Relic will then read the file on startup and instrument the appropriate classes. Example Lets take a closer look at these three options for custom instrumentation through an example. Suppose you have the following class: package com.example; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; public class CustomSample { public void sampleMethod() throws Exception { Runnable myRunnable = new Runnable() { @Override public void run() { firstPart(); secondPart(); } }; ScheduledExecutorService scheduledExecutor = Executors .newScheduledThreadPool(1); scheduledExecutor.scheduleWithFixedDelay(myRunnable, 0, 10000, TimeUnit.MILLISECONDS); } private void firstPart() { System.out.println("In the first part."); } private void secondPart() { System.out.println("In the second part."); } } Option 1 – The New Relic API To use the New Relic API, first put the newrelic-api.jar on your class path. Then the class itself must be modified. Suppose you want to count the number of times the method run is called. This can be accomplished by adding a call to the incrementCounter method in the New Relic API. The one input parameter to this method is the name of the metric. Below I have chosen to call the metric ‘CustomSample.run’. @Override public void run() { firstPart(); secondPart(); NewRelic.incrementCounter(“CustomSample.run”); } Meanwhile, you can use the method recordMetric from the New Relic API to record the amount of time the method firstPart is taking. This method takes in the name of the metric and the value of the metric as parameters. While I have chosen to provide the time below, the value can be anything that fits into a float. For example, I could have set the metric value to a float value resulting from some business logic. If you do choose to measure method times yourself, be sure to use the System.nanoTime() method instead of the System.currentTimeMillis() method. The millisecond time is based on the system clock which can get reset, resulting in start times that are later than stop times. private void firstPart() { long start = System.nanoTime(); System.out.println("In the first part."); long timeDifference = System.nanoTime() -start; NewRelic.recordMetric(“CustomSample.firstPart”, timeDifference); } Once you have made the code changes, recompile and restart your application with the Java agent. Option 2 – Annotations In order to use New Relics’s annotations, remember to set the property enable_custom_tracing to true in your newrelic.yml configuration file and put the newrelic-api.jar on your class path. If you want metrics on the run method, then the Trace annotation needs to be added. Additionally, since this method will likely be the start of a transaction, you need to include ‘dispatcher=true’ as shown below. When this property is set to true, a new transaction is started when the method is reached if a transaction is not already in progress. If the method is encountered after a transaction has been started, then that transaction will continue and a new one will not be created. The dispatcher property is defaulted to false. @Override @Trace(dispatcher=true) public void run() { firstPart(); secondPart(); } To monitor the method firstPart, you also need to add the Trace annotation to this method. Since the method firstPart will be called within the run method, meaning after the transaction has already been started, the dispatcher property can be defaulted to false. @Trace private void firstPart() { System.out.println("In the first part."); } Once you have made the code changes, recompile and restart your application with the Java agent. Option 3 – XML Extension Files In order to use XML extension files, you first need to create the XML file. XML extension files must follow the schema definition newrelic-extension.xsd which can be found within the Java agent zip file starting with version 2.10.0. You will also find an example file called newrelic-extension-example.xml which instruments some of the methods found in the JDK. Here are a few pointers when creating the file: 1. Be sure to always include a name and version. If you have two XML extension files with the same name, only the one with the higher version will be implemented. 2. The metricPrefix is an optional parameter on the instrumentation node which is used as part of the metric name. If not set, it is defaulted to ‘CUSTOM”. 3. Methods to be monitor are put into point cuts. A separate point cut must be used for each class. However, multiple methods from that class can be listed within the same point cut. The XML extension file for instrumenting the run method is shown below. Since run is actually in an inner class, the class name is com.example.CustomSample$1. Additionally, since run should be the start of a transaction, the attribute transactionStartPoint is set to true on the point cut node. com.example.CustomSample$1 run To also monitor the firstPart and secondPart methods from the CustomSample class, a point cut needs to be added to the XML above. This point cut is shown below. Note that the attribute transactionStartPoint is left to its default of false since both these methods should be called from within the run method and thus will be called within a transaction. Also note that if you wanted to exclude these methods from the stack trace, you could set the attribute excludeFromTransactionTrace to true on the point cut node. com.example.CustomSample firstPart secondPart Once the XML extension file is complete, be sure to validate the file. This can be accomplished with the following command. java -jar newrelic.jar instrument -file /path/to/extension.xml This validation application will check the XML syntax and validate that all classes and methods found in the XML file are present on the classpath. It will then either print out ‘PASS’ or ‘FAIL’ to the console. If the XML file fails validation, the reason it failed will also be printed out. Once the XML file is valid, set the property ‘extensions.dir’ in your newrelic.yml file to the directory where your XML file is located. Then restart your application with the agent. Conclusion I find these options for custom instrumentation to be very useful. However, a word of caution: custom instrumentation is designed to be used for only a few methods. You should not try to instrument every method of every class. Doing so will slow down the performance of your application. That said, I encourage you to try out custom instrumentation for yourself. More examples and documentation on the Java agent can be found here.

Overflow and underflow of values of various data types is a very common occurence in Java programs. This is usually because the beginners dont' pay proper attention to the default values of various data types. If we are creating a byte type variable and assigning it a value, we should be aware that the value will be treated as an int and hence a potential overflow condition. In Java the overflow and underflow are more serious because there is no warning or exception raised by the JVM when such a condition occurs. Some developers argue that the program should either crash or raise exception in such case but the decision for adding such behavior is in the hands of creators of programming language. By looking at a problem in your program, you can't straightway tell that an overflow or underflow condition has occured. It is only after debugging that we come to know of the real cause. Overflow in int As int data type is 32 bit in Java, any value that surpasses 32 bits gets rolled over. In numerical terms, it means that after incrementing 1 on Integer.MAX_VALUE (2147483647), the returned value will be -2147483648. In fact you don't need to remember these values and the constants Integer.MIN_VALUE and Integer.MAX_VALUE can be used. Underflow of int Underflow is the opposite of overflow. While we reach the upper limit in case of overflow, we reach the lower limit in case of underflow. Thus after decrementing 1 from Integer.MIN_VALUE, we reach Integer.MAX_VALUE. Here we have rolled over from the lowest value of int to the maximum value. For non-integer based data types, the overflow and underflow result in INFINITY and ZERO values. You may try the following lines to verify this: float f = 3.4028235E38f * 20f; System.out.println(f); Note: As with int data type, we have wrappers for all primitive data types. So we can easily see the upper and lower limit of each data type by looking at the MAX_VALUE and MIN_VALUE constants in these wrapper classes. Read more: http://extreme-java.blogspot.com/2012/11/overflow-and-underflow-of-data-types-in.html#ixzz2BvqFu7fk

Spock is a BBD testing framework that allows for easy BDD tests to be written. The framework is an extension upon JUnit which allows for easy IDE integration and using existing JUnit functionality. Spock tests are written in Groovy and can be used for writing a wide range of tests from small unit tests to full application integration tests. Without going into too much detail on how to write Spock based tests (see below for a few excellent links), lets go through how we can use the framework to build integration tests for testing a RESTful API. Our first RESTful API Test package com.wolfware.integration import groovyx.net.http.RESTClient import spock.lang.* import spock.lang.Specification import com.movideo.spock.extension.APIVersion import com.movideo.spock.extension.EnvironmentEndPoint @APIVersion(minimimApiVersion="1.0.0.0") class GetAuthenticationToken extends Specification { @EnvironmentEndPoint protected def environmentHost def "Get authentication token XML from API for valid account"() { given: "a valid account" def authenticationTokenRequestParams = ['key':"AAABBBCCC123", 'user':"[email protected]"] and: "a client to get the authentication token XML" def client = new RESTClient(environmentHost) when: "we attempt to retrieve authentication token XML" def resp = client.get(path : "/authenticate", query : authenticationTokenRequestParams) then: "we should get a valid authentication token XML response" assert resp.data.token.isEmpty() == false // lots more asserts } } As you can see, apart from the @APIVersion and @EnvironmentEndPoint annotations (these are Spock extensions as explained later), the spec is a fairly simple Spock test. This specification has a feature that, as the name suggests, gets a authentication token in XML format and validates it. Lets look at each step: Given The url parameters required to get a authentication token from the RESTful service When using the Groovy RestClient to call the RESTful service for the authentication token details Then We can assert all the details of the response. The thing I really like about Spock is the readability of the tests. From the name being a descriptive sentence rather than some short hand with _ throughout to make a valid method name to being able to easily see where setup of the test is done and then the expectations and assertions. Trying to test any environment RESTful service I've found that when trying to write integration tests, there has either been: Hard coded environment details and the code branched for each environment making it near impossible to keep code in sync as merge hell becomes the norm. Config files that define the environment are used to define environment details, again checked into each branch for each environment. Trying to follow the principles of continuous delivery, it would be great to be able to use the same code base to test against any environment. This is where Spock Extensions come into play to help us out. Spock Extensions In short Spock allows us to extend it to perform other functionality during the test life-cycle (a great post on extensions can be read on this excellent blog post). I've developed two extensions which help to make the idea of running the same test suite across different environments easier. The @EnvironmentEndPoint Extension The aim of this Spock extension is to have a placeholder variable in code that at run-time, can be defined with the environment host of the RESTful services that we want to test. package com.movideo.runtime.extension.custom import org.apache.commons.logging.Log import org.apache.commons.logging.LogFactory import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension import org.spockframework.runtime.extension.AbstractMethodInterceptor import org.spockframework.runtime.extension.IMethodInvocation import org.spockframework.runtime.model.FieldInfo import org.spockframework.runtime.model.SpecInfo /** * Spock Environment Annotation Extension */ class EnvironmentEndPointExtension extends AbstractAnnotationDrivenExtension { private static final Log LOG = LogFactory.getLog(getClass()); private static def config = new ConfigSlurper().parse(new File('src/test/resources/SpockConfig.groovy').toURL()) /** * env environment variable * * Defaults to {@code LOCAL_END_POINT} */ private static final String envString = System.getProperties().getProperty("env", config.envHost); static { LOG.info("Environment End Point [" + envString + "]") } /** * {@inheritDoc} */ @Override void visitFieldAnnotation(EnvironmentEndPoint annotation, FieldInfo field) { def interceptor = new EnvironmentInterceptor(field, envString) interceptor.install(field.parent.getTopSpec()) } } /** * * Environment Intercepter * */ class EnvironmentInterceptor extends AbstractMethodInterceptor { private final FieldInfo field private final String envString EnvironmentInterceptor(FieldInfo field, String envString) { this.field = field this.envString = envString } private void injectEnvironmentHost(target) { field.writeValue(target, envString) } @Override void interceptSetupMethod(IMethodInvocation invocation) { injectEnvironmentHost(invocation.target) invocation.proceed() } @Override void install(SpecInfo spec) { spec.setupMethod.addInterceptor this } } The EnvironmentEndPointExtension class defines the following: config: is a ConfigSlurper that parses a config file 'SpockConfig.groovy' that is used to define the default environment host (envHost) envString: gets the value of 'env' from all System Properties (these include run-time properties) and defaults to config.envHost With the environment host able to be accessed by Spock, now we need to inject this into the placeholder variable for Spock tests to access. An interceptor is created which is used to inject(field.writeValue method) the value of the environment host into the placeholder variable. This placeholder is the one that the @EnvironmentEndPoint is annotating. When the test is run, the interceptor sets the placeholder variable and the test can then use this value as the host for the RestClient object. When running the Spock tests either the default value from the config file will be used or the JVM argument -Denv=? can be used. This makes running the same test code base against any environment so much easier. A note on Gradle builds. By default, Gradle will not pass through JVM arguments through to forked processes such as running tests. The code snippet below shows how to achieve this: /* * Required to pass all system properties to Test tasks. * Not default for Gradle to pass system properties through to forked processes. */ tasks.withType(Test) { def config = new ConfigSlurper().parse(new File('src/test/resources/SpockConfig.groovy').toURL()) systemProperty 'env', System.getProperty('env', config.envHost) } This allows all tasks that are a type of 'Test' to have some custom code run. In this case, we are defining the 'SpockConfig.groovy' config file and then setting 'systemPropery' within Gradle Test tasks to 'env' and either getting the value from the passed in JVM argument or from the config file. With this code in the build.gradle, we're able to run all tests via a Gradle test build, which will produce lovely test reports (in Gradle HTML and JUnit XML). The @APIVersion Extension Another integration testing problem I've found is that if we try and develop our tests first (or at least during the process of developing a feature or bug fix) that running the same tests against an environment that doesn't yet have the new code base (but we are using the same test code base everywhere), we'll have failing tests that aren't really failures as the new code isn't there yet. To help solve this problem, I've developed the @APIVersion extension to help with this issue. As newly developed code should be deployed with a new version, we can use this version to compare to a minimum version that a test can be run against. package com.movideo.runtime.extension.custom import groovyx.net.http.RESTClient import java.lang.annotation.Annotation import java.util.regex.Pattern import org.apache.commons.logging.Log import org.apache.commons.logging.LogFactory import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension import org.spockframework.runtime.model.FeatureInfo import org.spockframework.runtime.model.SpecInfo /** * API Version Extension * */ class APIVersionExtension extends AbstractAnnotationDrivenExtension { /** * Logger */ private static final Log LOG = LogFactory.getLog(getClass()); /** * */ private static def config = new ConfigSlurper().parse(new File('src/test/resources/SpockConfig.groovy').toURL()) /** * env environment variable * * Defaults to {@code LOCAL_END_POINT} */ private static final String envString = System.getProperties().getProperty("env", config.envHost); /** * Version REGX pattern */ private static final def VERSION_PATTERN = Pattern.compile(".", Pattern.LITERAL); /** * Max version length */ private static final def MAX_VERSION_LENGTH = 4; /** * Current API Version */ private static final def CURRENT_API_VERSION = getDeployedAPIVersion(); /** * {@inheritDoc} */ @Override void visitFeatureAnnotation(APIVersion annotation, FeatureInfo feature) { if(!isApiVersionGreaterThanMinApiVersion(annotation, feature.name)) { feature.setSkipped(true) } } /** * {@inheritDoc} */ @Override public void visitSpecAnnotation(APIVersion annotation, SpecInfo spec) { if(!isApiVersionGreaterThanMinApiVersion(annotation, spec.name)) { spec.setSkipped(true) } } /** * Get the current deployed API version * * Performs a HTTP request to the current deployed API version. Parses the returned data and get the {@code version} node data. * @return current deployed API version */ private static String getDeployedAPIVersion() { def apiVersion = null try { def client = new RESTClient(envString) def resp = client.get(path : config.versionServiceUri) apiVersion = resp.data.version LOG.info("Current deployed API version [" + apiVersion + "]"); } catch (ex) { APIVersionError apiVersionError = new APIVersionError("Error occurred attempting to get current deployed API version from %s", envString + config.versionServiceUri); apiVersionError.setStackTrace(ex.stackTrace); throw apiVersionError; } return apiVersion } * @param annotation * @param infoName * @return */ private boolean isApiVersionGreaterThanMinApiVersion(APIVersion annotation, String infoName) { def isApiVersionGreaterThanMinApiVersion = true def minApiVersionRequired = annotation.minimimApiVersion(); // normalise both version id's def apiVersionNormalised = normaliseVersion(CURRENT_API_VERSION); def minApiVersionRequiredNormalised = normaliseVersion(minApiVersionRequired); // compare version id's int cmp = apiVersionNormalised.compareTo(minApiVersionRequiredNormalised); // if the comparison is less than 0, min API version is greater than the deployed API version if(cmp < 0) { LOG.info("min api version [" + minApiVersionRequired + "] greater than api version [" + CURRENT_API_VERSION + "], skipping [" + infoName + "]") isApiVersionGreaterThanMinApiVersion = false } return isApiVersionGreaterThanMinApiVersion } * @param version * @return */ private String normaliseVersion(String version) { String[] split = VERSION_PATTERN.split(version); StringBuilder sb = new StringBuilder(); for (String s : split) { sb.append(String.format("%" + MAX_VERSION_LENGTH + 's', s)); } return sb.toString(); } } The @APIVersion extension defines the same environment config as the @EnvironmentEndPoint extension does so that the environment can be injected and used purely for accessing the API version endpoint without the need for @EnvironmentEndPoint. The RESTful API version endpoint is required to be setup and publicly available. The @APIVersion extension will call this service to get details about the version of RESTful API. The version response data should be as follows: Media API 1.51.1 The @APIVersion extension will look for the version data to define what the current deployed version of the RESTful API is. Once the version of the RESTful API is known, the extension then checks the minimum API version required. Example @APIVersion(minimimApiVersion="1.0.0.0") The extension then uses this value to compare against the response data version and if the required version is greater than that of the deployed RESTful API services, then the test is skipped. This extension annotation can be placed on Specification's or Feature's allowing whole Specs to have a minimum version and / or Features to have their own minimum version. This extension has made writing integration tests with Spock even more portable and allows for a 'build once' set of tests that can be run against any environment, with some small changes to allow getting the API version. The SpockConfig.groovy file Here is an example of the SpockConfig.groovy config file used to configure defaults for both @EnvironmentEndPoint and @APIVersion extensions. versionServiceUri="/public/serviceInformation" envHost="http://api.preview.movideo.com" The 'versionServiceUri' is required for @APIVersion extension as the URI for the RESTful API version The 'envHost' is required for both @APIVersion and @EnvironmentEndPoint extensions as the host of the RESTful API Go and start testing Hopefully these Spock extensions might help your Spock integration tests. The framework is really easy and fun to use to build essential tests for the whole test stack. Checkout my GitHub projects for the code for both extensions. Hope this post has been helpful and hopefully I'll post something sooner for my next post. References and really helpful links Spock Homepage Annotation Driven Extensions With Spock

Integration Testing is an often overlooked area in enterprise development. This is primarily due to the associated complexities in setting up the necessary infrastructure for an integration test. For applications backed by databases, it’s fairly complicated and time-consuming to setup databases for integration tests, and also to clean those up once test is complete (ex. data files, schemas etc.), to ensure repeatability of tests. While there have been many tools (ex. DBUnit) and mechanisms (ex. rollback after test) to assist in this, the inherent complexity and issues have been there always. But if you are working with MongoDB, there’s a cool and easy way to do your unit tests, with almost the simplicity of writing a unit test with mocks. With ‘EmbedMongo’, we can easily setup an embedded MongoDB instance for testing, with in-built clean up support once tests are complete. In this article, we will walkthrough an example where EmbedMongo is used with JUnit for integration testing a Repository Implementation. Here’s the technology stack that we will be using. MongoDB 2.2.0 EmbedMongo 1.26 Spring Data – Mongo 1.0.3 Spring Framework 3.1 The Maven POM for the above setup looks like this. 4.0.0 com.yohanliyanage.blog.mongoit mongo-it 1.0 org.springframework.data spring-data-mongodb 1.0.3.RELEASE compile junit junit 4.10 test org.springframework spring-context 3.1.3.RELEASE compile de.flapdoodle.embed de.flapdoodle.embed.mongo 1.26 test Or if you prefer Gradle (by the way, Gradle is an awesome build tool which you should check out if you haven’t done so already). apply plugin: 'java' apply plugin: 'eclipse' sourceCompatibility = 1.6 group = "com.yohanliyanage.blog.mongoit" version = '1.0' ext.springVersion = '3.1.3.RELEASE' ext.junitVersion = '4.10' ext.springMongoVersion = '1.0.3.RELEASE' ext.embedMongoVersion = '1.26' repositories { mavenCentral() maven { url 'http://repo.springsource.org/release' } } dependencies { compile "org.springframework:spring-context:${springVersion}" compile "org.springframework.data:spring-data-mongodb:${springMongoVersion}" testCompile "junit:junit:${junitVersion}" testCompile "de.flapdoodle.embed:de.flapdoodle.embed.mongo:${embedMongoVersion}" } To begin with, here’s the document that we will be storing in Mongo. package com.yohanliyanage.blog.mongoit.model; import org.springframework.data.mongodb.core.index.Indexed; import org.springframework.data.mongodb.core.mapping.Document; /** * A Sample Document. * * @author Yohan Liyanage * */ @Document public class Sample { @Indexed private String key; private String value; public Sample(String key, String value) { super(); this.key = key; this.value = value; } public String getKey() { return key; } public void setKey(String key) { this.key = key; } public String getValue() { return value; } public void setValue(String value) { this.value = value; } } To assist with storing and managing this document, let’s write up a simple Repository implementation. The Repository Interface is as follows. package com.yohanliyanage.blog.mongoit.repository; import java.util.List; import com.yohanliyanage.blog.mongoit.model.Sample; /** * Sample Repository API. * * @author Yohan Liyanage * */ public interface SampleRepository { /** * Persists the given Sample. * @param sample */ void save(Sample sample); /** * Returns the list of samples with given key. * @param sample * @return */ List findByKey(String key); } And the implementation… package com.yohanliyanage.blog.mongoit.repository; import java.util.List; import static org.springframework.data.mongodb.core.query.Query.query; import static org.springframework.data.mongodb.core.query.Criteria.*; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.data.mongodb.core.MongoOperations; import org.springframework.stereotype.Repository; import com.yohanliyanage.blog.mongoit.model.Sample; /** * Sample Repository MongoDB Implementation. * * @author Yohan Liyanage * */ @Repository public class SampleRepositoryMongoImpl implements SampleRepository { @Autowired private MongoOperations mongoOps; /** * {@inheritDoc} */ public void save(Sample sample) { mongoOps.save(sample); } /** * {@inheritDoc} */ public List findByKey(String key) { return mongoOps.find(query(where("key").is(key)), Sample.class); } /** * Sets the MongoOps implementation. * * @param mongoOps the mongoOps to set */ public void setMongoOps(MongoOperations mongoOps) { this.mongoOps = mongoOps; } } To wire this up, we need a Spring Bean Configuration. Note that we do not need this for testing. But for the sake of completion, I have included this. The XML configuration is as follows. And now we are ready to write the Integration Test for our Repository Implementation using Embed Mongo. Ideally, the integration tests should be placed in a separate source directory, just like we place our unit tests (ex. src/test/java => src/integration-test/java). However, neither Maven nor Gradle supports this out of the box (yet – v1.2. For Gradle, there’s an on going discussion for this facility). Nevertheless, both Maven and Gradle are flexible, so you can configure the POM / build.gradle to handle this. However, to keep this discussion simple and focused, I will be placing the Integration Tests in the ‘src/test/java’, but I do not recommend this for a real application. Let’s start writing up the Integration Test. First, let’s begin with a simple JUnit based Test for the methods. package com.yohanliyanage.blog.mongoit.repository; import static org.junit.Assert.fail; import org.junit.After; import org.junit.Before; import org.junit.Test; /** * Integration Test for {@link SampleRepositoryMongoImpl}. * * @author Yohan Liyanage */ public class SampleRepositoryMongoImplIntegrationTest { private SampleRepositoryMongoImpl repoImpl; @Before public void setUp() throws Exception { repoImpl = new SampleRepositoryMongoImpl(); } @After public void tearDown() throws Exception { } @Test public void testSave() { fail("Not yet implemented"); } @Test public void testFindByKey() { fail("Not yet implemented"); } } When this JUnit Test Case initializes, we need to fire up EmbedMongo to start an embedded Mongo server. Also, when the Test Case ends, we need to cleanup the DB. The below code snippet does this. package com.yohanliyanage.blog.mongoit.repository; import static org.junit.Assert.fail; import java.io.IOException; import org.junit.*; import org.springframework.data.mongodb.core.MongoTemplate; import com.mongodb.Mongo; import com.yohanliyanage.blog.mongoit.model.Sample; import de.flapdoodle.embed.mongo.MongodExecutable; import de.flapdoodle.embed.mongo.MongodProcess; import de.flapdoodle.embed.mongo.MongodStarter; import de.flapdoodle.embed.mongo.config.MongodConfig; import de.flapdoodle.embed.mongo.config.RuntimeConfig; import de.flapdoodle.embed.mongo.distribution.Version; import de.flapdoodle.embed.process.extract.UserTempNaming; /** * Integration Test for {@link SampleRepositoryMongoImpl}. * * @author Yohan Liyanage */ public class SampleRepositoryMongoImplIntegrationTest { private static final String LOCALHOST = "127.0.0.1"; private static final String DB_NAME = "itest"; private static final int MONGO_TEST_PORT = 27028; private SampleRepositoryMongoImpl repoImpl; private static MongodProcess mongoProcess; private static Mongo mongo; private MongoTemplate template; @BeforeClass public static void initializeDB() throws IOException { RuntimeConfig config = new RuntimeConfig(); config.setExecutableNaming(new UserTempNaming()); MongodStarter starter = MongodStarter.getInstance(config); MongodExecutable mongoExecutable = starter.prepare(new MongodConfig(Version.V2_2_0, MONGO_TEST_PORT, false)); mongoProcess = mongoExecutable.start(); mongo = new Mongo(LOCALHOST, MONGO_TEST_PORT); mongo.getDB(DB_NAME); } @AfterClass public static void shutdownDB() throws InterruptedException { mongo.close(); mongoProcess.stop(); } @Before public void setUp() throws Exception { repoImpl = new SampleRepositoryMongoImpl(); template = new MongoTemplate(mongo, DB_NAME); repoImpl.setMongoOps(template); } @After public void tearDown() throws Exception { template.dropCollection(Sample.class); } @Test public void testSave() { fail("Not yet implemented"); } @Test public void testFindByKey() { fail("Not yet implemented"); } } The initializeDB() method is annotated with @BeforeClass to start this before test case beings. This method fires up an embedded MongoDB instance which is bound to the given port, and exposes a Mongo object which is set to use the given database. Internally, EmbedMongo creates the necessary data files in temporary directories. When this method executes for the first time, EmbedMongo will download the necessary Mongo implementation (denoted by Version.V2_2_0 in above code) if it does not exist already. This is a nice facility specially when it comes to Continuous Integration servers. You don’t have to manually setup Mongo in each of the CI servers. That’s one less external dependency for the tests. In the shutdownDB() method, which is annotated with @AfterClass, we stop the EmbedMongo process. This triggers the necessary cleanups in EmbedMongo to remove the temporary data files, restoring the state to where it was before Test Case was executed. We have now updated setUp() method to build a Spring MongoTemplate object which is backed by the Mongo instance exposed by EmbedMongo, and to setup our RepoImpl with that template. The tearDown() method is updated to drop the ‘Sample’ collection to ensure that each of our test methods start with a clean state. Now it’s just a matter of writing the actual test methods. Let’s start with the save method test. @Test public void testSave() { Sample sample = new Sample("TEST", "2"); repoImpl.save(sample); int samplesInCollection = template.findAll(Sample.class).size(); assertEquals("Only 1 Sample should exist collection, but there are " + samplesInCollection, 1, samplesInCollection); } We create a Sample object, pass it to repoImpl.save(), and assert to make sure that there’s only one Sample in the Sample collection. Simple, straight-forward stuff. And here’s the test method for findByKey method. @Test public void testFindByKey() { // Setup Test Data List samples = Arrays.asList( new Sample("TEST", "1"), new Sample("TEST", "25"), new Sample("TEST2", "66"), new Sample("TEST2", "99")); for (Sample sample : samples) { template.save(sample); } // Execute Test List matches = repoImpl.findByKey("TEST"); // Note: Since our test data (populateDummies) have only 2 // records with key "TEST", this should be 2 assertEquals("Expected only two samples with key TEST, but there are " + matches.size(), 2, matches.size()); } Initially, we setup the data by adding a set of Sample objects into the data store. It’s important that we directly use template.save() here, because repoImpl.save() is a method under-test. We are not testing that here, so we use the underlying “trusted” template.save() during data setup. This is a basic concept in Unit / Integration testing. Then we execute the method under test ‘findByKey’, and assert to ensure that only two Samples matched our query. Likewise, we can continue to write more tests for each of the repository methods, including negative tests. And here’s the final Integration Test file. package com.yohanliyanage.blog.mongoit.repository; import static org.junit.Assert.*; import java.io.IOException; import java.util.Arrays; import java.util.List; import org.junit.*; import org.springframework.data.mongodb.core.MongoTemplate; import com.mongodb.Mongo; import com.yohanliyanage.blog.mongoit.model.Sample; import de.flapdoodle.embed.mongo.MongodExecutable; import de.flapdoodle.embed.mongo.MongodProcess; import de.flapdoodle.embed.mongo.MongodStarter; import de.flapdoodle.embed.mongo.config.MongodConfig; import de.flapdoodle.embed.mongo.config.RuntimeConfig; import de.flapdoodle.embed.mongo.distribution.Version; import de.flapdoodle.embed.process.extract.UserTempNaming; /** * Integration Test for {@link SampleRepositoryMongoImpl}. * * @author Yohan Liyanage */ public class SampleRepositoryMongoImplIntegrationTest { private static final String LOCALHOST = "127.0.0.1"; private static final String DB_NAME = "itest"; private static final int MONGO_TEST_PORT = 27028; private SampleRepositoryMongoImpl repoImpl; private static MongodProcess mongoProcess; private static Mongo mongo; private MongoTemplate template; @BeforeClass public static void initializeDB() throws IOException { RuntimeConfig config = new RuntimeConfig(); config.setExecutableNaming(new UserTempNaming()); MongodStarter starter = MongodStarter.getInstance(config); MongodExecutable mongoExecutable = starter.prepare(new MongodConfig(Version.V2_2_0, MONGO_TEST_PORT, false)); mongoProcess = mongoExecutable.start(); mongo = new Mongo(LOCALHOST, MONGO_TEST_PORT); mongo.getDB(DB_NAME); } @AfterClass public static void shutdownDB() throws InterruptedException { mongo.close(); mongoProcess.stop(); } @Before public void setUp() throws Exception { repoImpl = new SampleRepositoryMongoImpl(); template = new MongoTemplate(mongo, DB_NAME); repoImpl.setMongoOps(template); } @After public void tearDown() throws Exception { template.dropCollection(Sample.class); } @Test public void testSave() { Sample sample = new Sample("TEST", "2"); repoImpl.save(sample); int samplesInCollection = template.findAll(Sample.class).size(); assertEquals("Only 1 Sample should exist in collection, but there are " + samplesInCollection, 1, samplesInCollection); } @Test public void testFindByKey() { // Setup Test Data List samples = Arrays.asList( new Sample("TEST", "1"), new Sample("TEST", "25"), new Sample("TEST2", "66"), new Sample("TEST2", "99")); for (Sample sample : samples) { template.save(sample); } // Execute Test List matches = repoImpl.findByKey("TEST"); // Note: Since our test data (populateDummies) have only 2 // records with key "TEST", this should be 2 assertEquals("Expected only two samples with key TEST, but there are " + matches.size(), 2, matches.size()); } } On a side note, one of the key concerns with Integration Tests is the execution time. We all want to keep our test execution times as low as possible, ideally a couple of seconds to make sure that we can run all the tests during CI, with minimal build and verification times. However, since Integration Tests rely on underlying infrastructure, usually Integration Tests take time to run. But with EmbedMongo, this is not the case. In my machine, above test suite runs in 1.8 seconds, and each test method takes only .166 seconds max. See the screenshot below. I have uploaded the code for above project into GitHub. You can download / clone it from here: https://github.com/yohanliyanage/blog-mongo-integration-tests. For more information regarding EmbedMongo, refer to their site at GitHub https://github.com/flapdoodle-oss/embedmongo.flapdoodle.de.

For some an XML schema is a strict set of rules for how the XML document must be structured. But for others it is a general guideline to indicate what the XML should look like. This means that sometimes people want to accept input that doesn't conform to the XML schema for some reason. In this example I will demonstrate how this can be done by leveraging a SAX XMLFilter. Java Model Below is the Java model that will be used for this example. Customer package blog.namespace.sax; import javax.xml.bind.annotation.XmlRootElement; @XmlRootElement public class Customer { private String name; public String getName() { return name; } public void setName(String name) { this.name = name; } } package-info We will use the package level @XmlSchema annotation to specify the namespace qualification for our model. @XmlSchema( namespace="http://www.example.com/customer", elementFormDefault=XmlNsForm.QUALIFIED) package blog.namespace.sax; import javax.xml.bind.annotation.*; XML Input (input.xml) Even though our metadata specified that all the elements should be qualified with a namespace (http://www.example.com/customer) our input document is not namespace qualified. An XMLFilter will be used to add the namespace during the unmarshal operation. Jane Doe XMLFilter (NamespaceFilter) The easiest way to create an XMLFilter is to extend XMLFilterImpl. For our use case we will override the startElement and endElement methods. In each of these methods we will call the corresponding super method passing in the default namespace as the URI parameter. package blog.namespace.sax; import org.xml.sax.*; import org.xml.sax.helpers.XMLFilterImpl; public class NamespaceFilter extends XMLFilterImpl { private static final String NAMESPACE = "http://www.example.com/customer"; @Override public void endElement(String uri, String localName, String qName) throws SAXException { super.endElement(NAMESPACE, localName, qName); } @Override public void startElement(String uri, String localName, String qName, Attributes atts) throws SAXException { super.startElement(NAMESPACE, localName, qName, atts); } } Demo In the demo code below we will do a SAX parse of the XML document. The XMLReader will be wrapped in our XMLFilter. We will leverage JAXB's UnmarshallerHandler as the ContentHandler. Once the parse has been done we can ask the UnmarshallerHandler for the resulting Customer object. package blog.namespace.sax; import javax.xml.bind.*; import javax.xml.parsers.*; import org.xml.sax.*; public class Demo { public static void main(String[] args) throws Exception { // Create the JAXBContext JAXBContext jc = JAXBContext.newInstance(Customer.class); // Create the XMLFilter XMLFilter filter = new NamespaceFilter(); // Set the parent XMLReader on the XMLFilter SAXParserFactory spf = SAXParserFactory.newInstance(); SAXParser sp = spf.newSAXParser(); XMLReader xr = sp.getXMLReader(); filter.setParent(xr); // Set UnmarshallerHandler as ContentHandler on XMLFilter Unmarshaller unmarshaller = jc.createUnmarshaller(); UnmarshallerHandler unmarshallerHandler = unmarshaller .getUnmarshallerHandler(); filter.setContentHandler(unmarshallerHandler); // Parse the XML InputSource xml = new InputSource("src/blog/namespace/sax/input.xml"); filter.parse(xml); Customer customer = (Customer) unmarshallerHandler.getResult(); // Marshal the Customer object back to XML Marshaller marshaller = jc.createMarshaller(); marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); marshaller.marshal(customer, System.out); } } Output Below is the output from running the demo code. Note how the output contains the namespace qualification based on the metadata. Jane Doe Further Reading If you enjoyed this post then you may also be interested in: JAXB & Namespaces Preventing Entity Expansion Attacks in JAXB