Data access, specifically SQL access from within Java, has never been nice. This is in large part due to the fact that the JDBC api has a lot of ceremony. Java 7 vastly improved things with ARM blocks by taking away a lot of the ceremony around managing database objects such as Statements and ResultSets but fundamentally the code flow is still the same. Java 8 Lambdas gives us a very nice tool for improving the flow of JDBC. Out first attempt at improving things here is very simply to make it easy to work with ajava.sql.ResultSet. Here we simply wrap the ResultSet iteration and then delegate it to Lambda function. This is very similar in concept to Spring's JDBCTemplate. NOTE: I've released All the code snippets you see here under an Apache 2.0 license on Github. First we create a functional interface called ResultSetProcessor as follows: @FunctionalInterface public interface ResultSetProcessor { public void process(ResultSet resultSet, long currentRow) throws SQLException; } Very straightforward. This interface takes the ResultSet and the current row of theResultSet as a parameter. Next we write a simple utility to which executes a query and then calls ourResultSetProcessor each time we iterate over the ResultSet: public static void select(Connection connection, String sql, ResultSetProcessor processor, Object... params) { try (PreparedStatement ps = connection.prepareStatement(sql)) { int cnt = 0; for (Object param : params) { ps.setObject(++cnt, param)); } try (ResultSet rs = ps.executeQuery()) { long rowCnt = 0; while (rs.next()) { processor.process(rs, rowCnt++); } } catch (SQLException e) { throw new DataAccessException(e); } } catch (SQLException e) { throw new DataAccessException(e); } } Note I've wrapped the SQLException in my own unchecked DataAccessException. Now when we write a query it's as simple as calling the select method with a connection and a query: select(connection, "select * from MY_TABLE",(rs, cnt)-> { System.out.println(rs.getInt(1)+" "+cnt) }); So that's great, but I think we can do more... One of the nifty Lambda additions in Java is the new Streams API. This would allow us to add very powerful functionality with which to process a ResultSet. Using the Streams API over a ResultSet however creates a bit more of a challenge than the simple select with Lambda in the previous example. The way I decided to go about this is create my own Tuple type which represents a single row from a ResultSet. My Tuple here is the relational version where a Tuple is a collection of elements where each element is identified by an attribute, basically a collection of key value pairs. In our case the Tuple is ordered in terms of the order of the columns in the ResultSet. The code for the Tuple ended up being quite a bit so if you want to take a look, see the GitHub project in the resources at the end of the post. Currently the Java 8 API provides the java.util.stream.StreamSupport object which provides a set of static methods for creating instances of java.util.stream.Stream. We can use this object to create an instance of a Stream. But in order to create a Stream it needs an instance ofjava.util.stream.Spliterator. This is a specialised type for iterating and partitioning a sequence of elements, the Stream needs for handling operations in parallel. Fortunately the Java 8 api also provides the java.util.stream.Spliterators class which can wrap existing Collection and enumeration types. One of those types being ajava.util.Iterator. Now we wrap a query and ResultSet in an Iterator: public class ResultSetIterator implements Iterator { private ResultSet rs; private PreparedStatement ps; private Connection connection; private String sql; public ResultSetIterator(Connection connection, String sql) { assert connection != null; assert sql != null; this.connection = connection; this.sql = sql; } public void init() { try { ps = connection.prepareStatement(sql); rs = ps.executeQuery(); } catch (SQLException e) { close(); throw new DataAccessException(e); } } @Override public boolean hasNext() { if (ps == null) { init(); } try { boolean hasMore = rs.next(); if (!hasMore) { close(); } return hasMore; } catch (SQLException e) { close(); throw new DataAccessException(e); } } private void close() { try { rs.close(); try { ps.close(); } catch (SQLException e) { //nothing we can do here } } catch (SQLException e) { //nothing we can do here } } @Override public Tuple next() { try { return SQL.rowAsTuple(sql, rs); } catch (DataAccessException e) { close(); throw e; } } } This class basically delegates the iterator methods to the underlying result set and then on the next() call transforms the current row in the ResultSet into my Tuple type. And that's the basics done (This class will need a little bit more work though). All that's left is to wire it all together to make a Stream object. Note that due to the nature of a ResultSet it's not a good idea to try process them in parallel, so our stream cannot process in parallel. public static Stream stream(final Connection connection, final String sql, final Object... parms) { return StreamSupport .stream(Spliterators.spliteratorUnknownSize( new ResultSetIterator(connection, sql), 0), false); } Now it's straightforward to stream a query. In the usage example below I've got a table TEST_TABLE with an integer column TEST_ID which basically filters out all the non even numbers and then runs a count: long result = stream(connection, "select TEST_ID from TEST_TABLE") .filter((t) -> t.asInt("TEST_ID") % 2 == 0) .limit(100) .count(); And that's it! We now have a very powerful way of working with a ResultSet. So all this code is available under an Apache 2.0 license on GitHub here. I've rather lamely dubbed the project "lambda tuples," and the purpose really is to experiment and see where you can take Java 8 and Relational DB access, so please download or feel free to contribute.

This is a code snippet to retrieve the path of the current running web application project in java public String getPath() throws UnsupportedEncodingException { String path = this.getClass().getClassLoader().getResource("").getPath(); String fullPath = URLDecoder.decode(path, "UTF-8"); String pathArr[] = fullPath.split("/WEB-INF/classes/"); System.out.println(fullPath); System.out.println(pathArr[0]); fullPath = pathArr[0]; String reponsePath = ""; // to read a file from webcontent reponsePath = new File(fullPath).getPath() + File.separatorChar + "newfile.txt"; return reponsePath; }

This post demonstrates a Groovy script for uncompressing files with the 7-Zip archive format. The two primary objectives of this post are to demonstrate uncompressing 7-Zip files with Groovy and the handy 7-Zip-JBindingand to call out and demonstrate some key characteristics of Groovy as a scripting language. The 7-Zip page describes 7-Zip as "a file archiver with a high compression ratio." The page further adds, "7-Zip is open source software. Most of the source code is under the GNU LGPL license." More license information in available on the site along with information on the 7z format ("LZMA is default and general compression method of 7z format"). The 7-Zip page describes it as "a Java wrapper for 7-Zip C++ library" that "allows extraction of many archive formats using a very fast native library directly from Java through JNI." The 7z format is based on "LZMA andLZMA2 compression." Although there is an LZMA SDK available, it is easier to use the open source (SourceForge)7-Zip-JBinding project when manipulating 7-Zip files with Java. A good example of using Java with 7-Zip-JBinding to uncompress 7z files is available in the StackOverflowthread Decompress files with .7z extension in java. Dark Knight's response indicates how to use Java with 7-Zip-JBinding to uncompress a 7z file. I adapt Dark Knight's Java code into a Groovy script in this post. To demonstrate the adapted Groovy code for uncompressing 7z files, I first need a 7z file that I can extract contents from. The next series of screen snapshots show me using Windows 7-Zip installed on my laptop to compress the six PDFs available under the Guava Downloads page into a single 7z file called Guava.7z. Six Guava PDFs Sitting in a Folder Contents Selected and Right-Click Menu to Compress to 7z Format Guava.7z Compressed Archive File Created With a 7z file in place, I now turn to the adapted Groovy script that will extract the contents of this Guava.7zfile. As mentioned previously, this Groovy script is an adaptation of the Java code provided by Dark Knight on aStackOverflow thread. unzip7z.groovy // // This Groovy script is adapted from Java code provided at // http://stackoverflow.com/a/19403933 import static java.lang.System.err as error import java.io.File import java.io.FileNotFoundException import java.io.FileOutputStream import java.io.IOException import java.io.RandomAccessFile import java.util.Arrays import net.sf.sevenzipjbinding.ExtractOperationResult import net.sf.sevenzipjbinding.ISequentialOutStream import net.sf.sevenzipjbinding.ISevenZipInArchive import net.sf.sevenzipjbinding.SevenZip import net.sf.sevenzipjbinding.SevenZipException import net.sf.sevenzipjbinding.impl.RandomAccessFileInStream import net.sf.sevenzipjbinding.simple.ISimpleInArchive import net.sf.sevenzipjbinding.simple.ISimpleInArchiveItem if (args.length < 1) { println "USAGE: unzip7z.groovy .7z\n" System.exit(-1) } def fileToUnzip = args[0] try { RandomAccessFile randomAccessFile = new RandomAccessFile(fileToUnzip, "r") ISevenZipInArchive inArchive = SevenZip.openInArchive(null, new RandomAccessFileInStream(randomAccessFile)) ISimpleInArchive simpleInArchive = inArchive.getSimpleInterface() println "${'Hash'.center(10)}|${'Size'.center(12)}|${'Filename'.center(10)}" println "${'-'.multiply(10)}+${'-'.multiply(12)}+${'-'.multiply(10)}" simpleInArchive.getArchiveItems().each { item -> final int[] hash = new int[1] if (!item.isFolder()) { final long[] sizeArray = new long[1] ExtractOperationResult result = item.extractSlow( new ISequentialOutStream() { public int write(byte[] data) throws SevenZipException { //Write to file try { File file = new File(item.getPath()) file.getParentFile()?.mkdirs() FileOutputStream fos = new FileOutputStream(file) fos.write(data) fos.close() } catch (Exception e) { printExceptionStackTrace("Unable to write file", e) } hash[0] ^= Arrays.hashCode(data) // Consume data sizeArray[0] += data.length return data.length // Return amount of consumed data } }) if (result == ExtractOperationResult.OK) { println(String.format("%9X | %10s | %s", hash[0], sizeArray[0], item.getPath())) } else { error.println("Error extracting item: " + result) } } } } catch (Exception e) { printExceptionStackTrace("Error occurs", e) System.exit(1) } finally { if (inArchive != null) { try { inArchive.close() } catch (SevenZipException e) { printExceptionStackTrace("Error closing archive", e) } } if (randomAccessFile != null) { try { randomAccessFile.close() } catch (IOException e) { printExceptionStackTrace("Error closing file", e) } } } /** * Prints the stack trace of the provided exception to standard error without * Groovy meta data trace elements. * * @param contextMessage String message to precede stack trace and provide context. * @param exceptionToBePrinted Exception whose Groovy-less stack trace should * be printed to standard error. * @return Exception derived from the provided Exception but without Groovy * meta data calls. */ def Exception printExceptionStackTrace( final String contextMessage, final Exception exceptionToBePrinted) { error.print "${contextMessage}: ${org.codehaus.groovy.runtime.StackTraceUtils.sanitize(exceptionToBePrinted).printStackTrace()}" } In my adaptation of the Java code into the Groovy script shown above, I left most of the exception handling in place. Although Groovy allows exceptions to be ignored whether they are checked or unchecked, I wanted to maintain this handling in this case to make sure resources are closed properly and that appropriate error messages are presented to users of the script. One thing I did change was to make all of the output that is error-related be printed to standard error rather than to standard output. This required a few changes. First, I used Groovy's capability to rename something that is statically imported (see my related post Groovier Static Imports) to reference "java.lang.System.err" as "error" so that I could simply use "error" as a handle in the script rather than needing to use "System.err" to access standard error for output. Because Throwable.printStackTrace() already writes to standard error rather than standard output, I just used it directly. However, I placed calls to it in a new method that would first runStackTraceUtils.sanitize(Throwable) to remove Groovy-specific calls associated with Groovy's runtime dynamic capabilities from the stack trace. There were some other minor changes to the script as part of making it Groovier. I used Groovy's iteration on the items in the archive file rather than the Java for loop, removed semicolons at the ends of statements, used Groovy's String GDK extension for more controlled output reporting [to automatically center titles and tomultiply a given character by the appropriate number of times it needs to exist], and took advantage ofGroovy's implicit inclusion of args to add a check to ensure file for extraction was provided to the script. With the file to be extracted in place and the Groovy script to do the extracting ready, it is time to extract the contents of the Guava.7z file I demonstrated generating earlier in this post. The following command will run the script and places the appropriate 7-Zip-JBinding JAR files on the classpath. groovy -classpath "C:/sevenzipjbinding/lib/sevenzipjbinding.jar;C:/sevenzipjbinding/lib/sevenzipjbinding-Windows-x86.jar" unzip7z.groovy C:\Users\Dustin\Downloads\Guava\Guava.7z Before showing the output of running the above script against the indicated Guava.7z file, it is important to note the error message that will occur if the native operating system specific 7-Zip-JBinding JAR (sevenzipjbinding-Windows-x86.jar in my laptop's case) is not included on the classpath of the script. As the last screen snapshot indicates, neglecting to include the native JAR on the classpath leads to the error message: "Error occurs: java.lang.RuntimeException: SevenZipJBinding couldn't be initialized automaticly using initialization from platform depended JAR and the default temporary directory. Please, make sure the correct 'sevenzipjbinding-.jar' file is in the class path or consider initializing SevenZipJBinding manualy using one of the offered initialization methods: 'net.sf.sevenzipjbinding.SevenZip.init*()'" Although I simply added C:/sevenzipjbinding/lib/sevenzipjbinding-Windows-x86.jar to my script's classpath to make it work on this laptop, a more robust script might detect the operating system and apply the appropriate JAR to the classpath for that operating system. The 7-Zip-JBinding Download pagefeatures multiple platform-specific downloads (including platform-specific JARs) such assevenzipjbinding-4.65-1.06-rc-extr-only-Windows-amd64.zip, sevenzipjbinding-4.65-1.06-rc-extr-only-Mac-x86_64.zip, sevenzipjbinding-4.65-1.06-rc-extr-only-Mac-i386.zip, and sevenzipjbinding-4.65-1.06-rc-extr-only-Linux-i386.zip. Once the native 7-Zip-JBinding JAR is included on the classpath along with the core sevenzipjbinding.jar JAR, the script runs beautifully as shown in the next screen snapshot. The script extracts the contents of the 7z file into the same working directory as the Groovy script. A further enhancement would be to modify the script to accept a directory to which to write the extracted files or might write them to the same directory as the 7z archive file by default instead. Use of Groovy's built-in CLIBuilder support could also improve the script. Groovy is my preferred language of choice when scripting something that makes use of the JVM and/or of Java libraries and frameworks. Writing the script that is the subject of this post has been another reminder of that.

when you build applications based on mule ee (enterprise edition) and you are using maven to build your projects, you will notice you have dependencies to libraries that are not available in the public maven repos. to add these libraries to your local maven repo the mule distribution comes with a script ‘populate_m2_repo’ which is described here how to use it. now that is okay if you are the only developer and you are running your continuous integration on your local machine. in my case we are using artifactory as our company maven repository and also our build server is using it as the maven repo. so what i wanted was not to populate my local repository but the artifactory instance with all mule libraries. to do so i did two things: first make sure that maven is authorised to add libraries to artifactory. you can do this by adding the following to your settings.xml: artifactory admin password second step is to modify the original ‘populate_m2_repo.groovy’ script. replace the following line: mvn(["install:install-file", "-dgroupid=${project.groupid}", "-dartifactid=${project.artifactid}", "-dversion=${version}", "-dpackaging=pom", "-dfile=${localpom.canonicalpath}"]) with mvn(["deploy:deploy-file", "-dgroupid=${project.groupid}", "-dartifactid=${project.artifactid}", "-dversion=${version}", "-dpackaging=pom", "-dfile=${localpom.canonicalpath}", "-drepositoryid=arti", "-durl=http://localhost:8080/artifactory/libs-release-local" ]) and do the same for the line: def args = ["install:install-file", "-dgroupid=${pomprops.groupid}", "-dartifactid=${pomprops.artifactid}", "-dversion=${pomprops.version}", "-dpackaging=jar", "-dfile=${f.canonicalpath}", "-dpomfile=${localpom.canonicalpath}"] by replacing it with: def args = ["deploy:deploy-file", "-dgroupid=${pomprops.groupid}", "-dartifactid=${pomprops.artifactid}", "-dversion=${pomprops.version}", "-dpackaging=jar", "-dfile=${f.canonicalpath}", "-dpomfile=${localpom.canonicalpath}", "-drepositoryid=arti", "-durl=http://localhost:8080/artifactory/libs-release-local" ] now you can run the script with: ./populate_m2_repo bla as you can see it doesn’t really matter what you supply as m2_repo_home here because the libraries are uploaded to artifactory anyway. if you want you can replace the hardcoded url for artifactory in the script with the supplied parameter but in my case this solution was sufficient

This post is following up the experiment we ran exactly a year ago comparing the performance of different GC algorithms in real-life settings. We took the same experiment, expanded the tests to contain the G1 garbage collector and ran the tests on different platform. This year our tests were run with the following Garbage Collectors: -XX:+UseParallelOldGC -XX:+UseConcMarkSweepGC -XX:+UseG1GC Description of the environment The experiment was ran on out-of-the-box JIRA configuration. The motivation for the test run was loud and clear – Minecraft, Dalvik-based Angry Bird and Eclipse asides, JIRA should be one of the most popular Java applications out there. And opposed to the alternatives it is a more typical representative of what most of us are dealing with on the everyday business – after all Java is still by far most used in server side Java EE apps. What also affected our decision was – the engineers from Atlassian ship nicely packaged load tests along the JIRA download. So we had a benchmark to use for our configuration. We carefully unzipped our fresh JIRA 6.1 download and installed it on a Mac OS X Mavericks. And ran the bundled tests without changing anything in the default memory settings. The Atlassian team had been kind enough to set them for us: -Xms256m -Xmx768m -XX:MaxPermSize=256m The tests used JIRA functionality in different common ways – creating tasks, assigning tasks, resolving tasks, searching and discovering tasks, etc. Total runtime for the test was 30 minutes. We ran the test using three different garbage collection algorithms – Parallel, CMS and G1 were used in our case. Each test started with a fresh JVM boot, followed by prepopulating the storage to the exactly the same state. Only after the preparations we launched the load generation. Results During each run we have collected GC logs using -XX:+PrintGCTimeStamps -Xloggc:/tmp/gc.log -XX:+PrintGCDetails and analyzed this statistics with the help of GCViewer The results can be aggregated as follows. Note that all measurements are in milliseconds: Parallel CMS G1 Total GC pauses 20 930 18 870 62 000 Max GC pause 721 64 50 Interpretation and results First stop – Parallel GC (-XX:+UseParallelOldGC). Out of the 30 minutes the tests took to complete, we spent close to 21 seconds in GC pauses with the parallel collector. And the longest pause took 721 milliseconds. So let us take this as the baseline: GC cycles reduced the throughput by 1.1% of the total runtime. And the worst-case latency was 721ms. Next contestant: CMS (-XX:+UseConcMarkSweepGC). Again, 30 minutes of tests out of which we lost a bit less than 19 seconds to GC. Throughput-wise this is roughly in the same neighbourhood as the parallel mode. Latency on the other hand has been improved significantly - the worst-case latency is reduced more than 10 times! We are now facing just 64ms as the maximum pause time from the GC. Last experiment used the newest and shiniest GC algorithm available – G1 (-XX:+UseG1GC). The very same tests were run and throughput-wise we saw results suffering severely. This time our application spent more than a minute waiting for the GC to complete. Comparing this to the just 1% of the overhead with CMS, we are now facing close to 3.5% effect on the throughput. But if you really do not care about throughput and want to squeeze out the last bit from the latency then – we have improved around 20% comparing to the already-good CMS – using G1 saw the longest GC pause only taking 50ms. Conclusion As always, trying to summarize such an experiment into a single conclusion is dangerous. So if you have time and required skills – definitely go ahead and measure your own environment instead of adopting to one-size-fits-all solution. But if I would dare to make such a conclusion, I would say that CMS is still the best “default” option to go with. G1 throughput is still so much worse that the improved latency is usually not worth it. If you enjoyed the content, consider subscribe to either our RSS feed or Twitter stream – we continue to publish on different performance optimization topics.

eclipse is very workspace centric: it only knows and deals with files in the workspace. so it is easy to compare and merge files present in the workspace: i select both files/folders and compare them with each other: compare with each other but what if the files and folders are not in the workspace? hidden option to compare external files as outlined in this post , it needs a special plugin to search for files outside the eclipse workspace. and doing a file or folder compare outside of the workspace requires a trick as shown in this post . thanks to a tip from john there is another (hidden) way in eclipse to compare external files keyboard shortcut the trick is described in this article and requires a keyboard shortcut assigned. select the menu window > preferences > general > keys and assign a shortcut key for ‘ compare with other resource ‘: compare with other resource key binding (i’m using ctrl+shift+home above). comparing to compare, i have first to select a file, folder or project, then i press my shortcut. then the following dialog shows up (with the selection as default): select resource to compare if the dialog does not show up, then i probably have not selected a file or folder in the eclipse project view. now i can select the external files or folders to compare with: selected external files with this, i can now compare and merge my files and folders with the eclipse compare view: compare view in eclipse i can select two files/folders and then press the shortcut, and it will populate the search dialog values. drag & drop that compare dialog has a nice feature: i can drag&drop files and folders too: c:\programdata\processor expert\cwmcu_pe5_00\examples\frdm-kl25z\frdm-kl25z_rnet\sources summary with ‘ compare with other resource ‘ i have a way to compare files/folders, and i’m not limited to the workspace files. the only disadvantage is that i need to assign a shortcut for it first. beside of that: yet another hidden treasure in eclipse . happy comparing

One of the most welcoming features of the new Mule Studio 3.4 is the Maven support. I was very keen to try out this new feature. I grabbed one of the projects I was working on, and imported it into Mule Studio through File -> Import -> Existing Maven Projects. Everything is as good as it gets. However I had one issue. Every time I wanted to run my flows as Mule Application, Mule Studio was doing a whole build of my project, including running the tests. Since this was a large project, I wanted to avoid running all the tests every time I needed to start the application. So I started by adding -DskipTests=true as a VM argument in the run configuration, but this did not work. My second attempt was to add the MAVEN_OPTS environmental variable and set it to -DskipTests=true, so back to the Mule Studio run configuration, clicked the Environment Variables table, set it there. Again, unfortunately this did not work. Worry not, there is a way. The third and final attempt was to check if the Mule Studio Maven support provides its own configuration, and luckily it does. So to fix it, Window -> Preferences (or MuleStudio -> Preference if you are using a MAC), navigate to Mule Studio on the left hand panel, expand that, and choose “Maven Settings”. In the configuration panel on the right hand side, you can type -DskipTests=true in the text box labelled as “MAVEN_OPTS environment variable”. Running my flow now does not run the tests. One small tip, -DskipTests=true and -Dmaven.test.skip=true are slightly different. If you go for the second option, Maven won’t even build your test classes, hence if you try to run any JUnit test from Mule Studio after a build, it will fail with ClassNotFoundException. Therefore I recommend the first option.

when configuring an azure virtual network one of the most common things you'll want to do is setup a point-to-site vpn so that you can actually get to your servers to manage and maintain them. azure point-to-site vpns use client certificates to secure connections which can be quite complicated to configure so microsoft has gone the extra mile to make it easy for you to configure and get setup – sadly at the cost of losing the ability to connect through the command line or through powershell – let's change that. current state of play == no command line vpn connections normally when you want to launch a vpn from the cli or powershell in windows you can simply use the following command: rasdial "my home vpn" the azure pre-packaged vpn doesn't allow this because it's really just not a normal vpn. it's something else , something mysterious - not a normal native windows vpn connection. when you run the azure vpn through the command line you get this (you'll see a hint as to why i'd be using azure point-to-site in this screenshot): azure vpns don't appear to support this. if you want to keep your servers behind a private network in azure and use continuous deployment to get your code into production this makes it hard to deploy without a human being around. not really the best case scenario – especially when you remind yourself that automated builds aim to do away with human error altogether. what the azure point-to-site looks like out of the box when you first go to setup a point-to-site vpn into your azure virtual network microsoft points you at a page that walks you through creating a client certificate on your local machine to use as authentication. they then get you to download a package for setting up the azure vpn ras dialler on your local machine. this is accessed from within the azure "networks" page for your virtual network. you install this package and then whenever connecting you're greeted with a connection screen that you might of seen in a previous life. and by seen i don't mean that windows azure virtual networks have been around for ages. but more that the login screen may look familiar. this is because this login screen is a microsoft " connection manager " login screen and has been around for a while. example from technet (note extremely dated bitmap awesomeness): connection manager is used to pre-package vpn and dial up connections for easy-install distribution in a large organisation. this also means we can reconstruct the underlying vpn connection and use it as a normal vpn – claiming back our cli super powers. digging through the details so what we really want to know is: what is this mystical vpn technology the people at microsoft have bestowed upon us? here's how i started getting more information about the implementation: connecting once successfully then disconnect. open it up again to connect and click on properties then clicking on view log you'll then be greeted by something that looks like this: ****************************************************************** operating system : windows nt 6.2 dialler version : 7.2.9200.16384 connection name : my azure virtual network all users/single user : single user start date/time : 24/11/2013, 7:50:31 ****************************************************************** module name, time, log id, log item name, other info for connection type, 0=dial-up, 1=vpn, 2=vpn over dial-up ****************************************************************** [cmdial32] 7:50:31 03 pre-init event callingprocess = c:\windows\system32\cmmon32.exe [cmdial32] 7:50:39 04 pre-connect event connectiontype = 1 [cmdial32] 7:50:39 06 pre-tunnel event username = myclientsslcertificate domain = dunsetting = [obfuscated azure gateway id] tunnel devicename = tunneladdress = [obfuscated azure gateway id].cloudapp.net [cmdial32] 7:50:44 07 connect event [cmdial32] 7:50:44 08 custom action dll actiontype = connect actions description = to update your routing table actionpath = c:\users\doug\appdata\roaming\microsoft\network\connections\cm\[obfuscated azure gateway id]\cmroute.dll returnvalue = 0x0 [cmmon32] 7:56:21 23 external disconnect [cmdial32] 7:56:21 13 disconnect event callingprocess = c:\windows\explorer.exe more importantly you'll see this path included in the connection: within this folder is all the magic connection manager odds and ends. apologies for the [obfuscated], simply the path contains information to my azure endpoint. within this folder you'll see a bunch of files: most importantly there is a pbk file – a personal phonebook. this is what stores the connect settings for the vpn as is a commonly distributed way of sending out connection settings in the enterprise. if you run this on its own you'll actually be able to connect to the vpn directly (without your network routes being updated). this phonebook is where we can steal our settings from to recreate a command line driven connection. setting it up open up the properties of your azure point-to-site vpn phonebook above, and copy the connection address. it will look like this: azuregateway-[guid].cloudapp.net open network sharing centre , and create a new connection. then select connect to a workplace . select that you'll "use my internet connection". then enter your azure point-to-site vpn address and then give your new connection a name. remember this name for later then click create to save your vpn. now open the connection properties for your newly created vpn. this is where we'll use the settings in your azure diallers config to setup your connection. i'll save you the hassle of showing you me copying the settings from one connection to another and instead i'll just focus on what you need to set them to. flick over to the options tab and then click ppp settings . click the 2 missing options enable software compression and negotiate multi-link for single-link connections . set the type of vpn to secure socket tunnelling protocol (sstp), turn on eap and select microsoft: smart card of other certificate as the authentication type. then click on properties . select "use a certificate on this computer", un-tick "connect to these servers", and then select the certificate that uses your azure endpoint uri as its certificate name and then save out. then flick over to the network tab. open tcp/ipv4 then advanced then untick use default gateway on remote network . this setting stops internet traffic going over the vpn while you're connected so you can still surf reddit while managing your azure environment. close the vpn configuration panel. you now have a working vpn connection to azure. when you connect using windows you'll be asked to select the name of the client certificate you'll be authenticating with. you select the certificate you created and uploaded into azure before you setup your connection. when you connect using the command line you don't need to specify your certificate: rasdial "azure vpn" but there's one catch: your local machine's route table doesn't know when to send any traffic to your azure virtual network. the network link is there, but windows doesn't know what to send over your internet link and what to send over the vpn link. you see microsoft did a few things when they packaged your connection manager, and one of these things was to also copy a file called "cmroute.dll" and call this after connection to route your traffic onto your virtual network. this file altered your routing table to route traffic to your virtual network subnets through the vpn connection . we can do the same thing – so lets go about it. what's this about routing... rooting (for the english speakers in the room) my azure virtual network consists of the following network range: 10.0.0.0/8 i also have the following subnets for different machines groups. 10.0.1.0/24 (web servers) 10.0.2.0/24 (application servers) 10.0.3.0/24 (management services) my pptp connections, or point-to-site connections sit on the range: 172.16.0/24 this means that when i connect to the azure vpn i will get an ip address in this range. example: 172.16.0.17 when this happens we need to tell windows to route all traffic going to my 10.0.x.x range ip addresses through the ip address that has been given to us by azure's vpn rras service. you can see your current routing table by entering route print into a command prompt or powershell console. automating the routing additions luckily the windows task scheduler supports event listeners that allow us to watch for vpn connections and run commands off the back of them. take the below powershell script below and save it for arguments sake in c:\scripts\updateroutetableforazurevpn.ps1 ############################################################# # adds ip routes to azure vpn through the point-to-site vpn ############################################################# # define your azure subnets $ips = @("10.0.1.0", "10.0.2.0","10.0.3.0") # point-to-site ip address range # should be the first 4 octets of the ip address '172.16.0.14' == '172.16.0. $azurepptprange = "172.16.0." # find the current new dhcp assigned ip address from azure $azureipaddress = ipconfig | findstr $azurepptprange # if azure hasn't given us one yet, exit and let u know if (!$azureipaddress){ "you do not currently have an ip address in your azure subnet." exit 1 } $azureipaddress = $azureipaddress.split(": ") $azureipaddress = $azureipaddress[$azureipaddress.length-1] $azureipaddress = $azureipaddress.trim() # delete any previous configured routes for these ip ranges foreach($ip in $ips) { $routeexists = route print | findstr $ip if($routeexists) { "deleting route to azure: " + $ip route delete $ip } } # add our new routes to azure virtual network foreach($subnet in $ips) { "adding route to azure: " + $subnet echo "route add $ip mask 255.255.255.0 $azureipaddress" route add $subnet mask 255.255.255.0 $azureipaddress } now execute the following from an elevated command prompt window. this tells windows to add an event listener based task that looks for events to our "azure vpn" connection and if it sees them, it runs our powershell script. schtasks /create /f /tn "vpn connection update" /tr "powershell.exe -noninteractive -command c:\scripts\updateroutetableforazurevpn.ps1" /sc onevent /ec application /mo "*[system[(level=4 or level=0) and (eventid=20225)]] and *[eventdata[data='azure vpn']] " if i then connect to my vpn the above script should execute. after connecting if i check my routing table by entering route print into a console application we have our routes to azure added correctly. we're done! with that we're now able to fully use an azure point-to-site vpn simply from the command line. this means we can use it as part of a build server deployment, or if you're working on it all the time you can simply set it up to connect every time you login to windows . command line usage rasdial "[connection name]" rasdial "[connection name]" /disconnect for my connection named "azure vpn" this command line usage becomes: rasdial "azure vpn" rasdial "azure vpn" /disconnect

recently i was working on installing and configuring a new instance of jenkins . for some reason, which is out of this post’s context, i wanted to make jenkins run with a specific version of the java environment. fortunately it was something really easy. this post is mainly a reminder to me, next time i’d like to do the same jenkins by default uses the jre which located under the jre sub-directory of your jenkins installation home ( %jenkins_home ). to change this find the file named jenkins.xml in which is located in your %jenkins_home directory. edit it and look for the following section %base%\jre\bin\java now change the content of the executable property to point to your favorite jre. you can describe it as an absolute or relative path or you can even use, environment variables. save the file and restart jenkins. that’s it! enjoy!

One of the breaking changes in Neo4j 2.0.0-RC1 compared to previous versions is that the -[?]-> syntax for matching optional relationships has been retired and replaced with the OPTIONAL MATCH construct. An example where we might want to match an optional relationship could be if we want to find colleagues that we haven’t worked with given the following model: Suppose we have the following data set: CREATE (steve:Person {name: "Steve"}) CREATE (john:Person {name: "John"}) CREATE (david:Person {name: "David"}) CREATE (paul:Person {name: "Paul"}) CREATE (sam:Person {name: "Sam"}) CREATE (londonOffice:Office {name: "London Office"}) CREATE UNIQUE (steve)-[:WORKS_IN]->(londonOffice) CREATE UNIQUE (john)-[:WORKS_IN]->(londonOffice) CREATE UNIQUE (david)-[:WORKS_IN]->(londonOffice) CREATE UNIQUE (paul)-[:WORKS_IN]->(londonOffice) CREATE UNIQUE (sam)-[:WORKS_IN]->(londonOffice) CREATE UNIQUE (steve)-[:COLLEAGUES_WITH]->(john) CREATE UNIQUE (steve)-[:COLLEAGUES_WITH]->(david) We might write the following query to find people from the same office as Steve but that he hasn’t worked with: MATCH (person:Person)-[:WORKS_IN]->(office)<-[:WORKS_IN]-(potentialColleague) WHERE person.name = "Steve" AND office.name = "London Office" WITH person, potentialColleague MATCH (potentialColleague)-[c?:COLLEAGUES_WITH]-(person) WHERE c IS null RETURN potentialColleague ==> +----------------------+ ==> | potentialColleague | ==> +----------------------+ ==> | Node[4]{name:"Paul"} | ==> | Node[5]{name:"Sam"} | ==> +----------------------+ We first find which office Steve works in and find the people who also work in that office. Then we optionally match the ‘COLLEAGUES_WITH’ relationship and only return people who Steve doesn’t have that relationship with. If we run that query in 2.0.0-RC1 we get this exception: ==> SyntaxException: Question mark is no longer used for optional patterns - use OPTIONAL MATCH instead (line 1, column 199) ==> "MATCH (person:Person)-[:WORKS_IN]->(office)<-[:WORKS_IN]-(potentialColleague) WHERE person.name = "Steve" AND office.name = "London Office" WITH person, potentialColleague MATCH (potentialColleague)-[c?:COLLEAGUES_WITH]-(person) WHERE c IS null RETURN potentialColleague" ==> Based on that advice we might translate our query to read like this: MATCH (person:Person)-[:WORKS_IN]->(office)<-[:WORKS_IN]-(potentialColleague) WHERE person.name = "Steve" AND office.name = "London Office" WITH person, potentialColleague OPTIONAL MATCH (potentialColleague)-[c:COLLEAGUES_WITH]-(person) WHERE c IS null RETURN potentialColleague If we run that we get back more people than we’d expect: ==> +------------------------+ ==> | potentialColleague | ==> +------------------------+ ==> | Node[15]{name:"John"} | ==> | Node[14]{name:"David"} | ==> | Node[13]{name:"Paul"} | ==> | Node[12]{name:"Sam"} | ==> +------------------------+ The reason this query doesn’t work as we’d expect is because the WHERE clause immediately following OPTIONAL MATCH is part of the pattern rather than being evaluated afterwards as we’ve become used to. The OPTIONAL MATCH part of the query matches a ‘COLLEAGUES_WITH’ relationship where the relationship is actually null, something of a contradiction! However, since the match is optional a row is still returned. If we include ‘c’ in the RETURN part of the query we can see that this is the case: MATCH (person:Person)-[:WORKS_IN]->(office)<-[:WORKS_IN]-(potentialColleague) WHERE person.name = "Steve" AND office.name = "London Office" WITH person, potentialColleague OPTIONAL MATCH (potentialColleague)-[c:COLLEAGUES_WITH]-(person) WHERE c IS null RETURN potentialColleague, c ==> +---------------------------------+ ==> | potentialColleague | c | ==> +---------------------------------+ ==> | Node[15]{name:"John"} | | ==> | Node[14]{name:"David"} | | ==> | Node[13]{name:"Paul"} | | ==> | Node[12]{name:"Sam"} | | ==> +---------------------------------+ If we take out the WHERE part of the OPTIONAL MATCH the query is a bit closer to what we want: MATCH (person:Person)-[:WORKS_IN]->(office)<-[:WORKS_IN]-(potentialColleague) WHERE person.name = "Steve" AND office.name = "London Office" WITH person, potentialColleague OPTIONAL MATCH (potentialColleague)-[c:COLLEAGUES_WITH]-(person) RETURN potentialColleague, c ==> +-----------------------------------------------+ ==> | potentialColleague | c | ==> +-----------------------------------------------+ ==> | Node[2]{name:"John"} | :COLLEAGUES_WITH[5]{} | ==> | Node[3]{name:"David"} | :COLLEAGUES_WITH[6]{} | ==> | Node[4]{name:"Paul"} | | ==> | Node[5]{name:"Sam"} | | ==> +-----------------------------------------------+ If we introduce a WITH after the OPTIONAL MATCH we can choose to filter out those people that we’ve already worked with: MATCH (person:Person)-[:WORKS_IN]->(office)<-[:WORKS_IN]-(potentialColleague) WHERE person.name = "Steve" AND office.name = "London Office" WITH person, potentialColleague OPTIONAL MATCH (potentialColleague)-[c:COLLEAGUES_WITH]-(person) WITH potentialColleague, c WHERE c IS null RETURN potentialColleague If we evaluate that query it returns the same output as our original query: ==> +----------------------+ ==> | potentialColleague | ==> +----------------------+ ==> | Node[4]{name:"Paul"} | ==> | Node[5]{name:"Sam"} | ==> +----------------------+

Since Groovy 2.2 we can subtract a part of a String value using a regular expression pattern. The first match found is replaced with an empty String. In the following sample code we see how the first match of the pattern is removed from the String: // Define regex pattern to find words starting with gr (case-insensitive). def wordStartsWithGr = ~/(?i)\s+Gr\w+/ assert ('Hello Groovy world!' - wordStartsWithGr) == 'Hello world!' assert ('Hi Grails users' - wordStartsWithGr) == 'Hi users' // Remove first match of a word with 5 characters. assert ('Remove first match of 5 letter word' - ~/\b\w{5}\b/) == 'Remove match of 5 letter word' // Remove first found numbers followed by a whitespace character. assert ('Line contains 20 characters' - ~/\d+\s+/) == 'Line contains characters' Code written with Groovy 2.2.

How do you create a range from 1 to 10 in SQL? Have you ever thought about it? This is such an easy problem to solve in any imperative language, it’s ridiculous. Take Java (or C, whatever) for instance: for (int i = 1; i <= 10; i++) System.out.println(i); This was easy, right? Things even look more lean when using functional programming. Take Scala, for instance: (1 to 10) foreach { t => println(t) } We could fill about 25 pages about various ways to do the above in Scala, agreeing on how awesome Scala is (or what hipsters we are). But how to create a range in SQL? … And we’ll exclude using stored procedures, because that would be no fun. In SQL, the data source we’re operating on are tables. If we want a range from 1 to 10, we’d probably need a table containing exactly those ten values. Here are a couple of good, bad, and ugly options of doing precisely that in SQL. OK, they’re mostly bad and ugly. By creating a table The dumbest way to do this would be to create an actual temporary table just for that purpose: CREATE TABLE "1 to 10" AS SELECT 1 value FROM DUAL UNION ALL SELECT 2 FROM DUAL UNION ALL SELECT 3 FROM DUAL UNION ALL SELECT 4 FROM DUAL UNION ALL SELECT 5 FROM DUAL UNION ALL SELECT 6 FROM DUAL UNION ALL SELECT 7 FROM DUAL UNION ALL SELECT 8 FROM DUAL UNION ALL SELECT 9 FROM DUAL UNION ALL SELECT 10 FROM DUAL See also this SQLFiddle This table can then be used in any type of select. Now that’s pretty dumb but straightforward, right? I mean, how many actual records are you going to put in there? By using a VALUES() table constructor This solution isn’t that much better. You can create a derived table and manually add the values from 1 to 10 to that derived table using the VALUES() table constructor. In SQL Server, you could write: SELECT V FROM ( VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10) ) [1 to 10](V) See also this SQLFiddle By creating enough self-joins of a sufficent number of values Another “dumb”, yet a bit more generic solution would be to create only a certain amount of constant values in a table, view or CTE (e.g. two) and then self join that table enough times to reach the desired range length (e.g. four times). The following example will produce values from 1 to 10, “easily”: WITH T(V) AS ( SELECT 0 FROM DUAL UNION ALL SELECT 1 FROM DUAL ) SELECT V FROM ( SELECT 1 + T1.V + 2 * T2.V + 4 * T3.V + 8 * T4.V V FROM T T1, T T2, T T3, T T4 ) WHERE V <= 10 ORDER BY V See also this SQLFiddle By using grouping sets Another way to generate large tables is by using grouping sets, or more specifically by using the CUBE() function. This works much in a similar way as the previous example when self-joining a table with two records: SELECT ROWNUM FROM ( SELECT 1 FROM DUAL GROUP BY CUBE(1, 2, 3, 4) ) WHERE ROWNUM <= 10 See also this SQLFiddle By just taking random records from a “large enough” table In Oracle, you could probably use ALL_OBJECTs. If you’re only counting to 10, you’ll certainly get enough results from that table: SELECT ROWNUM FROM ALL_OBJECTS WHERE ROWNUM <= 10 See also this SQLFiddle What’s so “awesome” about this solution is that you can cross join that table several times to be sure to get enough values: SELECT ROWNUM FROM ALL_OBJECTS, ALL_OBJECTS, ALL_OBJECTS, ALL_OBJECTS WHERE ROWNUM <= 10 OK. Just kidding. Don’t actually do that. Or if you do, don’t blame me if your productive system runs low on memory. By using the awesome PostgreSQL GENERATE_SERIES() function Incredibly, this isn’t part of the SQL standard. Neither is it available in most databases but PostgreSQL, which has the GENERATE_SERIES() function. This is much like Scala’s range notation: (1 to 10) SELECT * FROM GENERATE_SERIES(1, 10) See also this SQLFiddle By using CONNECT BY If you’re using Oracle, then there’s a really easy way to create such a table using the CONNECT BY clause, which is almost as convenient as PostgreSQL’s GENERATE_SERIES() function: SELECT LEVEL FROM DUAL CONNECT BY LEVEL < 10 See also this SQLFiddle By using a recursive CTE Recursive common table expressions are cool, yet utterly unreadable. the equivalent of the above Oracle CONNECT BY clause when written using a recursive CTE would look like this: WITH "1 to 10"(V) AS ( SELECT 1 FROM DUAL UNION ALL SELECT V + 1 FROM "1 to 10" WHERE V < 10 ) SELECT * FROM "1 to 10" See also this SQLFiddle By using Oracle’s MODEL clause A decent “best of” comparison of how to do things in SQL wouldn’t be complete without at least one example using Oracle’s MODEL clause (see this awesome use-case for Oracle’s spreadsheet feature). Use this clause only to make your co workers really angry when maintaining your SQL code. Bow before this beauty! SELECT V FROM ( SELECT 1 V FROM DUAL ) T MODEL DIMENSION BY (ROWNUM R) MEASURES (V) RULES ITERATE (10) ( V[ITERATION_NUMBER] = CV(R) + 1 ) ORDER BY 1 See also this SQLFiddle Conclusion There aren’t actually many nice solutions to do such a simple thing in SQL. Clearly, PostgreSQL’s GENERATE_SERIES() table function is the most beautiful solution. Oracle’s CONNECT BY clause comes close. For all other databases, some trickery has to be applied in one way or another. Unfortunately.

Unfortunately, I have seen it misused several times when a better design would be the solution.

At the Graph Database meet up in Antwerp, we discussed how you would model a hyper edge in a property graph like Neo4j, and I realized that I’d done this in my football graph without realizing. A hyper edge is defined as follows: A hyperedge is a connection between two or more vertices, or nodes, of a hypergraph. A hypergraph is a graph in which generalized edges (called hyperedges) may connect more than two nodes with discrete properties. In Neo4j, an edge (or relationship) can only be between itself or another node; there’s no way of creating a relationship between more than 2 nodes. I had problems when trying to model the relationship between a player and a football match because I wanted to say that a player participated in a match and represented a specific team in that match. I started out with the following model: Unfortunately, creating a direct relationship from the player to the match means that there’s no way to work out which team they played for. This information is useful because sometimes players transfer teams in the middle of a season and we want to analyze how they performed for each team. In a property graph, we need to introduce an extra node which links the match, player and team together: Although we are forced to adopt this design it actually helps us realize an extra entity in our domain which wasn’t visible before – a player’s performance in a match. If we want to capture information about a players’ performance in a match we can store it on this node. We can also easily aggregate players stats by following the played relationship without needing to worry about the matches they played in. The Neo4j manual has a few more examples of domain models containing hyper edges which are worth having a look at if you want to learn more.

The lack of Base64 encoding API in Java is, in my opinion, by far one of the most annoying holes in the libraries. Finally Java 8 includes a decent API for it in the java.util package. Here is a short introduction of this new API (apparently it has a little more than the regular encode/decode API). The java.util.Base64 Utility Class The entry point to Java 8's Base64 support is the java.util.Base64 class. It provides a set of static factory methods used to obtain one of the three Base64 encodes and decoders: Basic URL MIME Basic Encoding The standard encoding we all think about when we deal with Base64: no line feeds are added to the output and the output is mapped to characters in the Base64 Alphabet: A-Za-z0-9+/ (we see in a minute why is it important). Here is a sample usage: // Encode String asB64 = Base64.getEncoder().encodeToString("some string".getBytes("utf-8")); System.out.println(asB64); // Output will be: c29tZSBzdHJpbmc= // Decode byte[] asBytes = Base64.getDecoder().decode("c29tZSBzdHJpbmc="); System.out.println(new String(asBytes, "utf-8")); // And the output is: some string It can't get simpler than that and, unlike in earlier versions of Java, there is any need for external dependencies (commons-codec), Sun classes (sun.misc.BASE64Decoder) or JAXB's DatatypeConverter (Java 6 and above). However it gets even better. URL Encoding Most of us are used to get annoyed when we have to encode something to be later included in a URL or as a filename - the problem is that the Base64 Alphabet contains meaningful characters in both URIs and filesystems (most specifically the forward slash (/)). The second type of encoder uses the alternative "URL and Filename safe" Alphabet which includes -_ (minus and underline) instead of +/. See the following example: String basicEncoded = Base64.getEncoder().encodeToString("subjects?abcd".getBytes("utf-8")); System.out.println("Using Basic Alphabet: " + basicEncoded); String urlEncoded = Base64.getUrlEncoder().encodeToString("subjects?abcd".getBytes("utf-8")); System.out.println("Using URL Alphabet: " + urlEncoded); The output will be: Using Basic Alphabet: c3ViamVjdHM/YWJjZA== Using URL Alphabet: c3ViamVjdHM_YWJjZA== The sample above illustrates some content which if encoded using a basic encoder will result in a string containing a forward slash while when using a URL safe encoder the output will include an underscore instead (URL Safe encoding is described in clause 5 of the RFC) MIME Encoding The MIME encoder generates a Base64 encoded output using the Basic Alphabet but in a MIME friendly format: each line of the output is no longer than 76 characters and ends with a carriage return followed by a linefeed (\r\n). The following example generates a block of text (this is needed just to make sure we have enough 'body' to encode into more than 76 characters) and encodes it using the MIME encoder: StringBuilder sb = new StringBuilder(); for (int t = 0; t < 10; ++t) { sb.append(UUID.randomUUID().toString()); } byte[] toEncode = sb.toString().getBytes("utf-8"); String mimeEncoded = Base64.getMimeEncoder().encodeToString(toEncode); System.out.println(mimeEncoded); The output: NDU5ZTFkNDEtMDVlNy00MDFiLTk3YjgtMWRlMmRkMWEzMzc5YTJkZmEzY2YtM2Y2My00Y2Q4LTk5 ZmYtMTU1NzY0MWM5Zjk4ODA5ZjVjOGUtOGMxNi00ZmVjLTgyZjctNmVjYTU5MTAxZWUyNjQ1MjJj NDMtYzA0MC00MjExLTk0NWMtYmFiZGRlNDk5OTZhMDMxZGE5ZTYtZWVhYS00OGFmLTlhMjgtMDM1 ZjAyY2QxNDUyOWZiMjI3NDctNmI3OC00YjgyLThiZGQtM2MyY2E3ZGNjYmIxOTQ1MDVkOGQtMzIz Yi00MDg0LWE0ZmItYzkwMGEzNDUxZTIwOTllZTJiYjctMWI3MS00YmQzLTgyYjUtZGRmYmYxNDA4 Mjg3YTMxZjMxZmMtYTdmYy00YzMyLTkyNzktZTc2ZDc5ZWU4N2M5ZDU1NmQ4NWYtMDkwOC00YjIy LWIwYWItMzJiYmZmM2M0OTBm Wrapping All encoders and decoders created by the java.util.Base64 class support streams wrapping - this is a very elegant construct - both coding and efficiency wise (since no redundant buffering are needed) - to stream in and out buffers via encoders and decoders. The sample bellow illustrates how a FileOutputStream can be wrapped with an encoder and a FileInputStream is wrapped with a decoder - in both cases there is no need to buffer the content: public void wrapping() throws IOException { String src = "This is the content of any resource read from somewhere" + " into a stream. This can be text, image, video or any other stream."; // An encoder wraps an OutputStream. The content of /tmp/buff-base64.txt will be the // Base64 encoded form of src. try (OutputStream os = Base64.getEncoder().wrap(new FileOutputStream("/tmp/buff-base64.txt"))) { os.write(src.getBytes("utf-8")); } // The section bellow illustrates a wrapping of an InputStream and decoding it as the stream // is being consumed. There is no need to buffer the content of the file just for decoding it. try (InputStream is = Base64.getDecoder().wrap(new FileInputStream("/tmp/buff-base64.txt"))) { int len; byte[] bytes = new byte[100]; while ((len = is.read(bytes)) != -1) { System.out.print(new String(bytes, 0, len, "utf-8")); } } }

In the last post I wrote about Nathan and my attempts at the Kaggle Titanic Problem, I mentioned that our next step was to try out scikit-learn, so I thought I should summarize where we’ve got up to. We needed to write a classification algorithm to work out whether a person onboard the Titanic survived, and luckily, scikit-learn has extensive documentation on each of the algorithms. Unfortunately almost all those examples use numpy data structures and we’d loaded the data using pandas and didn’t particularly want to switch back! Luckily it was really easy to get the data into numpy format by calling ‘values’ on the pandas data structure, something we learnt from a reply on Stack Overflow. For example if we were to wire up an ExtraTreesClassifier which worked out survival rate based on the ‘Fare’ and ‘Pclass’ attributes we could write the following code: import pandas as pd from sklearn.ensemble import ExtraTreesClassifier from sklearn.cross_validation import cross_val_score train_df = pd.read_csv("train.csv") et = ExtraTreesClassifier(n_estimators=100, max_depth=None, min_samples_split=1, random_state=0) columns = ["Fare", "Pclass"] labels = train_df["Survived"].values features = train_df[list(columns)].values et_score = cross_val_score(et, features, labels, n_jobs=-1).mean() print("{0} -> ET: {1})".format(columns, et_score)) To start with with read in the CSV file which looks like this: $ head -n5 train.csv PassengerId,Survived,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked 1,0,3,"Braund, Mr. Owen Harris",male,22,1,0,A/5 21171,7.25,,S 2,1,1,"Cumings, Mrs. John Bradley (Florence Briggs Thayer)",female,38,1,0,PC 17599,71.2833,C85,C 3,1,3,"Heikkinen, Miss. Laina",female,26,0,0,STON/O2. 3101282,7.925,,S 4,1,1,"Futrelle, Mrs. Jacques Heath (Lily May Peel)",female,35,1,0,113803,53.1,C123,S Next we create our classifier which “fits a number of randomized decision trees (a.k.a. extra-trees) on various sub-samples of the dataset and use averaging to improve the predictive accuracy and control over-fitting.“ i.e. a better version of a random forest. On the next line we describe the features we want the classifier to use, then we convert the labels and features into numpy format so we can pass the to the classifier. Finally we call the cross_val_score function which splits our training data set into training and test components and trains the classifier against the former and checks its accuracy using the latter. If we run this code we’ll get roughly the following output: $ python et.py ['Fare', 'Pclass'] -> ET: 0.687991021324) This is actually a worse accuracy than we’d get by saying that females survived and males didn’t. We can introduce ‘Sex’ into the classifier by adding it to the list of columns: columns = ["Fare", "Pclass", "Sex"] If we re-run the code we’ll get the following error: $ python et.py An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (514, 0)) ... Traceback (most recent call last): File "et.py", line 14, in et_score = cross_val_score(et, features, labels, n_jobs=-1).mean() File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/cross_validation.py", line 1152, in cross_val_score for train, test in cv) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/externals/joblib/parallel.py", line 519, in __call__ self.retrieve() File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/externals/joblib/parallel.py", line 450, in retrieve raise exception_type(report) sklearn.externals.joblib.my_exceptions.JoblibValueError/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/externals/joblib/my_exceptions.py:26: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message, : JoblibValueError ___________________________________________________________________________ Multiprocessing exception: ... ValueError: could not convert string to float: male ___________________________________________________________________________ This is a slightly verbose way of telling us that we can’t pass non numeric features to the classifier – in this case ‘Sex’ has the values ‘female’ and ‘male’. We’ll need to write a function to replace those values with numeric equivalents. train_df["Sex"] = train_df["Sex"].apply(lambda sex: 0 if sex == "male" else 1) Now if we re-run the classifier we’ll get a slightly more accurate prediction: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) The next step is to use the classifier against the test data set, so let’s load the data and run the prediction: test_df = pd.read_csv("test.csv") et.fit(features, labels) et.predict(test_df[columns].values) Now if we run that: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) Traceback (most recent call last): File "et.py", line 22, in et.predict(test_df[columns].values) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 444, in predict proba = self.predict_proba(X) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 479, in predict_proba X = array2d(X, dtype=DTYPE) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/utils/validation.py", line 91, in array2d X_2d = np.asarray(np.atleast_2d(X), dtype=dtype, order=order) File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/numeric.py", line 235, in asarray return array(a, dtype, copy=False, order=order) ValueError: could not convert string to float: male which is the same problem we had earlier! We need to replace the ‘male’ and ‘female’ values in the test set too so we’ll pull out a function to do that now. def replace_non_numeric(df): df["Sex"] = df["Sex"].apply(lambda sex: 0 if sex == "male" else 1) return df Now we’ll call that function with our training and test data frames: train_df = replace_non_numeric(pd.read_csv("train.csv")) ... test_df = replace_non_numeric(pd.read_csv("test.csv")) If we run the program again: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) Traceback (most recent call last): File "et.py", line 26, in et.predict(test_df[columns].values) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 444, in predict proba = self.predict_proba(X) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/ensemble/forest.py", line 479, in predict_proba X = array2d(X, dtype=DTYPE) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/utils/validation.py", line 93, in array2d _assert_all_finite(X_2d) File "/Library/Python/2.7/site-packages/scikit_learn-0.14.1-py2.7-macosx-10.8-intel.egg/sklearn/utils/validation.py", line 27, in _assert_all_finite raise ValueError("Array contains NaN or infinity.") ValueError: Array contains NaN or infinity. There are missing values in the test set so we’ll replace those with average values from our training set using an Imputer: from sklearn.preprocessing import Imputer imp = Imputer(missing_values='NaN', strategy='mean', axis=0) imp.fit(features) test_df = replace_non_numeric(pd.read_csv("test.csv")) et.fit(features, labels) print et.predict(imp.transform(test_df[columns].values)) If we run that it completes successfully: $ python et.py ['Fare', 'Pclass', 'Sex'] -> ET: 0.813692480359) [0 1 0 0 1 0 0 1 1 0 0 0 1 0 1 1 0 0 1 1 0 0 1 0 1 0 1 0 1 0 0 0 1 0 1 0 0 0 0 1 0 0 0 1 1 0 0 0 1 1 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 1 1 0 0 1 1 0 1 0 1 0 0 1 0 1 1 0 0 0 0 0 1 0 1 0 1 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 1 1 1 1 0 1 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 1 1 1 1 0 0 1 0 0 1 0 0 0 0 0 0 1 1 1 1 1 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 0 1 0 1 0 0 0 0 1 0 0 1 0 1 0 1 0 1 0 1 0 0 1 0 0 0 1 0 0 0 0 1 0 1 1 1 1 1 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 1 0 1 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 1 1 1 0 1 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0 1 1 0 0 0 1 0 1 0 0 1 0 1 1 0 1 0 0 0 1 1 0 1 0 0 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 1 1 0 0 0 1 0 1 0 0 1 0 1 0 0 1 0 0 1 1 1 1 0 0 1 0 0 0] The final step is to add these values to our test data frame and then write that to a file so we can submit it to Kaggle. The type of those values is ‘numpy.ndarray’ which we can convert to a pandas Series quite easily: predictions = et.predict(imp.transform(test_df[columns].values)) test_df["Survived"] = pd.Series(predictions) We can then write the ‘PassengerId’ and ‘Survived’ columns to a file: test_df.to_csv("foo.csv", cols=['PassengerId', 'Survived'], index=False) Then output file looks like this: $ head -n5 foo.csv PassengerId,Survived 892,0 893,1 894,0 The code we’ve written is on github in case it’s useful to anyone.

Introduction I had an interesting conversation the other day about custom domain-specific languages and we happened to talk about a feature of Spring that I’ve used before but doesn’t seem to be widely known: the static application context. This post illustrates a basic example I wrote that introduces the static application context and shows how it might be useful. It’s also an interesting topic as it shows some of the well-architected internals of the Spring framework. Most uses of Spring start with XML or annotations and wind up with an application context instance. Behind the scenes, Spring has been working hard to instantiate objects, inject properties, invoke context aware listeners, and so forth. There are a set of classes internal to Spring to help this process along, as Spring needs to hold all of the configuration data about beans before any beans are instantiated. (This is because the beans may be defined in any order, and Spring doesn’t have the exhaustive set of dependencies until all beans are defined.) Spring Static Application Context Spring offers a class called StaticApplicationContext that gives programmatic access from Java to this whole configuration and registration process. This means we can define an entire application context from pure Java code, without using XML or Java annotations or any other tricks. The Javadoc for StaticApplicationContext is here, but an example is coming. Why might we use this? As the Javadoc says, it’s mainly useful for testing. Spring uses it for its own testing, but I’ve found it useful for testing applications that use Spring or other dependency management frameworks. Often, for unit testing, we want to inject different objects into a class from those used in production (e.g. mock objects, or objects that simulate remote invocation, database, or messaging). Of course, we can just keep a separate Spring XML configuration file for testing, but it’s very nice to have our whole configuration right there in the Java unit test class as it makes it easier to maintain. Example I’ve added an example to my intro-to-java repository on GitHub. I created aStaticContext class that provides a very basic Java domain-specific language (DSL) for Spring beans. This is just to make it easier to use from the unit test. The DSL only includes the most basic Spring capabilities: register a bean, set properties, and wire dependencies. package org.anvard.introtojava.spring; import org.springframework.beans.MutablePropertyValues; import org.springframework.beans.factory.config.ConstructorArgumentValues; import org.springframework.beans.factory.config.RuntimeBeanReference; import org.springframework.beans.factory.support.RootBeanDefinition; import org.springframework.context.ApplicationContext; import org.springframework.context.support.StaticApplicationContext; public class StaticContext { public class BeanContext { private String name; private Class beanClass; private ConstructorArgumentValues args; private MutablePropertyValues props; private BeanContext(String name, Class beanClass) { this.name = name; this.beanClass = beanClass; this.args = new ConstructorArgumentValues(); this.props = new MutablePropertyValues(); } public BeanContext arg(Object arg) { args.addGenericArgumentValue(arg); return this; } public BeanContext arg(int index, Object arg) { args.addIndexedArgumentValue(index, arg); return this; } public BeanContext prop(String name, Object value) { props.add(name, value); return this; } public BeanContext ref(String name, String beanRef) { props.add(name, new RuntimeBeanReference(beanRef)); return this; } public void build() { RootBeanDefinition def = new RootBeanDefinition(beanClass, args, props); ctx.registerBeanDefinition(name, def); } } private StaticApplicationContext ctx; private StaticContext() { this.ctx = new StaticApplicationContext(); } public static StaticContext create() { return new StaticContext(); } public ApplicationContext build() { ctx.refresh(); return ctx; } public BeanContext bean(String name, Class beanClass) { return new BeanContext(name, beanClass); } } This class uses several classes that are normally internal to Spring: StaticApplicationContext: Holds bean definitions and provides regular Java methods for registering beans. ConstructorArgumentValues: A smart list for a bean’s constructor arguments. Can hold both wire-by-type and indexed constructor arguments. MutablePropertyValues: A smart list for a bean’s properties. Can hold regular objects and references to other Spring beans. RuntimeBeanReference: A reference by name to a bean in the context. Used for wiring beans together because it allows Spring to delay resolution of a dependency until it’s been instantiated. The StaticContext class uses the builder pattern and provides for method chaining. This makes for cleaner use from our unit test code. Here’s the simplest example: @Test public void basicBean() { StaticContext sc = create(); sc.bean("basic", InnerBean.class).prop("prop1", "abc"). prop("prop2", "def").build(); ApplicationContext ctx = sc.build(); assertNotNull(ctx); InnerBean bean = (InnerBean) ctx.getBean("basic"); assertNotNull(bean); assertEquals("abc", bean.getProp1()); assertEquals("def", bean.getProp2()); } A slightly more realistic example that includes wiring beans together is not much more complicated: @Test public void innerBean() { StaticContext sc = create(); sc.bean("outer", OuterBean.class).prop("prop1", "xyz"). ref("inner", "inner").build(); sc.bean("inner", InnerBean.class).prop("prop1", "ghi"). prop("prop2", "jkl").build(); ApplicationContext ctx = sc.build(); assertNotNull(ctx); InnerBean inner = (InnerBean) ctx.getBean("inner"); assertNotNull(inner); assertEquals("ghi", inner.getProp1()); assertEquals("jkl", inner.getProp2()); OuterBean outer = (OuterBean) ctx.getBean(OuterBean.class); assertNotNull(outer); assertEquals("xyz", outer.getProp1()); assertEquals(inner, outer.getInner()); } Note that once we build the context, we can use it like any other Spring application context, including fetching beans by name or type. Also note that the two contexts we created here are completely separate, which is important for unit testing. Conclusion Much like my post on custom Spring XML, the static application context is a specialty feature that isn’t intended for everyday users of Spring. But I’ve found it convenient when unit testing and it provides an interesting peek into how Spring works.

It is a very rare usecase, but sometimes you need it. How to embed Maven in your application, so that you can programatically run goals? Short answer is: it's tricky. I dabbled into the matter for my java webapp automatic syncing project, and at some point I decided not to embed it. Ultimately, I used a library that does what I needed, but anyway, here are the steps and tools that might be helpful. What you usually need the embedded maven for, is to execute some goals on a maven project. There are two scenarios. The first one is, if you are running inside the maven container, i.e. you are writing a mojo/plugin. Then it's fairly easy, because you have everything managed by the already-initialized plexus container. In that case you can use the mojo-executor. Easy to use, but expects a "project", "pluginManager" and "session", which you can't easily obtain. The second scenario is completely embedded maven. There is a library that does what I needed it to do (thanks to MariuszS for pointing it out) - it's Maven Embedder. Its usage is described in this SO question. Use both the first and the second answer. Before finding that library, I tried two more libraries: the jenkins maven embedded and the Maven Invoker. The problem in both libraries is: they need a maven home. That is, the path to where a maven installation resides. Which is kind of contrary to the idea of "embedded" maven. If the Maven Embedder suits you, you can stop reading. However, there might be cases where the Maven Embedder might not be what you are looking for. In that case, you should use one of the two aforementioned libraries. So, how to find and set a maven home? Ask the user to specify it. Not too much of a hassle, probably Use M2_HOME. One of the libraries uses that by default, but the problem is it might not be set. I don't usually set it, for example. If it is not, you can fallback to the previous approach Scan the entire file system for a maven installation - sounds ok, and it can be done only once, and then stored in some entry. The problem is - there might not be a maven installation. Even if it's a developer's machine - IDEs (Eclipse, at least) have an "embedded" maven. And while it probably stores it somewhere internally in the same format a manual installation would, it may change it's path or structure depending on the version. You can, of course, re-scan the file tree every once in a while to find such an installation Download Maven programatically yourself. Then you can be sure where it is located and that it will always be located there in the same format. The problem here is version mismatch - the user might be using another version of maven. Making the version configurable is an option. All of these work in some cases, and don't work in others. So, in order of preference: 1. make sure you really need to embed maven 2. use the Maven Embedder 3. use another option with its considerations

we all know junit test-classes can be parameterized , which means that for a given set of test-elements, the test class is instantiated a few times, but using constructors for that isn’t always what you want. i’ve taken the stringsorttest from this blog as an example. @runwith(parameterized.class) public class stringsorttest { @parameters public static collection data() { return arrays.aslist(new object[][] { { "abc", "abc" }, { "cba", "abc" }, }); } private final string input; private final string expected; public stringsorttest(final string input, final string expected) { this.input = input; this.expected = expected; } @test public void testsort() { assertequals(expected, mysortmethod(input)); } } this is pretty darn obnoxious sometimes if you have multiple sets of data for various tests, which all go through the constructor, which would force you to write multiple test classes. testng solves this better by allowing you to provide separate data sets to individual test methods using the @dataprovider annotation . but don’t worry, now you can achieve the same with the junit-dataprovider , available on github. pull in the dependency with e.g. maven. com.tngtech.java junit-dataprovider 1.5.0 test the above example now could be rewritten as: @runwith(dataproviderrunner.class) public class stringsorttest { @dataprovider public static object[][] data() { return new object[][] { { "abc", "abc" }, { "cba", "abc" }, }; } @test @usedataprovider("data") public void testsort(final string input, final string expected) { assertequals(expected, mysortmethod(input)); } } you’ll see: no constructor. in this example it doesn’t have many benefits, except maybe for less boiler-plate, but now you can create as many @dataprovider-annotated methods which are fed directly to your @usedataprovider-annotated testmethod(s). sidenote for eclipse: if you’re using that ide, the junit plugin is unable to map the names of the passed/failed testmethods to the ones in the testclass correctly. if a method fails, you’ll have to find it back manually in the testclass (or vote for the patch andreas smidt created to get this fixed in eclipse). in the mean time, if you’re stuck with junit and you’d love to use this feature you’re so accustomed to using with testng, go ahead and try the junit-dataprovider now.

I came across a debate about whether naming a class [Something]Manager is proper. Mainly, I came across this article, but also found other references here and here. I already had my own thoughts on this topic and decided I would share them with you. I started thinking on this a while ago when working in a Swing-powered desktop app. We decided we would make use of the Listeners model that Swing establishes for its components. To register a listener in a Swing component you say: // Some-Type = {Action, Key, Mouse, Focus, ...} component.add[Some-Type]Listener(listener); That's not a problem until you need to dynamically remove those listeners, or override them (remove and add a new listener that listens to the same event but fires a different action), or do any other 'unnatural' thing with them. This turns out to be a little difficult if you use the interface Swing provides out-of-the-box. For instance, to remove a listener, you need to pass the whole instance you want to remove; you would have to hold the reference to the listener if you intend to remove it some time later. So I decided I would take a step further and try to abstract the mess by creating a Manager class. What you could do with this class was something like this: EventsListenersManager.registerListener( "some unique name", component, listener); EventsListenersManager.overrideListener("some unique name", newListener); EventsListenerManager.disableListener("some unique name"); EventsListenerManager.enableListener("some unique name"); EventsListenerManager.unregisterListener("some unique name"); What we gained here??? We got rid of the explicit specification of [Some-Type] and have a single function for every type of listener; and we defined a unique name for the binding, which will be used as an id to remove, enable/disable, and override listeners; a.k.a. no need to hold references. Obviously, it is now convenient to always use the manager’s interface, since it reduces the need to trick our code and gives us a simple and more readable interface to achieve many things with listeners. But that’s not the only advantage of this manager class. You see, manager classes in general have one big advantage: they create an abstraction of a service and work as a centralized point to add logic of any nature, like security logic or performance logic. In the case of our EventsListenersManager, we put some extra logic to enchance the way we can use listeners by providing an interface that simplifies their use: it is easier to make a switch between two listeners now than it was before. Another thing we could have done was to restrict the number of listeners registered in one component for performance sake (Hint: listeners execution don’t run in parallel). So manager classes seem to be a good thing, but we can’t make a manager for every object we want to control in an application. This would consume time and would make us start thinking in a manager class even when we don’t need it. But we can discover a pattern for the need of defining a manager. I’ve seen them used when the resources they control are expensive, like database connections, media resources, hardware resources, etc; and as I see it, expensiveness can come in many flavors: Listeners are expensive because they may be difficult to play with and can be a bottleneck in performance (we must keep them short and fast). Connections are expensive to construct. Media resources are expensive to load. Hardware resources are expensive because they are scarce. So we create different managers for them: We create a manager for listeners to ease and control their use. We create a connections pool in order to limit and control the number of connections to a database. Games developers create a resources manager to have a single point to access resources and reduce the possibility to load them multiple times. We all know there is a driver for every piece of hardware. That’s their manager. Here we can see multiple variations of manager classes implementations, but they all attempt to solve a single thing: reducing the cost we may incur in when using expensive resources directly. I think people should have this line of thought: make a manager class for every expensive resource you detect in your application. You will be more than thankful when you want to add extra logic and you only have to go to one place. Also, enforce the use of manager objects instead of accessing the resources directly. The next time you are planning to make a manager class, ask yourself: is the 'thing' I want to control expensive in any way???