Let's take a simple example of a thread that periodically does some clean up, but in between sleeps most of the time.

ken fogel is the program coordinator and chairperson of the computer science technology program at dawson college in montreal, canada. he is also a program consultant to and part-time instructor in the computer institute of concordia university 's school of extended learning. he blogs at omniprogrammer.com and tweets @omniprof . his regular columns about netbeans in education are listed here . i am old enough to remember commercial applications that had a console interface. my students have no experience with the console. when java was introduced the era of such applications was over. instead you used awt, then swing and now javafx. for the student starting out in programming these gui frameworks add complexity they may not be ready for. therefore i start my classes using console input and output. input can be divided into two categories. the first is values. the primitive types such as int and double are value types. as i explain it, a value is a something you can perform both mathematical and logical operations on. the second is strings. a string does not have a value in the mathematical sense. instead it has a state. therefore it cannot be used in a mathematical expression. state can be compared and so strings can be used is logical expressions such as determining if two strings are the same. if you are ordering strings then you can also determine if one string is greater or less than the other. java does not have any console input routines for values. to java everything you type is a string. this has meant that for many years every java textbook author provided a console input routine that could convert strings to values. java 1.5 made all these routines redundant with the introduction of the scanner class. i could now teach input without needing to go into detail about wrapper classes or the numberformatexception in a standard approach. /** * ask the user for their weight */ public void inputweight() { scanner sc = new scanner(system.in); system.out.print("please enter your weight: "); int weight = sc.nextint(); system.out.println("weight = " + weight); } there is a problem with this code. if the user enters a string that cannot be converted we get the numberformatexception we want to avoid. scanner provides a solution with its ‘hasnext. . .’ methods. these methods determine if the user input can be converted to the target value. you can reject input and avoid the exception. /** * ask the user for their weight */ public void inputweight() { scanner sc = new scanner(system.in); system.out.print("please enter your weight: "); int weight; // accept input and test to see if it’s an integer if (sc.hasnextint()) { // success, it was an integer so get it weight = sc.nextint(); system.out.println("weight = " + weight); } else { // failure so retrieve the input as a string string bad = sc.next(); // inform the user what went wrong system.out.println(bad + " cannot be converted to an integer"); } } now when you run the program you get the following for a good input. this is what you get for a bad input. i will return to numeric input in another article as you do not want a program to end when the user makes one mistake. character input to make my assignments more interesting i have the students create a menu from which they must choose an action. i like to use letters as the menu choice. the problem is that there is no nextchar() method in scanner. instead you need to use next() which will accept anything that you enter. public void displaymenu() { scanner sc = new scanner(system.in); system.out.println("select an account"); system.out.println("a: savings"); system.out.println("b: checking"); system.out.println("c: exit"); system.out.print("please choose a, b or c: "); string choice = sc.next(); system.out.println("you chose: " + choice); } which could result in the following: the first improvement is to convert the string into a character. to do this the string must only be one character long. rather than checking the length you can just extract the first character in the string using the string class charat method. string choice = sc.next(); char letter = choice.charat(0); system.out.println("you chose: " + letter); this will work and if you enter “a” you will get ‘a’. the problem now is that the case of the character you entered should not matter. this is easy to solve by adding one more method from string into the mix. char letter = choice.touppercase().charat(0); almost perfect but since we are using string input the user can enter any string they want. so “bacon” will result in ‘b’. we also need some way to control the allowable letters. in this menu the choices are a, b and c. the solution is to use the ‘hasnext’ method for ‘next’. not the normal, no parameter version, but the version that takes a pattern. pattern is a polite way of saying regular expression. from https://xkcd.com/208/ just as we did with integer input we first test the string that the user has entered. the regular expression will be the pattern that will be used to determine if the string is correct. this regular expression is probably the simplest one you can write. we want a single character, either upper or lower case and is one of the three allowable characters. here it is: [abcabc] the square brackets indicate that this is a character class expression that can match only a single character. what goes inside the brackets are the allowable characters, in this case a, b, c, a, b or c. since these characters are adjacent in the ascii/unicode table they can also be written as a range. [a-ca-c] putting this all together we get a routine that will only accept a single letter. public void displaymenu() { scanner sc = new scanner(system.in); system.out.println("select an account"); system.out.println("a: savings"); system.out.println("b: checking"); system.out.println("c: exit"); system.out.print("please choose a, b or c: "); char letter; if (sc.hasnext("[a-ca-c]")) { string choice = sc.next(); letter = choice.touppercase().charat(0); system.out.println("you chose: " + letter); } else { string choice = sc.next(); system.out.println(choice + " is not valid input"); } } now if you enter ‘a’: but if you enter ‘aardvark’: in my next article i will look at console input a bit more and i will introduce you to the beethoven test.

In this post, I look at the basics of connecting to a Cassandra database from a Java client. I will use the DataStax Java Client JAR in order to do so.

What is CXF? Apache CXF is an open source services framework. CXF helps you build and develop services using frontend programming APIs, like JAX-WS and JAX-RS. These services can speak a variety of protocols such as SOAP, XML/HTTP, RESTful HTTP, or CORBA and work over a variety of transports such as HTTP, JMS etc. How CXF Works? As you can see here and here, how cxf service calls are processed,most of the functionality in the Apache CXF runtime is implemented by interceptors. Every endpoint created by the Apache CXF runtime has potential interceptor chains for processing messages. The interceptors in the these chains are responsible for transforming messages between the raw data transported across the wire and the Java objects handled by the endpoint’s implementation code. Interceptors in CXF When a CXF client invokes a CXF server, there is an outgoing interceptor chain for the client and an incoming chain for the server. When the server sends the response back to the client, there is an outgoing chain for the server and an incoming one for the client. Additionally, in the case of SOAPFaults, a CXF web service will create a separate outbound error handling chain and the client will create an inbound error handling chain. The interceptors are organized into phases to ensure that processing happens on the proper order.Various phases involved during the Interceptor chains are listed in CXF documentation here. Adding your custom Interceptor involves extending one of the Abstract Intereceptor classes that CXF provides, and providing a phase when that interceptor should be invoked. AbstractPhaseInterceptor class - This abstract class provides implementations for the phase management methods of the PhaseInterceptor interface. The AbstractPhaseInterceptor class also provides a default implementation of the handleFault() method. Developers need to provide an implementation of the handleMessage() method. They can also provide a different implementation for the handleFault() method. The developer-provided implementations can manipulate the message data using the methods provided by the generic org.apache.cxf.message.Message interface. For applications that work with SOAP messages, Apache CXF provides an AbstractSoapInterceptor class. Extending this class provides the handleMessage() method and the handleFault() method with access to the message data as an org.apache.cxf.binding.soap.SoapMessage object. SoapMessage objects have methods for retrieving the SOAP headers, the SOAP envelope, and other SOAP metadata from the message. Below piece of code will show, how we can add a Custom Object as Header to an outgoing request – Spring Configuration - Interceptor :- public class SoapHeaderInterceptor extends AbstractSoapInterceptor { public SoapHeaderInterceptor() { super(Phase.POST_LOGICAL); } @Override public void handleMessage(SoapMessage message) throws Fault { List headers = message.getHeaders(); TestHeader testHeader = new TestHeader(); JAXBElement testHeaders = new ObjectFactory() .createTestHeader(testHeader); try { Header header = new Header(testHeaders.getName(), testHeader, new JAXBDataBinding(TestHeader.class)); headers.add(header); message.put(Header.HEADER_LIST, headers); } catch (JAXBException e) { e.printStackTrace(); } }

Learn more about using the Maven Release plugin on Jenkins, including subversion source control, artifactory, continuous integration, and more.

Learn about string interning, a method of storing only one copy of each distinct string value, which must be immutable.

For doing SSL with SOAP, there’s a few things you need to setup. C:\Program Files (x86)\SmartBear\soapUI-Pro-4.5.1\jre\lib\security Also did it at the main jre at C:\Program Files (x86)\Java\jre7\lib\security keytool -genkey -alias svs -keyalg RSA -keystore keystore.jks -keysize 2048 git config --global core.autocrlf true javax.net.ssl.trustStore=<> javax.net.ssl.trustStorePassword=<> If these properties are not set, the default ones will be picked up from your the default location.[$JAVA_HOME/lib/security/jssecacerts, $JAVA_HOME/lib/security/cacerts] To view the contents of keystore file, use: keytool -list -v -keystore file.keystore -storepass changeit To debug the ssl handshake process and view the certificates, set the VM parameter -Djavax.net.debug=all keytool -genkey -keyalg RSA -alias selfsigned -keystore keystore.jks -storepass changeit -validity 360 -keysize 2048 -Djava.net.preferIPv4Stack=true added to soapui.bat C:\Program Files (x86)\SmartBear\SoapUI-4.6.3\bin -Djavax.net.debug=ssl,trustmanager http://docs.oracle.com/cd/E19509-01/820-3503/ggfgo/index.html http://www.sslshopper.com/article-most-common-java-keytool-keystore-commands.html http://ianso.blogspot.com/2009/12/building-ws-security-enabled-soap.html http://javarevisited.blogspot.com/2012/09/difference-between-truststore-vs-keyStore-Java-SSL.html http://javarevisited.blogspot.com/2012/03/add-list-certficates-java-keystore.html http://www.ibm.com/developerworks/java/library/j-jws17/index.html http://www.coderanch.com/t/223027/Web-Services/java/SOAP-HTTPS-SSL http://ruchirawageesha.blogspot.in/2010/07/how-to-create-clientserver-keystores.html http://stackoverflow.com/questions/11001102/how-to-programmatically-set-the-sslcontext-of-a-jax-ws-client http://busylog.net/ssl-java-keytool-soap-and-eclipse/ http://www.sslshopper.com/article-how-to-create-a-self-signed-certificate-using-java-keytool.html openssl s_client -showcerts -host webservices-cert.storedvalue.com -port 443 keytool -keystore clientkeystore -genkey -alias client wsdl2java.bat -uri my.wsdl -o svsproj -p com.agilemobiledeveloper.service -d xmlbeans -t -ss -ssi -sd -g -ns2p System.setProperty("javax.net.ssl.keyStore", keystore.jks"); System.setProperty("javax.net.ssl.keyStorePassword", "changeit"); System.setProperty("javax.net.ssl.trustStore", "clientkeystore"); System.setProperty("javax.net.ssl.trustStorePassword", "changeit"); setx -m JAVA_HOME "C:\Program Files\Java\jdk1.7.0_51″ setx -m javax.net.ssl.keyStore "keystore.jks"); setx -m javax.net.ssl.keyStorePassword "changeit"); setx -m javax.net.ssl.trustStore "keystore.jks"); setx -m javax.net.ssl.trustStorePassword "passwordislong");

Further exploring what's wrong with Java 8, we turn our discussion to monads.

When the first early access versions of Java 8 were made available, what seemed the most important (r)evolution were lambdas. This is now changing and many developers seem to think now that streams are the most valuable Java 8 feature. And this is because they believe that by changing a single word in their programs (replacing stream with parallelStream) they will make these programs work in parallel. Many Java 8 evangelists have demonstrated amazing examples of this. Is there something wrong with this? No. Not something. Many things: Running in parallel may or may not be a benefit. It depends what you are using this feature for. Java 8 parallel streams may make your programs run faster. Or not. Or even slower. Thinking about streams as a way to achieve parallel processing at low cost will prevent developers to understand what is really happening. Streams are not directly linked to parallel processing. Most of the above problems are based upon a misunderstanding: parallel processing is not the same thing as concurrent processing. And most examples shown about “automatic parallelization” with Java 8 are in fact examples of concurrent processing. Thinking about map, filter and other operations as “internal iteration” is a complete nonsense (although this is not a problem with Java 8, but with the way we use it). So, what are streams According to Wikipedia: “a stream is a potentially infinite analog of a list, given by the inductive definition: data Stream a = Cons a (Stream a) Generating and computing with streams requires lazy evaluation, either implicitly in a lazily evaluated language or by creating and forcing thunks in an eager language.” One most important think to notice is that Java is what Wikipedia calls an “eager” language, which means Java is mostly strict (as opposed to lazy) in evaluating things. For example, if you create a List in Java, all elements are evaluated when the list is created. This may surprise you, since you may create an empty list and add elements after. This is only because either the list is mutable (and you are replacing a null reference with a reference to something) or you are creating a new list from the old one appended with the new element. Lists are created from something producing its elements. For example: List list = Arrays.asList(1, 2, 3, 4, 5); Here the producer is an array, and all elements of the array are strictly evaluated. It is also possible to create a list in a recursive way, for example the list starting with 1 and where all elements are equals to 1 plus the previous element and smaller than 6. In Java < 8, this translates into: List list = new ArrayList(); for(int i = 0; i < 6; i++) { list.add(i); } One may argue that the for loop is one of the rare example of lazy evaluation in Java, but the result is a list in which all elements are evaluated. What happens if we want to apply a function to all elements of this list? We may do this in a loop. For example, if with want to increase all elements by 2, we may do this: for(int i = 0; i < list.size(); i++) { list.set(i, list.get(i) * 2); } However, this does not allow using an operation that changes the type of the elements, for example increasing all elements by 10%. The following solution solves this problem: List list2 = new ArrayList(); for(int i = 0; i < list.size(); i++) { list2.add(list.get(i) * 1.2); } This form allows the use of a the Java 5 for each syntax: List list2 = new ArrayList<>(); for(Integer i : list) { list2.add(i * 1.2); } or the Java 8 syntax: List list2 = new ArrayList<>(); list.forEach(x -> list2.add(x * 1.2)); So far, so good. But what if we want to increase the value by 10% and then divide it by 3? The trivial answer would be to do: List list2 = new ArrayList<>(); list.forEach(x -> list2.add(x * 1.2)); List list3 = new ArrayList<>(); list2.forEach(x -> list3.add(x / 3)); This is far from optimal because we are iterating twice on the list. A much better solution is: List list2 = new ArrayList<>(); for(Integer i : list) { list2.add(i * 1.2 / 3); } Let aside the auto boxing/unboxing problem for now. In Java 8, this can be written as: List list2 = new ArrayList<>(); list.forEach(x -> list2.add(x * 1.2 / 3)); But wait... This is only possible because we see the internals of the Consumer bound to the list, so we are able to manually compose the operations. If we had: List list2 = new ArrayList<>(); list.forEach(consumer1); List list3 = new ArrayList<>(); list2.forEach(consumer2); How could we know how to compose them? No way. In Java 8, the Consumer interface has a default method andThen. We could be tempted to compose the consumers this way: list.forEach(consumer1.andThen(consumer2)); but this will result in an error, because andThen is defined as: default Consumer andThen(Consumer after) { Objects.requireNonNull(after); return (T t) -> { accept(t); after.accept(t); }; } This means that we can't use andThen to compose consumers of different types. In fact, we have it all wrong since the beginning. What we need is to bind the list to a function in order to get a new list, such as: Function function1 = x -> x * 1.2; Function function2 = x -> x / 3; list.bind(function1).bind(function2); where the bind method would be defined in a special FList class like: public class FList { final List list; public FList(List list) { this.list = list; } public FList bind(Function f) { List newList = new ArrayList(); for (T t : list) { newList.add(f.apply(t)); } return new FList(newList); } } and we would use it as in the following example: new Flist<>(list).bind(function1).bind(function2); The only trouble we have then is that binding twice would require iterating twice on the list. This is because bind is evaluated strictly. What we would need is a lazy evaluation, so that we could iterate only once. The problem here is that the bind method is not a real binding. It is in reality a composition of a real binding and a reduce. "Reducing" is applying an operation to each element of the list, resulting in the combination of this element and the result of the same operation applied to the previous element. As there is no previous element when we start from the first element, we start with an initial value. For example, applying (x) -> r + x, where r is the result of the operation on the previous element, or 0 for the first element, gives the sum of all elements of the list. Applying () -> r + 1 to each element, starting with r = 0 gives the length of the list. (This may not be the more efficient way to get the length of the list, but it is totally functional!) Here, the operation is add(element) and the initial value is an empty list. And this occurs only because the function application is strictly evaluated. What Java 8 streams give us is the same, but lazily evaluated, which means that when binding a function to a stream, no iteration is involved! Binding a Function to a Stream gives us a Stream with no iteration occurring. The resulting Stream is not evaluated, and this does not depend upon the fact that the initial stream was built with evaluated or non evaluated data. In functional languages, binding a Function to a Stream is itself a function. In Java 8, it is a method, which means it's arguments are strictly evaluated, but this has nothing to do with the evaluation of the resulting stream. To understand what is happening, we can imagine that the functions to bind are stored somewhere and they become part of the data producer for the new (non evaluated) resulting stream. In Java 8, the method binding a function T -> U to a Stream, resulting in a Stream is called map. The function binding a function T -> Stream to a Stream, resulting in a Stream is called flatMap. Where is flatten? Most functional languages also offer a flatten function converting a Stream> into a Stream, but this is missing in Java 8 streams. It may not look like a big trouble since it is so easy to define a method for doing this. For example, given the following function: Function> f = x -> Stream.iterate(1, y -> y + 1).limit(x); Stream stream = Stream.iterate(1, x -> x + 1); Stream stream2 = stream.limit(5).flatMap(f); System.out.println(stream2.collect(toList())) to produce: [1, 1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5] Using map instead of flatMap: Stream stream = Stream.iterate(1, x -> x + 1); Stream stream2 = stream.limit(5).map(f); System.out.println(stream2.collect(toList())) will produce a stream of streams: [java.util.stream.SliceOps$1@12133b1, java.util.stream.SliceOps$1@ea2f77, java.util.stream.SliceOps$1@1c7353a, java.util.stream.SliceOps$1@1a9515, java.util.stream.SliceOps$1@f49f1c] Converting this stream of streams of integers to a stream of integers is very straightforward using the functional paradigm: one just need to flatMap the identity function to it: System.out.println(stream2.flatMap(x -> x).collect(toList())); It is however strange that a flatten method has not been added to the stream, knowing the strong relation that ties map, flatMap, unit and flatten, where unit is the function from T to Stream, represented by the method: Stream Stream.of(T... t) When are stream evaluated? Streams are evaluated when we apply to them some specific operations called terminal operation. This may be done only once. Once a terminal operation is applied to a stream, is is no longer usable. Terminal operations are: forEach forEachOrdered toArray reduce collect min max count anyMatch allMatch noneMatch findFirst findAny iterator spliterator Some of these methods are short circuiting. For example, findFirst will return as soon as the first element will be found. Non terminal operations are called intermediate and can be stateful (if evaluation of an element depends upon the evaluation of the previous) or stateless. Intermediate operations are: filter map mapTo... (Int, Long or Double) flatMap flatMapTo... (Int, Long or Double) distinct sorted peek limit skip sequential parallel unordered onClose Several intermediate operations may be applied to a stream, but only one terminal operation may be use. So what about parallel processing? One most advertised functionality of streams is that they allow automatic parallelization of processing. And one can find the amazing demonstrations on the web, mainly based of the same example of a program contacting a server to get the values corresponding to a list of stocks and finding the highest one not exceeding a given limit value. Such an example may show an increase of speed of 400 % and more. But this example as little to do with parallel processing. It is an example of concurrent processing, which means that the increase of speed will be observed also on a single processor computer. This is because the main part of each “parallel” task is waiting. Parallel processing is about running at the same time tasks that do no wait, such as intensive calculations. Automatic parallelization will generally not give the expected result for at least two reasons: The increase of speed is highly dependent upon the kind of task and the parallelization strategy. And over all things, the best strategy is dependent upon the type of task. The increase of speed in highly dependent upon the environment. In some environments, it is easy to obtain a decrease of speed by parallelizing. Whatever the kind of tasks to parallelize, the strategy applied by parallel streams will be the same, unless you devise this strategy yourself, which will remove much of the interest of parallel streams. Parallelization requires: A pool of threads to execute the subtasks, Dividing the initial task into subtasks, Distributing subtasks to threads, Collating the results. Without entering the details, all this implies some overhead. It will show amazing results when: Some tasks imply blocking for a long time, such as accessing a remote service, or There are not many threads running at the same time, and in particular no other parallel stream. If all subtasks imply intense calculation, the potential gain is limited by the number of available processors. Java 8 will by default use as many threads as they are processors on the computer, so, for intensive tasks, the result is highly dependent upon what other threads may be doing at the same time. Of course, if each subtask is essentially waiting, the gain may appear to be huge. The worst case is if the application runs in a server or a container alongside other applications, and subtasks do not imply waiting. In such a case, (for example running in a J2EE server), parallel streams will often be slower that serial ones. Imagine a server serving hundreds of requests each second. There are great chances that several streams might be evaluated at the same time, so the work is already parallelized. A new layer of parallelization at the business level will most probably make things slower. Worst: there are great chances that the business applications will see a speed increase in the development environment and a decrease in production. And that is the worst possible situation. Edit: for a better understanding of why parallel streams in Java 8 (and the Fork/Join pool in Java 7) are broken, refer to these excellent articles by Edward Harned: A Java Fork-Join Calamity A Java Parallel Calamity What streams are good for Stream are a useful tool because they allow lazy evaluation. This is very important in several aspect: They allow functional programming style using bindings. They allow for better performance by removing iteration. Iteration occurs with evaluation. With streams, we can bind dozens of functions without iterating. They allow easy parallelization for task including long waits. Streams may be infinite (since they are lazy). Functions may be bound to infinite streams without problem. Upon evaluation, there must be some way to make them finite. This is often done through a short circuiting operation. What streams are not good for Streams should be used with high caution when processing intensive computation tasks. In particular, by default, all streams will use the same ForkJoinPool, configured to use as many threads as there are cores in the computer on which the program is running. If evaluation of one parallel stream results in a very long running task, this may be split into as many long running sub-tasks that will be distributed to each thread in the pool. From there, no other parallel stream can be processed because all threads will be occupied. So, for computation intensive stream evaluation, one should always use a specific ForkJoinPool in order not to block other streams. To do this, one may create a Callable from the stream and submit it to the pool: List list = // A list of objects Stream stream = list.parallelStream().map(this::veryLongProcessing); Callable> task = () -> stream.collect(toList()); ForkJoinPool forkJoinPool = new ForkJoinPool(4); List newList = forkJoinPool.submit(task).get() This way, other parallel streams (using their own ForkJoinPool) will not be blocked by this one. In other words, we would need a pool of ForkJoinPool in order to avoid this problem. If a program is to be run inside a container, one must be very careful when using parallel streams. Never use the default pool in such a situation unless you know for sure that the container can handle it. In a Java EE container, do not use parallel streams. Previous articles What's Wrong with Java 8, Part I: Currying vs Closures What's Wrong in Java 8, Part II: Functions & Primitives

whats this all about then? let’s start with a short story. a few weeks back i proposed a change on the a java core libs mailing list to override some methods which are currently final . this stimulated several discussion topics - one of which was the extent to which a performance regression would be introduced by taking a method which was final and stopping it from being final . i had some ideas about whether there would be a performance regression or not, but i put these aside to try and enquire as to whether there were any sane benchmarks published on the subject. unfortunately i couldn’t find any. that’s not to say that they don’t exist or that other people haven’t investigated the situation, but that i didn't see any public peer-reviewed code. so - time to write some benchmarks. benchmarking methodology so i decided to use the ever-awesome jmh framework in order to put together these benchmarks. if you aren't convinced that a framework will help you get accurate benchmarking results then you should look at this talk by aleksey shipilev , who wrote the framework, or nitsan wakart's really cool blog post which explains how it helps. in my case i wanted to understand what influenced the performance of method invocation. i decided to try out different variations of methods calls and measure the cost. by having a set of benchmarks and changing only one factor at a time, we can individually rule out or understand how different factors or combinations of factors influence method invocation costs. inlining let's squish these method callsites down. simultaneously the most and least obvious influencing factor is whether there is a method call at all! it's possible for the actual cost of a method call to be optimized away entirely by the compiler. there are, broadly speaking, two ways to reduce the cost of the call. one is to directly inline the method itself, the other is to use an inline cache. don't worry - these are pretty simple concepts but there's a bit of terminology involved which needs to be introduced. let's pretend that we have a class called foo , which defines a method called bar . class foo { void bar() { ... } } we can call the bar method by writing code that looks like this: foo foo = new foo(); foo.bar(); the important thing here is the location where bar is actually invoked - foo.bar() - this is referred to as a callsite . when we say a method is being "inlined" what is means is that the body of the method is taken and plopped into the callsite, in place of a method call. for programs which consist of lots of small methods (i'd argue, a properly factored program) the inlining can result in a significant speedup. this is because the program doesn't end up spending most of its time calling methods and not actually doing work! we can control whether a method is inlined or not in jmh by using the compilercontrol annotations. we'll come back to the concept of an inline cache a bit later. hierarchy depth and overriding methods do parents slow their children down? if we're choosing to remove the final keyword from a method it means that we'll be able to override it. this is another factor which we consequently need to take into account. so i took methods and called them at different levels of a class hierarchy and also had methods which were overridden at different levels of the hierarchy. this allowed me to understand or eliminate how deep class hierarchies interfere with overriding costs. polymorphism animals: how any oo concept is described. when i mentioned the idea of a callsite earlier i sneakily avoided a fairly important issue. since it's possible to override a non- final method in a subclass, our callsites can end up invoking different methods. so perhaps i pass in a foo or it's child - baz - which also implements a bar(). how does your compiler know which method to invoke? methods are by default virtual (overridable) in java it has to lookup the correct method in a table, called a vtable, for every invocation. this is pretty slow, so optimizing compilers are always trying to reduce the lookup costs involved. one approach we mentioned earlier is inlining, which is great if your compiler can prove that only one method can be called at a given callsite. this is called a monomorphic callsite. unfortunately much of the time the analysis required to prove a callsite is monomorphic can end up being impractical. jit compilers tend to take an alternative approach of profiling which types are called at a callsite and guessing that if the callsite has been monomorphic for it's first n calls then it's worth speculatively optimising based on the assumption that it always will be monomorphic. this speculative optimisation is frequently correct, but because it's not always right the compiler needs to inject a guard before the method call in order to check the type of the method. monomorphic callsites aren't the only case we want to optimise for though. many callsites are what is termed bimorphic - there are two methods which can be invoked. you can still inline bimorphic callsites by using your guard code to check which implementation to call and then jumping to it. this is still cheaper than a full method invocation. it's also possible to optimise this case using an inline cache. an inline cache doesn't actually inline the method body into a callsite but it has a specialised jump table which acts like a cache on a full vtable lookup. the hotspot jit compiler supports bimorphic inline caches and declares that any callsite with 3 or more possible implementations is megamorphic . this splits out 3 more invocation situations for us to benchmark and investigate: the monomorphic case, the bimorphic case and the megamorphic case. results let's groups up results so it's easier to see the wood from the trees, i've presented the raw numbers along with a bit of analysis around them. the specific numbers/costs aren't really of that much interest. what is interesting is the ratios between different types of method call and that the associated error rates are low. there's quite a significant difference going on - 6.26x between the fastest and slowest. in reality the difference is probably larger because of the overhead associated with measuring the time of an empty method. the source code for these benchmarks is available on github . the results aren't all presented in one block to avoid confusion. the polymorphic benchmarks at the end come from running polymorphicbenchmark , whilst the others are from javafinalbenchmark simple callsites benchmark mode samples mean mean error units c.i.j.javafinalbenchmark.finalinvoke avgt 25 2.606 0.007 ns/op c.i.j.javafinalbenchmark.virtualinvoke avgt 25 2.598 0.008 ns/op c.i.j.javafinalbenchmark.alwaysoverriddenmethod avgt 25 2.609 0.006 ns/op our first set of results compare the call costs of a virtual method, a final method and a method which has a deep hierarchy and gets overridden. note that in all these cases we've forced the compiler to not inline the methods. as we can see the difference between the times is pretty minimal and and our mean error rates show it to be of no great importance. so we can conclude that simply adding the final keyword isn't going to drastically improve method call performance. overriding the method also doesn't seem to make much difference either. inlining simple callsites benchmark mode samples mean mean error units c.i.j.javafinalbenchmark.inlinablefinalinvoke avgt 25 0.782 0.003 ns/op c.i.j.javafinalbenchmark.inlinablevirtualinvoke avgt 25 0.780 0.002 ns/op c.i.j.javafinalbenchmark.inlinablealwaysoverriddenmethod avgt 25 1.393 0.060 ns/op now, we've taken the same three cases and removed the inlining restriction. again the final and virtual method calls end up being of a similar time to each other. they are about 4x faster than the non-inlineable case, which i would put down to the inlining itself. the always overridden method call here ends up being between the two. i suspect that this is because the method itself has multiple possible subclass implementations and consequently the compiler needs to insert a type guard. the mechanics of this are explained above in more detail under polymorphism . class hierachy impact benchmark mode samples mean mean error units c.i.j.javafinalbenchmark.parentmethod1 avgt 25 2.600 0.008 ns/op c.i.j.javafinalbenchmark.parentmethod2 avgt 25 2.596 0.007 ns/op c.i.j.javafinalbenchmark.parentmethod3 avgt 25 2.598 0.006 ns/op c.i.j.javafinalbenchmark.parentmethod4 avgt 25 2.601 0.006 ns/op c.i.j.javafinalbenchmark.inlinableparentmethod1 avgt 25 1.373 0.006 ns/op c.i.j.javafinalbenchmark.inlinableparentmethod2 avgt 25 1.368 0.004 ns/op c.i.j.javafinalbenchmark.inlinableparentmethod3 avgt 25 1.371 0.004 ns/op c.i.j.javafinalbenchmark.inlinableparentmethod4 avgt 25 1.371 0.005 ns/op wow - that's a big block of methods! each of the numbered method calls (1-4) refer to how deep up a class hierarchy a method was invoked upon. so parentmethod4 means we called a method declared on the 4th parent of the class. if you look at the numbers there is very little difference between 1 and 4. so we can conclude that hierarchy depth makes no difference. the inlineable cases all follow the same pattern: hierarchy depth makes no difference. our inlineable method performance is comparable to inlinablealwaysoverriddenmethod , but slower than inlinablevirtualinvoke . i would again put this down to the type guard being used. the jit compiler can profile the methods to figure out only one is inlined, but it can't prove that this holds forever. class hierachy impact on final methods benchmark mode samples mean mean error units c.i.j.javafinalbenchmark.parentfinalmethod1 avgt 25 2.598 0.007 ns/op c.i.j.javafinalbenchmark.parentfinalmethod2 avgt 25 2.596 0.007 ns/op c.i.j.javafinalbenchmark.parentfinalmethod3 avgt 25 2.640 0.135 ns/op c.i.j.javafinalbenchmark.parentfinalmethod4 avgt 25 2.601 0.009 ns/op c.i.j.javafinalbenchmark.inlinableparentfinalmethod1 avgt 25 1.373 0.004 ns/op c.i.j.javafinalbenchmark.inlinableparentfinalmethod2 avgt 25 1.375 0.016 ns/op c.i.j.javafinalbenchmark.inlinableparentfinalmethod3 avgt 25 1.369 0.005 ns/op c.i.j.javafinalbenchmark.inlinableparentfinalmethod4 avgt 25 1.371 0.003 ns/op this follows the same pattern as above - the final keyword seems to make no difference. i would have thought it was possible here, theoretically, for inlinableparentfinalmethod4 to be proved inlineable with no type guard but it doesn't appear to be the case. polymorphism monomorphic: 2.816 +- 0.056 ns/op bimorphic: 3.258 +- 0.195 ns/op megamorphic: 4.896 +- 0.017 ns/op inlinable monomorphic: 1.555 +- 0.007 ns/op inlinable bimorphic: 1.555 +- 0.004 ns/op inlinable megamorphic: 4.278 +- 0.013 ns/op finally we come to the case of polymorphic dispatch. monomorphoric call costs are roughly the same as our regular virtual invoke call costs above. as we need to do lookups on larger vtables, they become slower as the bimorphic and megamorphic cases show. once we enable inlining the type profiling kicks in and our monomorphic and bimorphic callsites come down the cost of our "inlined with guard" method calls. so similar to the class hierarchy cases, just a bit slower. the megamorphic case is still very slow. remember that we've not told hotspot to prevent inlining here, it just doesn't implement polymorphic inline cache for callsites more complex than bimorphic. what did we learn? i think it's worth noting that there are plenty of people who don't have a performance mental model that accounts for different types of method calls taking different amounts of time and plenty of people who understand they take different amounts of time but don't really have it quite right. i know i've been there before and made all sorts of bad assumptions. so i hope this investigation has been helpful to people. here's a summary of claims i'm happy to stand by. there is a big difference between the fastest and slowest types of method invocation. in practice the addition or removal of the final keyword doesn't really impact performance, but, if you then go and refactor your hierarchy things can start to slow down. deeper class hierarchies have no real influence on call performance. monomorphic calls are faster than bimorphic calls. bimorphic calls are faster than megamorphic calls. the type guard that we see in the case of profile-ably, but not provably, monomorphic callsites does slow things down quite a bit over a provably monomorphic callsite. i would say that the cost of the type guard is my personal "big revelation". it's something that i rarely see talked about and often dismissed as being irrelevant. caveats and further work of course this isn't a conclusive treatment of the topic area! this blog has just focussed on type related factors surrounding method invoke performance. one factor i've not mentioned is the heuristics surrounding method inlining due to body size or call stack depth. if your method is too large it won't get inlined at all, and you'll still end up paying for the cost of the method call. yet another reason to write small, easy to read, methods. i've not looked into how invoking over an interface affects any of these situations. if you've found this interesting then there's an investigation of invoke interface performance on the mechanical sympathy blog. one factor that we've completely ignored here is the impact of method inlining on other compiler optimisations. when compilers are performing optimisations which only look at one method (intra-procedural optimisation) they really want as much information as they can get in order to optimize effectively. the limitations of inlining can significantly reduce the scope that other optimisations have to work with. tying the explanation right down to the assembly level to dive into more detail on the issue. perhaps these are topics for a future blog post. thanks to aleksey shipilev for feedback on the benchmarks and to martin thompson , aleksey, martijn verburg , sadiq jaffer and chris west for the very helpful feedback on the blog post.

Java 8 comes with a new Optional type, similar to what is available in other languages. This post will go over how this new type is meant to be used, namely what is it's main use case. What is the Optional type? Optional is a new container type that wraps a single value, if the value is available. So it's meant to convey the meaning that the value might be absent. Take for example this method: public Optional findCustomerWithSSN(String ssn) { ... } Returning Optional adds explicitly the possibility that there might not be a customer for that given social security number. This means that the caller of the method is explicitly forced by the type system to think about and deal with the possibility that there might not be a customer with that SSN. The caller will have to to something like this: Optional optional = findCustomerWithSSN(ssn); if (optional.isPresent()) { Customer customer = maybeCustomer.get(); ... use customer ... } else { ... deal with absence case ... } Or if applicable, provide a default value: Long value = findOptionalLong(ssn).orElse(0L); This use of optional is somewhat similar to the more familiar case of throwing checked exceptions. By throwing a checked exception, we use the compiler to enforce callers of the API to somehow handle an exceptional case. What is Optional trying to solve? Optional is an attempt to reduce the number of null pointer exceptions in Java systems, by adding the possibility to build more expressive APIs that account for the possibility that sometimes return values are missing. If Optional was there since the beginning, most libraries and applications would likely deal better with missing return values, reducing the number of null pointer exceptions and the overall number of bugs in general. What is Optional not trying to solve Optional is not meant to be a mechanism to avoid all types of null pointers. The mandatory input parameters of methods and constructors still have to be tested for example. Like when using null, Optional does not help with conveying the meaning of an absent value. In a similar way that null can mean many different things (value not found, etc.), so can an absent Optional value. The caller of the method will still have to check the javadoc of the method for understanding the meaning of the absent Optional, in order to deal with it properly. Also in a similar way that a checked exception can be caught in an empty block, nothing prevents the caller of calling get() and moving on. What is wrong with just returning null? The problem is that the caller of the function might not have read the javadoc for the method, and forget about handling the null case. This happens frequently and is one of the main causes of null pointer exceptions, although not the only one. How should Optional be used then? Optional should be used mostly as the return type of functions that might not return a value. In the context of domain driver development, this means certain service, repository or utility methods such as the one shown above. How should Optional NOT be used? Optional is not meant to be used in these contexts, as it won't buy us anything: in the domain model layer (not serializable) in DTOs (same reason) in input parameters of methods in constructor parameters How does Optional help with functional programming? In chained function calls, Optional provides method ifPresent(), that allows to chain functions that might not return values: findCustomerWithSSN(ssn).ifPresent(() -> System.out.println("customer exists!")); Useful Links This blog post from Oracle goes further into Optional and it's uses, comparing it with similar functionality in other languages - Tired of Null Pointer Exceptions This cheat sheet provides a thorough overview of Optional - Optional in Java 8 Cheat Sheet

If you’ve read the previous five blogs in this series, you’ll know that I’ve been building a Spring application that runs periodically to check a whole bunch of error logs for exceptions and then email you the results. Having written the code and the tests, and being fairly certain it’ll work the next and final step is to package the whole thing up and deploy it to a production machine. The actual deployment and packaging methods will depend upon your own organisation's processes and procedures. In this example, however, I’m going to choose the simplest way possible to create and deploy an executable JAR file. The first step was completed several weeks ago, and that’s defining our output as a JAR file in the Maven POM file, which, as you’ll probably already know, is done using the packaging element: jar It’s okay having a JAR file, but in this case there’s a further step involved: making it executable. To make a JAR file executable you need to add a MANIFEST.MF file and place it in a directory called META-INF. The manifest file is a file that describes the JAR file to both the JVM and human readers. As usual, there are a couple of ways of doing this, for example if you wanted to make life difficult for yourself, you could hand-craft your own file and place it in the META-INF directory inside the project’s src/main/resources directory. On the other hand, you could use themaven-jar plug-in and do it automatically. To do that, you need to to add the following to your POM file. org.apache.maven.plugins maven-jar-plugin 2.4 true com.captaindebug.errortrack.Main lib/ The interesting point here is the configuration element. It contains three sub-elements: addClasspath: this means that the plug-in will add the classpath to the MANIFEST.MF file so that the JVM can find all the support jars when running the app. mainClass: this tells the plug-in to add a Main-Class attribute to the MANIFEST.MF file, so that the JVM knows where to find the the entry point to the application. In this case it’s com.captaindebug.errortrack.Main classpathPrefix: this is really useful. It allows you to locate all the support jars in a different directory to the main part of the application. In this case I’ve chosen the very simple and short name of lib. If you run the build and then open up the resulting JAR file and extract and examine the /META-INF/MANIFEST.MFfile, you’ll find something rather like this: Manifest-Version: 1.0 Built-By: Roger Build-Jdk: 1.7.0_09 Class-Path: lib/spring-context-3.2.7.RELEASE.jar lib/spring-aop-3.2.7.RELEASE.jar lib/aopalliance-1.0.jar lib/spring-beans-3.2.7.RELEASE.jar lib/spring-core-3.2.7.RELEASE.jar lib/spring-expression-3.2.7.RELEASE.jar lib/slf4j-api-1.6.6.jar lib/slf4j-log4j12-1.6.6.jar lib/log4j-1.2.16.jar lib/guava-13.0.1.jar lib/commons-lang3-3.1.jar lib/commons-logging-1.1.3.jar lib/spring-context-support-3.2.7.RELEASE.jar lib/spring-tx-3.2.7.RELEASE.jar lib/quartz-1.8.6.jar lib/mail-1.4.jar lib/activation-1.1.jar Created-By: Apache Maven 3.0.4 Main-Class: com.captaindebug.errortrack.Main Archiver-Version: Plexus Archiver The last step is to marshall all the support jars into one directory, in this case the lib directory, so that the JVM can find them when you run the application. Again, there are two ways of approaching this: the easy way and the hard way. The hard way involves manually collecting together all the JAR files as defined by the POM (both direct and transient dependencies) and copying them to an output directory. The easy way involves getting the maven-dependency-plugin to do it for you. This involves adding the following to your POM file: org.apache.maven.plugins maven-dependency-plugin 2.5.1 copy-dependencies package copy-dependencies ${project.build.directory}/lib/ In this case you’re using the copy-dependencies goal executed in the package phase to copy all the project dependencies to the${project.build.directory}/lib/ directory - note that the final part of the directory path, lib, matches theclasspathPrefix setting from the previous step. In order to make life easier, I’ve also created a small run script: runme.sh: #!/bin/bash echo Running Error Tracking... java -jar error-track-1.0-SNAPSHOT.jar com.captaindebug.errortrack.Main And that’s about it. The application is just about complete. I’ve copied it to my build machine where it now monitors the Captain Debug Github sample apps and build. I could, and indeed may, add a few more features to the app. There are a few rough edges that need knocking off the code: for example is it best to run it as a separate app, or would it be a better idea to turn it into a web app? Furthermore, wouldn’t it be a good idea to ensure that the same errors aren’t reported twice? I may get around to thart soon... or maybe I'll talk about something else; so much to blog about so little time... The code for this blog is available on Github at: https://github.com/roghughe/captaindebug/tree/master/error-track. If you want to look at other blogs in this series take a look here… Tracking Application Exceptions With Spring Tracking Exceptions With Spring - Part 2 - Delegate Pattern Error Tracking Reports - Part 3 - Strategy and Package Private Tracking Exceptions - Part 4 - Spring's Mail Sender Tracking Exceptions - Part 5 - Scheduling With Spring

Cloudfoundry's java buildpack is supporting some popular jvm based applications. This article is oriented to the audiences already with experience of cloudfoundry/heroku buildpack who want to have more understanding of how buildpack and cloudfoundry works internally. cf push app -p app.war -b build-pack-url The above command demonstrates the usage of pushing a war file to cloudfoundry by using a custom buildpack (E.g. https://github.com/cloudfoundry/java-buildpack). However, what exactly happens inside, or how cloudfoundry bootstrap the war file with tomcat? There are three contracts phase that bridge communication between buildpack and cloudfoundry. The three phases are detect, compile and release, which are three ruby shell scripts: Java buildpack has multiple sub components, while each of them has all of these three phases (E.g. tomcat is one of the sub components, while it contained another layer of sub components). Detect Phase: detect phase is to check whether a particular buildpack/component applies to the deployed application. Take the war file example, tomcat applies only when https://github.com/cloudfoundry/java-buildpack/blob/master/lib/java_buildpack/container/tomcat.rb is true: def supports? web_inf? && !JavaBuildpack::Util::JavaMainUtils.main_class(@application) end The above code means, the tomcat applies when the application has a WEB-INF folder andthisisnot a main class bootstrapped application. Compile Phase: Compile phase would be the major/comprehensive work for a customized buildpack, while it is trying to build a file system on a lxc container. Take the example of our war application and tomcat example. In https://github.com/cloudfoundry/java-buildpack/blob/master/lib/java_buildpack/container/tomcat/tomcat_instance.rb def compile download(@version, @uri) { |file| expand file } link_to(@application.root.children, root) @droplet.additional_libraries << tomcat_datasource_jar if tomcat_datasource_jar.exist? @droplet.additional_libraries.link_to web_inf_lib end def expand(file) with_timing "Expanding Tomcat to #{@droplet.sandbox.relative_path_from(@droplet.root)}" do FileUtils.mkdir_p @droplet.sandbox shell "tar xzf #{file.path} -C #{@droplet.sandbox} --strip 1 --exclude webapps 2>&1" @droplet.copy_resources end The above code is all about preparing the tomcat and link the application files, so the application files will be available for the tomcat classpath. Before going to the code, we have to understand the working directory when the above code executes: . => working directory .app => @application, contains the extracted war archive .buildpack/tomcat => @droplet.sandbox .buildpack/jdk .buildpack/other needed components Inside compile method: download method will download tomcat binary file (specified here: https://github.com/cloudfoundry/java-buildpack/blob/master/config/tomcat.yml), and then extract the archive file to @droplet.sandbox directory. Then copy the resources folder's files to https://github.com/cloudfoundry/java-buildpack/tree/master/resources/tomcat/conf to @droplet.sandbox/conf Symlink the @droplet.sandbox/webapps/ROOT to .app/ Symlink additional libraries (comes from other component rather than application) to the WEB-INF/lib Note: All the symlinks use relative path, since when the container deployed to DEA, the absolute paths would be different. RELEASE PHASE: Release phase is to setup instructions of how to start tomcat. Look at the code in :https://github.com/cloudfoundry/java-buildpack/blob/master/lib/java_buildpack/container/tomcat.rb def command @droplet.java_opts.add_system_property 'http.port', '$PORT' [ @droplet.java_home.as_env_var, @droplet.java_opts.as_env_var, "$PWD/#{(@droplet.sandbox + 'bin/catalina.sh').relative_path_from(@droplet.root)}", 'run' ].flatten.compact.join(' ') end The above code does: Add java system properties http.port (referenced in tomcat server.xml) with environment properties ($PORT), this is the port on the DEA bridging to the lxc container already setup when the container was provisioned. instruction of how to run the tomcat Eg. "./bin/catalina.sh run"

Cyclop is a web-based tool for querying Cassandra databases with features like syntax highlighting and query completion.

Financial systems communicate by sending and receiving vast numbers of messages in many different formats. When people use terms like "vast" I normally think, "really..how many?" So lets quantify "vast" for the finance industry. Market data feeds from financial exchanges typically can be emitting tens or hundreds of thousands of message per second, and aggregate feeds like OPRA can peek at over 10 million messages per second with volumes growing year-on-year. This presentation gives a good overview. In this crazy world we still see significant use of ASCII encoded presentations, such as FIX tag value, and some more slightly sane binary encoded presentations like FAST. Some markets even commit the sin of sending out market data as XML! Well I cannot complain too much as they have at times provided me a good income writing ultra fast XML parsers. Last year the CME, who are a member the FIX community, commissioned Todd Montgomery, of 29West LBM fame, and myself to build the reference implementation of the new FIX Simple Binary Encoding (SBE) standard. SBE is a codec aimed at addressing the efficiency issues in low-latency trading, with a specific focus on market data. The CME, working within the FIX community, have done a great job of coming up with an encoding presentation that can be so efficient. Maybe a suitable atonement for the sins of past FIX tag value implementations. Todd and I worked on the Java and C++ implementation, and later we were helped on the .Net side by the amazing Olivier Deheurles at Adaptive. Working on a cool technical problem with such a team is a dream job. SBE Overview SBE is an OSI layer 6 presentation for encoding/decoding messages in binary format to support low-latency applications. Of the many applications I profile with performance issues, message encoding/decoding is often the most significant cost. I've seen many applications that spend significantly more CPU time parsing and transforming XML and JSON than executing business logic. SBE is designed to make this part of a system the most efficient it can be. SBE follows a number of design principles to achieve this goal. By adhering to these design principles sometimes means features available in other codecs will not being offered. For example, many codecs allow strings to be encoded at any field position in a message; SBE only allows variable length fields, such as strings, as fields grouped at the end of a message. The SBE reference implementation consists of a compiler that takes a message schema as input and then generates language specific stubs. The stubs are used to directly encode and decode messages from buffers. The SBE tool can also generate a binary representation of the schema that can be used for the on-the-fly decoding of messages in a dynamic environment, such as for a log viewer or network sniffer. The design principles drive the implementation of a codec that ensures messages are streamed through memory without backtracking, copying, or unnecessary allocation. Memory access patterns should not be underestimated in the design of a high-performance application. Low-latency systems in any language especially need to consider all allocation to avoid the resulting issues in reclamation. This applies for both managed runtime and native languages. SBE is totally allocation free in all three language implementations. The end result of applying these design principles is a codec that has ~25X greater throughput than Google Protocol Buffers (GPB) with very low and predictable latency. This has been observed in micro-benchmarks and real-world application use. A typical market data message can be encoded, or decoded, in ~25ns compared to ~1000ns for the same message with GPB on the same hardware. XML and FIX tag value messages are orders of magnitude slower again. The sweet spot for SBE is as a codec for structured data that is mostly fixed size fields which are numbers, bitsets, enums, and arrays. While it does work for strings and blobs, many my find some of the restrictions a usability issue. These users would be better off with another codec more suited to string encoding. Message Structure A message must be capable of being read or written sequentially to preserve the streaming access design principle, i.e. with no need to backtrack. Some codecs insert location pointers for variable length fields, such as string types, that have to be indirected for access. This indirection comes at a cost of extra instructions plus loosing the support of the hardware prefetchers. SBE's design allows for pure sequential access and copy-free native access semantics. Figure 1 SBE messages have a common header that identifies the type and version of the message body to follow. The header is followed by the root fields of the message which are all fixed length with static offsets. The root fields are very similar to a struct in C. If the message is more complex then one or more repeating groups similar to the root block can follow. Repeating groups can nest other repeating group structures. Finally, variable length strings and blobs come at the end of the message. Fields may also be optional. The XML schema describing the SBE presentation can be found here. SbeTool and the Compiler To use SBE it is first necessary to define a schema for your messages. SBE provides a language independent type system supporting integers, floating point numbers, characters, arrays, constants, enums, bitsets, composites, grouped structures that repeat, and variable length strings and blobs. A message schema can be input into the SbeTool and compiled to produce stubs in a range of languages, or to generate binary metadata suitable for decoding messages on-the-fly. java [-Doption=value] -jar sbe.jar SbeTool and the compiler are written in Java. The tool can currently output stubs in Java, C++, and C#. Programming with Stubs A full example of messages defined in a schema with supporting code can be found here. The generated stubs follow a flyweight pattern with instances reused to avoid allocation. The stubs wrap a buffer at an offset and then read it sequentially and natively. // Write the message header first MESSAGE_HEADER.wrap(directBuffer, bufferOffset, messageTemplateVersion) .blockLength(CAR.sbeBlockLength()) .templateId(CAR.sbeTemplateId()) .schemaId(CAR.sbeSchemaId()) .version(CAR.sbeSchemaVersion()); // Then write the body of the message car.wrapForEncode(directBuffer, bufferOffset) .serialNumber(1234) .modelYear(2013) .available(BooleanType.TRUE) .code(Model.A) .putVehicleCode(VEHICLE_CODE, srcOffset); Messages can be written via the generated stubs in a fluent manner. Each field appears as a generated pair of methods to encode and decode. // Read the header and lookup the appropriate template to decode MESSAGE_HEADER.wrap(directBuffer, bufferOffset, messageTemplateVersion); finalinttemplateId = MESSAGE_HEADER.templateId(); finalintactingBlockLength = MESSAGE_HEADER.blockLength(); finalintschemaId = MESSAGE_HEADER.schemaId(); finalintactingVersion = MESSAGE_HEADER.version(); // Once the template is located then the fields can be decoded. car.wrapForDecode(directBuffer, bufferOffset, actingBlockLength, actingVersion); finalStringBuilder sb = newStringBuilder(); sb.append("\ncar.templateId=").append(car.sbeTemplateId()); sb.append("\ncar.schemaId=").append(schemaId); sb.append("\ncar.schemaVersion=").append(car.sbeSchemaVersion()); sb.append("\ncar.serialNumber=").append(car.serialNumber()); sb.append("\ncar.modelYear=").append(car.modelYear()); sb.append("\ncar.available=").append(car.available()); sb.append("\ncar.code=").append(car.code()); The generated code in all languages gives performance similar to casting a C struct over the memory. On-The-Fly Decoding The compiler produces an intermediate representation (IR) for the input XML message schema. This IR can be serialised in the SBE binary format to be used for later on-the-fly decoding of messages that have been stored. It is also useful for tools, such as a network sniffer, that will not have been compiled with the stubs. A full example of the IR being used can be found here. Direct Buffers SBE provides an abstraction to Java, via the DirectBuffer class, to work with buffers that are byte[], heap or directByteBuffer buffers, and off heap memory addresses returned from Unsafe.allocateMemory(long) or JNI. In low-latency applications, messages are often encoded/decoded in memory mapped files via MappedByteBuffer and thus can be be transferred to a network channel by the kernel thus avoiding user space copies. C++ and C# have built-in support for direct memory access and do not require such an abstraction as the Java version does. A DirectBuffer abstraction was added for C# to support Endianess and encapsulate the unsafe pointer access. Message Extension and Versioning SBE schemas carry a version number that allows for message extension. A message can be extended by adding fields at the end of a block. Fields cannot be removed or reordered for backwards compatibility. Extension fields must be optional otherwise a newer template reading an older message would not work. Templates carry metadata for min, max, null, timeunit, character encoding, etc., these are accessible via static (class level) methods on the stubs. Byte Ordering and Alignment The message schema allows for precise alignment of fields by specifying offsets. Fields are by default encoded in LittleEndian form unless otherwise specified in a schema. For maximum performance native encoding with fields on word aligned boundaries should be used. The penalty for accessing non-aligned fields on some processors can be very significant. For alignment one must consider the framing protocol and buffer locations in memory. Message Protocols I often see people complain that a codec cannot support a particular presentation in a single message. However this is often possible to address with a protocol of messages. Protocols are a great way to split an interaction into its component parts, these parts are then often composable for many interactions between systems. For example, the IR implementation of schema metadata is more complex than can be supported by the structure of a single message. We encode IR by first sending a template message providing an overview, followed by a stream of messages, each encoding the tokens from the compiler IR. This allows for the design of a very fast OTF decoder which can be implemented as a threaded interrupter with much less branching than the typical switch based state machines. Protocol design is an area that most developers don't seem to get an opportunity to learn. I feel this is a great loss. The fact that so many developers will call an "encoding" such as ASCII a "protocol" is very telling. The value of protocols is so obvious when one gets to work with a programmer like Todd who has spent his life successfully designing protocols. Stub Performance The stubs provide a significant performance advantage over the dynamic OTF decoding. For accessing primitive fields we believe the performance is reaching the limits of what is possible from a general purpose tool. The generated assembly code is very similar to what a compiler will generate for accessing a C struct, even from Java! Regarding the general performance of the stubs, we have observed that C++ has a very marginal advantage over the Java which we believe is due to runtime inserted Safepoint checks. The C# version lags a little further behind due to its runtime not being as aggressive with inlining methods as the Java runtime. Stubs for all three languages are capable of encoding or decoding typical financial messages in tens of nanoseconds. This effectively makes the encoding and decoding of messages almost free for most applications relative to the rest of the application logic. Feedback This is the first version of SBE and we would welcome feedback. The reference implementation is constrained by the FIX community specification. It is possible to influence the specification but please don't expect pull requests to be accepted that significantly go against the specification. Support for Javascript, Python, Erlang, and other languages has been discussed and would be very welcome.

It's not always immediate why and in which part of the program is Hibernate generating a given SQL query, especially if we are dealing with code that we did not write ourselves. This post will go over how to configure Hibernate query logging, and use that together with other tricks to find out why and where in the program a given query is being executed. What does the Hibernate query log look like Hibernate has built-in query logging that looks like this: select /* load your.package.Employee */ this_.code, ... from employee this_ where this_.employee_id=? TRACE 12-04-2014@16:06:02 BasicBinder - binding parameter [1] as [NUMBER] - 1000 Why can't Hibernate log the actual query ? Notice that what is logged by Hibernate is the prepared statement sent by Hibernate to the JDBC driver plus it's parameters. The prepared statement has ? in the place of the query parameters, and the parameter values themselves are logged just bellow the prepared statement. This is not the same as the actual query sent to the database, as there is no way for Hibernate to log the actual query. The reason for this is that Hibernate only knows about the prepared statements and the parameters that it sends to the JDBC driver, and it's the driver that will build the actual queries and then send them to the database. In order to produce a log with the real queries, a tool like log4jdbc is needed, which will be the subject of another post. How to find out the origin of the query The logged query above contains a comment that allows to identify in most cases the origin of the query: if the query is due to a load by ID the comment is /* load your.entity.Name */, if it's a named query then the comment will contain the name of the query. If it's a one to many lazy initialization the comment will contain the name of the class and the property that triggered it, etc. In many cases the query comment created by is enough to identify the origin of the query. Setting up the Hibernate query log In order to obtain a query log, the following flags need to be set in the configuration of the session factory: ... true true true The example above is for Spring configuration of an entity manager factory. This is the meaning of the flags: show_sql enables query logging format_sql pretty prints the SQL use_sql_comments adds an explanatory comment In order to log the query parameters, the following log4j or equivalent configuration is needed: If everything else fails If the query comment added by the option use_sql_comments is not sufficient, then we can start by identifying the entity returned by the query based on the table names involved, and put a breakpoint in the constructor of the returned entity. If the entity does not have a constructor, then we can create one and put the breakpoint in the call to super(): @Entity public class Employee { public Employee() { super(); // put the breakpoint here } ... } When the breakpoint is hit, go to the IDE debug view containing the stack call of the program and go through it from top to bottom. The place where the query was made in the program will be there in the call stack.

In a previous post, we had shared a small function that generated random string in Java. It turns out that similar functionality is available from a class in the extremely useful apache commons lang library. If you are using maven, download the jar using the following dependency: commons-lang commons-lang 20030203.000129 The class we are interested in is RandomStringUtils. Listed below are some functions you may find useful. Generate and print a random string of length 5 from all characters available System.out.println(RandomStringUtils.random(5)); Generate and print random string of length 10 from upper and lower case alphabets System.out.println(RandomStringUtils.randomAlphabetic(10)); Generate and print a random number of length 12 System.out.println(RandomStringUtils.randomNumeric(12)); Generate and print a random string of length 5 using only a, b, c and d characters System.out.println(RandomStringUtils.random(10,new char[]{'a','b','c','d'}));

This post will go through how to setup the Hibernate Second-Level and Query caches, how they work and what are their most common pitfalls.

So when I heard about a new feature in Java 8 that was supposed to help mitigate bugs, I was excited.

Tony Hoare called the invention of the null reference the “billion dollars mistake”. May be the use of primitives in Java could be called the million dollars mistake. Primitives where created for one reason: performance. Primitives have nothing to do in an Object language. Introduction of auto boxing/unboxing was a good thing, but much more should have been done. It probably will be done (it is sometimes said to be on the Java 10 road map). In the meanwhile, we have to deal with primitives, and this is a hassle, specially when using functions. Functions in Java 5/6/7 Before Java 8, one could create functions like this: public interface Function { U apply(T t); } Function addTax = new Function() { @Override public Integer apply(Integer x) { return x / 100 * (100 + 10); } }; System.out.println(addTax.apply(100)); This code produces the following result: 110 What Java 8 gives us is the Function interface and the lambda syntax. We do not need anymore to define our own functional interface, and we may use the following syntax: Function addTax = x -> x / 100 * (100 + 10); System.out.println(addTax.apply(100)); Note that in the first example, we used an anonymous class to create a named function. In the second example, using the lambda syntax does not change anything about this. There is still an anonymous class, and a named function. One interesting question is “What is the type of x?” The type was manifest in the first example. Here, it is inferred because of the type of the function. Java knows the function argument type is an Integer because the type of the function is explicitly Function. The first Integer is the type of the argument, and the second Integer is the return type. Boxing is automatically used to convert int to Integer and back as needed. More on this later. Could we use an anonymous function? Yes, but we would have a problem with type. This does not work: System.out.println((x -> x / 100 * (100 + 10)).apply(100)); which means we can't substitute the identifier addTax with its value (the addTax function). We have to restore the type information that is now missing because Java 8 is simply not able to infer the type in this case. The most visible thing which has no explicit type here is the identifier x. So we might try: System.out.println((Integer x) -> x / 100 * 100 + 10).apply(100)); After all, int the first example, we could have written: Function addTax = (Integer x) -> x / 100 * 100 + 10; so it should be enough for Java to infer the type. But this does not work. What we have to do is specifying the type of the function. Specifying the type of its argument is not enough, even if the return type may be inferred. And there is a serious reason for this: Java 8 does not know anything about functions. Functions are ordinary object with ordinary methods that we may call. Nothing more. So we have to specify the type like this: System.out.println(((Function) x -> x / 100 * 100 + 10).apply(100)); Otherwise, it could translate to: System.out.println(((Whatever) x -> x / 100 * 100 + 10).whatever(100)); So the lambda is only syntactic sugar to simplify the Function (or Whatever) interface implementation by an anonymous class. It has in fact absolutely nothing to do with functions. Should Java had only the Function interface with its apply method, this would not be a big deal. But what about primitives? The Function interface would be fine if Java was an object language. But it is not. It is only vaguely oriented toward the use of objects (hence the name Object Oriented). The most important types in Java are the primitives. And primitives do not fit well in OOP. Auto boxing has been introduced in Java 5 to help us deal with this problem, but auto boxing as severe limitations in terms of performance, and this is related to how thing are evaluated in Java. Java is a strict language, so eager evaluation is the rule. The consequence is that each time we have a primitive and need an object, the primitive has to be boxed. And each time we have an object and need a primitive, it has to be unboxed. If we rely upon automatic boxing an unboxing, we may end with much overhead for multiple boxing and unboxing. Other languages have solved this problem differently, allowing only objects and dealing with conversion in the background. They may have “value classes”, which are objects that are backed with primitives. With this functionality, programmers only use objects and the compiler only use primitives (this is over simplified, but it gives an idea of the principle). By allowing programmers to explicitly manipulate primitives, Java makes things much more difficult and much less safe, because programmers are encouraged to use primitives as business types, which is total nonsense either in OOP or in FP. (I will come back to this in another article.) Let's say it abruptly: we should not care about the overhead of boxing and unboxing. If Java programs using this feature are too slow, the language should be fixed. We should not use bad programming techniques to work around language weaknesses. By using primitives, we make the language work against us, and not for us. If this problem is not solved through fixing the language, we should just use another language. But we probably can't for lot of bad reasons, the most important being that we are payed to program in Java and not in any other language. The result is that instead of solving business problems, we find ourselves solving Java problems. And using primitives is a Java problem, and a big one. Lets rewrite our example using primitives instead of objects. Our function takes an argument of type Integer and returns an Integer. To replace this, Java has the type IntUnaryOperator. Wow, this smells! And guess what, it is defined as: public interface IntUnaryOperator { int applyAsInt(int operand); ... } It would probably have been too simple to call the method apply. So, our example using primitives may be rewritten as: IntUnaryOperator addTax = x -> x / 100 * (100 + 10); System.out.println(addTax.applyAsInt(100)); or, using an anonymous function: System.out.println(((IntUnaryOperator) x -> x / 100 * (100 + 10)).applyAsInt(100)); If only for functions of int returning int, this would be simple. But it is much more complex. Java 8 has 43 (functional) interfaces in the java.util.function package. In reality, they do not all represent functions. They can be grouped as follows: 21 one argument functions, among which 2 are functions of object returning object and 19 are various cases of object to primitive and primitive to object functions. One of the two object to object functions is for the specific case when both argument and return value are of the same type. 9 two arguments functions, among which 2 are functions of (object, object) to object, and 7 are various cases of (object, object) to primitive or (primitive, primitive) to primitive. 7 are effects, and not functions, since they do not return any value and are supposed to be used only for their side effect. (It's somewhat strange to call these “functional interfaces”.) 5 are “suppliers”, which means functions that do not take an argument but return a value. These could be functions. In the functional world, these are special functions called nullary functions (to indicate that their arity, or number of arguments, is zero). As functions, their return value may never change, so they allow treating constants as functions. In Java 8, their role is to depend upon mutable context to return variable values. So, they are not functions. What a mess! And furthermore, the methods of these interfaces have different names. Object functions have a method named apply, where methods returning numeric primitives have method name applyAsInt, applyAsLong, or applyAsDouble. Functions returning boolean have a method called test, and suppliers have methods called get, or getAsInt, getAsLong, getAsDouble, or getAsBoolean. (They did not dare calling BooleanSupplier “Predicate” with a test method taking no argument. I really wonder why!) One thing to note is that there are no functions for byte, char, short and float. Nor are there functions for arity greater that two. Needless to say, this is totally ridiculous. But we have to stick with it. As long as Java can infer the type, we may think we have no problem. However, if you want to manipulate functions in a functional way, you will soon face the problem of Java being unable to infer a type. Worst, Java will sometime infer the type and stay silent while using a type which is no the one you intended. How to help discovering the right type Let's say we want to use a three arguments function. As there are no such functional interfaces in Java 8, you are left with a choice: create you own functional interface, or use currying, as we have seen in a previous article (What's wrong with Java 8 part I ). Creating a three object arguments functional interface returning object is straightforward: interface Function { R apply(T, t, U, u, V, v); } However, we may face two problems. The first one is that we may need to process primitives. Parametric types will not help us for this. You may create special versions of the function using primitives instead of objects. After all, with eight type of primitives, three arguments and one return value, there are only 6 561 different versions of this function. Why do you think Oracle did not put TriFunction in Java 8? (To be precise, they only put a very limited number of BiFunction where arguments are Object and return type int, long or double, or when argument and return types are of the same type int, long or Object, leading to a total of 9 out of 729 possible.) A much better solution is to use autoboxing. Just use Integer, Long, Boolean and so on and let Java handle this. Doing whatever else would be the root of all evil, i.e. premature optimization (see http://c2.com/cgi/wiki?PrematureOptimization). Another way to go (beside creating three arguments functional interface) is to use currying. This is mandatory if the arguments may not be evaluated at the same time. Furthermore, it allows using only functions of one argument, which limits the number of possible functions to 81. If we restrict ourselves to boolean, int, long and double, the number falls to 25 (four primitive types plus Object in two places equals 5 x 5). The problem is that it may be somewhat difficult to use currying with functions returning primitives or taking primitives as their argument. As an example, here is the same example used in our previous article (What's wrong with Java 8 part I ), but using primitives: IntFunction> intToIntCalculation = x -> y -> z -> x + y * z; private IntStream calculate(IntStream stream, int a) { return stream.map(intToIntCalculation.apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); Note that the result is not “a stream containing the values 5, 8, 11, 14 and 17”, no more than the initial stream would have contained the value 1, 2, 3, 4 and 5. newStream in not evaluated at this stage, so it does not contain values. (We'll talk about this in a next article). To see the result, we have to evaluate the stream, which may be forced by binding it to a terminal operation. This may be done through a call to the collect method. But before doing this, we will bind the result to one more non terminal function using the method boxed. The boxed methods binds to the stream a function converting primitives to the corresponding objects. This will simplify evaluation: System.out.println(newStream.boxed().collect(toList())); This prints: [5, 8, 11, 14, 17] We could as well use an anonymous function. However, Java is not be able to infer the type, so we must help it: private IntStream calculate(IntStream stream, int a) { return stream.map(((IntFunction>) x -> y -> z -> x + y * z).apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); Currying in itself is very easy. Just remember, as I said in a previous article, that: (x, y, z) -> w translates to x -> y -> z -> w Finding the right type is slightly more complicated. You have to remember that each time you apply an argument, you are returning a function, so you need a function from the type of the argument to an object type (because functions are objects). Here, each argument is of type int, so we need to use IntFunction parameterized with the type of the returned function. As the final type is IntUnaryOperator (as required by the map method of the IntStream class), the result is: IntFunction>> Here, we are applying two of the three parameters and all parameters are of type int, so the type is: IntFunction> This may be compared to the version using autoboxing: Function>> If you have problems determining the right type, start with the version using autoboxing, just replacing the final type you know you need (since it is the type of the argument of map): Function> Note that you may perfectly use this type in your program: private IntStream calculate(IntStream stream, int a) { return stream.map(((Function>) x -> y -> z -> x + y * z).apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); You may then replace each Function>) x -> y -> z -> x + y * z).apply(b).apply(a)); } and then to: private IntStream calculate(IntStream stream, int a) { return stream.map(((IntFunction>) x -> y -> z -> x + y * z).apply(b).apply(a)); } Note that all three versions compile and run. The only difference is whether autoboxing is used or not. When to be anonymous So, as we saw in the examples above, lambdas are very good at simplifying anonymous class creation, but there is rarely good reason not to name the instance that is created. Naming functions allows: function reuse function testing function replacement program maintenance program documentation Naming function plus currying will make your function completely independent from the environment (“referential transparency”), making you programs safer and more modular. There is however a difficulty. Using primitives makes it difficult to figure the type of curried function. And worst, primitive are not the right business types to use, so the compiler will not be able to help you in this area. To see why, look at this example: double tax = 10.24; double limit = 500.0; double delivery = 35.50; DoubleStream stream3 = DoubleStream.of(234.23, 567.45, 344.12, 765.00); DoubleStream stream4 = stream3.map(x -> { double total = x / 100 * (100 + tax); if ( total > limit) { total = total + delivery; } return total; }); To replace the anonymous “capturing” function by a named curried one, determining the correct type is not so difficult. There will be four arguments and it will return a DoubleUnaryOperator, so the type will be DoubleFunction>>. However, it is very easy to misplace the arguments: DoubleFunction>> computeTotal = x -> y -> z -> w -> { double total = w / 100 * (100 + x); if (total > y) { total = total + z; } return total; }; DoubleStream stream2 = stream.map(computeTotal.apply(tax).apply(limit).apply(delivery)); How can you be sure what x, y, z and w are ? There is in fact a simple rule: the arguments that are evaluated through the explicit use of the apply method come first, in the order they are applied, i.e. tax, limit, delivery, corresponding to x, y and z. The argument coming from the stream is applied last, so it corresponds to w. However, we are still having a problem: once the function is tested, we now that it is correct, but there is no way to be sure it will be used right. For example if we apply the parameters in the wrong order: DoubleStream stream2 = stream.map(computeTotal.apply(limit).apply(tax).apply(delivery)); we get [1440.8799999999999, 3440.2000000000003, 2100.2200000000003, 4625.5] instead of: [258.215152, 661.05688, 379.357888, 878.836] This means we have to test not only the function, but each use of it. Wouldn't it be nice if we could be sure that using the parameters in the wrong order would not compile? This is what using the right type system is about. Using primitives for business types is not good. It has never be. But now, with functions, we have one more reason not to do this. This will be the subject of another article. What's next We have seen how using primitives is somewhat more complicated that using objects. Functions using primitives are a real mess in Java 8. But the worst is to come. In a next article, we will talk about using primitives with streams.