This post will go through how to setup the Hibernate Second-Level and Query caches, how they work and what are their most common pitfalls.

So when I heard about a new feature in Java 8 that was supposed to help mitigate bugs, I was excited.

Tony Hoare called the invention of the null reference the “billion dollars mistake”. May be the use of primitives in Java could be called the million dollars mistake. Primitives where created for one reason: performance. Primitives have nothing to do in an Object language. Introduction of auto boxing/unboxing was a good thing, but much more should have been done. It probably will be done (it is sometimes said to be on the Java 10 road map). In the meanwhile, we have to deal with primitives, and this is a hassle, specially when using functions. Functions in Java 5/6/7 Before Java 8, one could create functions like this: public interface Function { U apply(T t); } Function addTax = new Function() { @Override public Integer apply(Integer x) { return x / 100 * (100 + 10); } }; System.out.println(addTax.apply(100)); This code produces the following result: 110 What Java 8 gives us is the Function interface and the lambda syntax. We do not need anymore to define our own functional interface, and we may use the following syntax: Function addTax = x -> x / 100 * (100 + 10); System.out.println(addTax.apply(100)); Note that in the first example, we used an anonymous class to create a named function. In the second example, using the lambda syntax does not change anything about this. There is still an anonymous class, and a named function. One interesting question is “What is the type of x?” The type was manifest in the first example. Here, it is inferred because of the type of the function. Java knows the function argument type is an Integer because the type of the function is explicitly Function. The first Integer is the type of the argument, and the second Integer is the return type. Boxing is automatically used to convert int to Integer and back as needed. More on this later. Could we use an anonymous function? Yes, but we would have a problem with type. This does not work: System.out.println((x -> x / 100 * (100 + 10)).apply(100)); which means we can't substitute the identifier addTax with its value (the addTax function). We have to restore the type information that is now missing because Java 8 is simply not able to infer the type in this case. The most visible thing which has no explicit type here is the identifier x. So we might try: System.out.println((Integer x) -> x / 100 * 100 + 10).apply(100)); After all, int the first example, we could have written: Function addTax = (Integer x) -> x / 100 * 100 + 10; so it should be enough for Java to infer the type. But this does not work. What we have to do is specifying the type of the function. Specifying the type of its argument is not enough, even if the return type may be inferred. And there is a serious reason for this: Java 8 does not know anything about functions. Functions are ordinary object with ordinary methods that we may call. Nothing more. So we have to specify the type like this: System.out.println(((Function) x -> x / 100 * 100 + 10).apply(100)); Otherwise, it could translate to: System.out.println(((Whatever) x -> x / 100 * 100 + 10).whatever(100)); So the lambda is only syntactic sugar to simplify the Function (or Whatever) interface implementation by an anonymous class. It has in fact absolutely nothing to do with functions. Should Java had only the Function interface with its apply method, this would not be a big deal. But what about primitives? The Function interface would be fine if Java was an object language. But it is not. It is only vaguely oriented toward the use of objects (hence the name Object Oriented). The most important types in Java are the primitives. And primitives do not fit well in OOP. Auto boxing has been introduced in Java 5 to help us deal with this problem, but auto boxing as severe limitations in terms of performance, and this is related to how thing are evaluated in Java. Java is a strict language, so eager evaluation is the rule. The consequence is that each time we have a primitive and need an object, the primitive has to be boxed. And each time we have an object and need a primitive, it has to be unboxed. If we rely upon automatic boxing an unboxing, we may end with much overhead for multiple boxing and unboxing. Other languages have solved this problem differently, allowing only objects and dealing with conversion in the background. They may have “value classes”, which are objects that are backed with primitives. With this functionality, programmers only use objects and the compiler only use primitives (this is over simplified, but it gives an idea of the principle). By allowing programmers to explicitly manipulate primitives, Java makes things much more difficult and much less safe, because programmers are encouraged to use primitives as business types, which is total nonsense either in OOP or in FP. (I will come back to this in another article.) Let's say it abruptly: we should not care about the overhead of boxing and unboxing. If Java programs using this feature are too slow, the language should be fixed. We should not use bad programming techniques to work around language weaknesses. By using primitives, we make the language work against us, and not for us. If this problem is not solved through fixing the language, we should just use another language. But we probably can't for lot of bad reasons, the most important being that we are payed to program in Java and not in any other language. The result is that instead of solving business problems, we find ourselves solving Java problems. And using primitives is a Java problem, and a big one. Lets rewrite our example using primitives instead of objects. Our function takes an argument of type Integer and returns an Integer. To replace this, Java has the type IntUnaryOperator. Wow, this smells! And guess what, it is defined as: public interface IntUnaryOperator { int applyAsInt(int operand); ... } It would probably have been too simple to call the method apply. So, our example using primitives may be rewritten as: IntUnaryOperator addTax = x -> x / 100 * (100 + 10); System.out.println(addTax.applyAsInt(100)); or, using an anonymous function: System.out.println(((IntUnaryOperator) x -> x / 100 * (100 + 10)).applyAsInt(100)); If only for functions of int returning int, this would be simple. But it is much more complex. Java 8 has 43 (functional) interfaces in the java.util.function package. In reality, they do not all represent functions. They can be grouped as follows: 21 one argument functions, among which 2 are functions of object returning object and 19 are various cases of object to primitive and primitive to object functions. One of the two object to object functions is for the specific case when both argument and return value are of the same type. 9 two arguments functions, among which 2 are functions of (object, object) to object, and 7 are various cases of (object, object) to primitive or (primitive, primitive) to primitive. 7 are effects, and not functions, since they do not return any value and are supposed to be used only for their side effect. (It's somewhat strange to call these “functional interfaces”.) 5 are “suppliers”, which means functions that do not take an argument but return a value. These could be functions. In the functional world, these are special functions called nullary functions (to indicate that their arity, or number of arguments, is zero). As functions, their return value may never change, so they allow treating constants as functions. In Java 8, their role is to depend upon mutable context to return variable values. So, they are not functions. What a mess! And furthermore, the methods of these interfaces have different names. Object functions have a method named apply, where methods returning numeric primitives have method name applyAsInt, applyAsLong, or applyAsDouble. Functions returning boolean have a method called test, and suppliers have methods called get, or getAsInt, getAsLong, getAsDouble, or getAsBoolean. (They did not dare calling BooleanSupplier “Predicate” with a test method taking no argument. I really wonder why!) One thing to note is that there are no functions for byte, char, short and float. Nor are there functions for arity greater that two. Needless to say, this is totally ridiculous. But we have to stick with it. As long as Java can infer the type, we may think we have no problem. However, if you want to manipulate functions in a functional way, you will soon face the problem of Java being unable to infer a type. Worst, Java will sometime infer the type and stay silent while using a type which is no the one you intended. How to help discovering the right type Let's say we want to use a three arguments function. As there are no such functional interfaces in Java 8, you are left with a choice: create you own functional interface, or use currying, as we have seen in a previous article (What's wrong with Java 8 part I ). Creating a three object arguments functional interface returning object is straightforward: interface Function { R apply(T, t, U, u, V, v); } However, we may face two problems. The first one is that we may need to process primitives. Parametric types will not help us for this. You may create special versions of the function using primitives instead of objects. After all, with eight type of primitives, three arguments and one return value, there are only 6 561 different versions of this function. Why do you think Oracle did not put TriFunction in Java 8? (To be precise, they only put a very limited number of BiFunction where arguments are Object and return type int, long or double, or when argument and return types are of the same type int, long or Object, leading to a total of 9 out of 729 possible.) A much better solution is to use autoboxing. Just use Integer, Long, Boolean and so on and let Java handle this. Doing whatever else would be the root of all evil, i.e. premature optimization (see http://c2.com/cgi/wiki?PrematureOptimization). Another way to go (beside creating three arguments functional interface) is to use currying. This is mandatory if the arguments may not be evaluated at the same time. Furthermore, it allows using only functions of one argument, which limits the number of possible functions to 81. If we restrict ourselves to boolean, int, long and double, the number falls to 25 (four primitive types plus Object in two places equals 5 x 5). The problem is that it may be somewhat difficult to use currying with functions returning primitives or taking primitives as their argument. As an example, here is the same example used in our previous article (What's wrong with Java 8 part I ), but using primitives: IntFunction> intToIntCalculation = x -> y -> z -> x + y * z; private IntStream calculate(IntStream stream, int a) { return stream.map(intToIntCalculation.apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); Note that the result is not “a stream containing the values 5, 8, 11, 14 and 17”, no more than the initial stream would have contained the value 1, 2, 3, 4 and 5. newStream in not evaluated at this stage, so it does not contain values. (We'll talk about this in a next article). To see the result, we have to evaluate the stream, which may be forced by binding it to a terminal operation. This may be done through a call to the collect method. But before doing this, we will bind the result to one more non terminal function using the method boxed. The boxed methods binds to the stream a function converting primitives to the corresponding objects. This will simplify evaluation: System.out.println(newStream.boxed().collect(toList())); This prints: [5, 8, 11, 14, 17] We could as well use an anonymous function. However, Java is not be able to infer the type, so we must help it: private IntStream calculate(IntStream stream, int a) { return stream.map(((IntFunction>) x -> y -> z -> x + y * z).apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); Currying in itself is very easy. Just remember, as I said in a previous article, that: (x, y, z) -> w translates to x -> y -> z -> w Finding the right type is slightly more complicated. You have to remember that each time you apply an argument, you are returning a function, so you need a function from the type of the argument to an object type (because functions are objects). Here, each argument is of type int, so we need to use IntFunction parameterized with the type of the returned function. As the final type is IntUnaryOperator (as required by the map method of the IntStream class), the result is: IntFunction>> Here, we are applying two of the three parameters and all parameters are of type int, so the type is: IntFunction> This may be compared to the version using autoboxing: Function>> If you have problems determining the right type, start with the version using autoboxing, just replacing the final type you know you need (since it is the type of the argument of map): Function> Note that you may perfectly use this type in your program: private IntStream calculate(IntStream stream, int a) { return stream.map(((Function>) x -> y -> z -> x + y * z).apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); You may then replace each Function>) x -> y -> z -> x + y * z).apply(b).apply(a)); } and then to: private IntStream calculate(IntStream stream, int a) { return stream.map(((IntFunction>) x -> y -> z -> x + y * z).apply(b).apply(a)); } Note that all three versions compile and run. The only difference is whether autoboxing is used or not. When to be anonymous So, as we saw in the examples above, lambdas are very good at simplifying anonymous class creation, but there is rarely good reason not to name the instance that is created. Naming functions allows: function reuse function testing function replacement program maintenance program documentation Naming function plus currying will make your function completely independent from the environment (“referential transparency”), making you programs safer and more modular. There is however a difficulty. Using primitives makes it difficult to figure the type of curried function. And worst, primitive are not the right business types to use, so the compiler will not be able to help you in this area. To see why, look at this example: double tax = 10.24; double limit = 500.0; double delivery = 35.50; DoubleStream stream3 = DoubleStream.of(234.23, 567.45, 344.12, 765.00); DoubleStream stream4 = stream3.map(x -> { double total = x / 100 * (100 + tax); if ( total > limit) { total = total + delivery; } return total; }); To replace the anonymous “capturing” function by a named curried one, determining the correct type is not so difficult. There will be four arguments and it will return a DoubleUnaryOperator, so the type will be DoubleFunction>>. However, it is very easy to misplace the arguments: DoubleFunction>> computeTotal = x -> y -> z -> w -> { double total = w / 100 * (100 + x); if (total > y) { total = total + z; } return total; }; DoubleStream stream2 = stream.map(computeTotal.apply(tax).apply(limit).apply(delivery)); How can you be sure what x, y, z and w are ? There is in fact a simple rule: the arguments that are evaluated through the explicit use of the apply method come first, in the order they are applied, i.e. tax, limit, delivery, corresponding to x, y and z. The argument coming from the stream is applied last, so it corresponds to w. However, we are still having a problem: once the function is tested, we now that it is correct, but there is no way to be sure it will be used right. For example if we apply the parameters in the wrong order: DoubleStream stream2 = stream.map(computeTotal.apply(limit).apply(tax).apply(delivery)); we get [1440.8799999999999, 3440.2000000000003, 2100.2200000000003, 4625.5] instead of: [258.215152, 661.05688, 379.357888, 878.836] This means we have to test not only the function, but each use of it. Wouldn't it be nice if we could be sure that using the parameters in the wrong order would not compile? This is what using the right type system is about. Using primitives for business types is not good. It has never be. But now, with functions, we have one more reason not to do this. This will be the subject of another article. What's next We have seen how using primitives is somewhat more complicated that using objects. Functions using primitives are a real mess in Java 8. But the worst is to come. In a next article, we will talk about using primitives with streams.

Earlier I had blogged about using Scala with Spring Boot and how the combination just works. There was one issue with the previous approach though - the only way to run the earlier configuration was to build the project into a jar file and run the jar file. ./gradlew build java -jar build/libs/spring-boot-scala-web-0.1.0.jar Spring boot comes with a gradle based plugin which should have allowed the project to run with a "gradle bootRun" command, this unfortunately gives an error for scala based projects. A good workaround is to use sbt for building and running Spring-boot based projects. The catch though is that with gradle and maven, the versions of the dependencies would have been managed through a parent pom, now these have to be explicitly specified. This is how a sample sbt build file with the dependencies spelled out looks: name := "spring-boot-scala-web" version := "1.0" scalaVersion := "2.10.4" sbtVersion := "0.13.1" seq(webSettings : _*) libraryDependencies ++= Seq( "org.springframework.boot" % "spring-boot-starter-web" % "1.0.2.RELEASE", "org.springframework.boot" % "spring-boot-starter-data-jpa" % "1.0.2.RELEASE", "org.webjars" % "bootstrap" % "3.1.1", "org.webjars" % "jquery" % "2.1.0-2", "org.thymeleaf" % "thymeleaf-spring4" % "2.1.2.RELEASE", "org.hibernate" % "hibernate-validator" % "5.0.2.Final", "nz.net.ultraq.thymeleaf" % "thymeleaf-layout-dialect" % "1.2.1", "org.hsqldb" % "hsqldb" % "2.3.1", "org.springframework.boot" % "spring-boot-starter-tomcat" % "1.0.2.RELEASE" % "provided", "javax.servlet" % "javax.servlet-api" % "3.0.1" % "provided" ) libraryDependencies ++= Seq( "org.apache.tomcat.embed" % "tomcat-embed-core" % "7.0.53" % "container", "org.apache.tomcat.embed" % "tomcat-embed-logging-juli" % "7.0.53" % "container", "org.apache.tomcat.embed" % "tomcat-embed-jasper" % "7.0.53" % "container" ) Here I am also using xsbt-web-plugin which is plugin for building scala web applications. xsbt-web-plugin also comes with commands to start-up tomcat or jetty based containers and run the applications within these containers, however I had difficulty in getting these to work. What worked is the runMain command to start up the Spring-boot main program through sbt: runMain mvctest.SampleWebApplication and xsbt-web-plugin allows the project to be packaged as a war file using the "package" command, this war deploys and runs without any issues in a standalone tomcat container. Here is a github project with these changes: https://github.com/bijukunjummen/spring-boot-scala-web.git

Learn all about Jersey/Jax RS streaming with JSON.

The Open Session in View (OSIV) pattern gives rise to different opinions in the Java development community. Let's go over OSIV and some of the pros and cons of this pattern. The problem The problem that OSIV solves is a mismatch between the Hibernate concept of session and it's lifecycle and the way that many server-side view technologies work. In a typical Java frontend application the service layer starts by querying some of the data needed to build the view. The remaining data needed can be lazy-loaded later, with the condition that the Hibernate session remains open - and there lies the problem. Between the moment that the service layer method finishes it's execution and the moment that the view is rendered, Hibernate has already committed the transaction and closed the session. When the view tries to lazy load the extra data that it needs, if finds the Hibernate session closed, causing a LazyInitializationException. The OSIV solution OSIV tackles this problem by ensuring that the Hibernate session is kept open all the way up to the rendering of the view - hence the name of the pattern. Because the session is kept open, no more LazyInitializationExceptions occur. The session or entity manager is kept open by means of a filter that is added to the request processing chain. In the case of JPA the OpenEntityManagerInViewFilter will create an entity manager at the beginning of the request, and then bind it to the request thread. The service layer will then be executed and the business transaction committed or rolled back, but the transaction manager will not remove the entity manager from the thread after the commit. When the view rendering starts, the transaction manager will then check if there is already an entity manager binded to the thread, and if so use it instead of creating a new one. After the request is processed, the filter will then unbind the entity manager from the thread. The end result is that the same entity manager used to commit the business transaction was kept around in the request thread, allowing the view rendering code to lazy load the needed data. Going back to the original problem Let's step back a moment and go back to the initial problem: the LazyInitializationException. Is this exception really a problem? This exception can also be seen as a warning sign of a wrongly written query in the service layer. When building a view and it's backing services, the developer knows upfront what data is needed, and can make sure that the needed data is loaded before the rendering starts. Several relation types such as one-to-many use lazy-loading by default, but that default setting can be overridden if needed at query time using the following syntax: select p FROM Person p left join fetch p.invoices This means that the lazy loading can be turned off on a case by case basis depending on the data needed by the view. OSIV in projects I've worked In projects I have worked that used OSIV, we could see via query logging that the database was getting hit with a high number of SQL queries, sometimes to the point that developers had to turn off the Hibernate SQL logging. The performance of these application was impacted, but it was kept manageable using second-level caches, and due to the fact that these where intranet-based applications with a limited number of users. Pros of OSIV The main advantage of OSIV is that it makes working with ORM and the database more transparent: Less queries need to be manually written Less awareness is required about the Hibernate session and how to solve LazyInitializationExceptions. Cons of OSIV OSIV seems to be easy to misuse and can accidentally introduce N+1 performance problems in the application. On projects I've worked OSIV did not work out well in the long-term. The alternative of writing custom queries that eager fetch data depending on the use case is manageable and turned out well in other projects I've worked. Alternatives to OSIV Besides the application-level solution of writing custom queries to pre-fetch the needed data, there are other framework-level aproaches to OSIV. The Seam Framework was built by some of the same developers as Hibernate , and solves the problem by introducing the notion of conversation. Can you let me know in the comments bellow your thoughts and experiences with OSIV, thanks for reading.

System Requirements: Eclipse Kepler Service Release 2 JDK 1.5 or above Installed Google App Engine SDK on eclipse – this is required for second version of this example. Create a google spreadsheet – login to your google account and create a new spreadsheet, if you want to read existing one then put that url in SPREADSHEET_URL. Once you create a new spreadsheet our url will be like – https://docs.google.com/spreadsheets/d/1L8xtAJfOObsXL-XemliUV10wkDHQNxjn6jKS4XwzYZ8/ but don’t put this url in SPREADSHEET_URL, Use the below url simply change the bold part. public static final String SPREADSHEET_URL = “https://spreadsheets.google.com/feeds/spreadsheets/1L8xtAJfOObsXL-XemliUV10wkDHQNxjn6jKS4XwzYZ8“; // Fill in google spreadsheet URI package org.gopaldas.readsps; import java.io.IOException; import java.net.URL; import com.google.gdata.client.spreadsheet.SpreadsheetService; import com.google.gdata.data.spreadsheet.ListEntry; import com.google.gdata.data.spreadsheet.ListFeed; import com.google.gdata.data.spreadsheet.SpreadsheetEntry; import com.google.gdata.data.spreadsheet.WorksheetEntry; import com.google.gdata.util.ServiceException; public class ReadSpreadsheet { public static final String GOOGLE_ACCOUNT_USERNAME = "[email protected]"; // Fill in google account username public static final String GOOGLE_ACCOUNT_PASSWORD = "xxxx"; // Fill in google account password public static final String SPREADSHEET_URL = "https://spreadsheets.google.com/feeds/spreadsheets/1L8xtAJfOObsXL-XemliUV10wkDHQNxjn6jKS4XwzYZ8"; //Fill in google spreadsheet URI public static void main(String[] args) throws IOException, ServiceException { /** Our view of Google Spreadsheets as an authenticated Google user. */ SpreadsheetService service = new SpreadsheetService("Print Google Spreadsheet Demo"); // Login and prompt the user to pick a sheet to use. service.setUserCredentials(GOOGLE_ACCOUNT_USERNAME, GOOGLE_ACCOUNT_PASSWORD); // Load sheet URL metafeedUrl = new URL(SPREADSHEET_URL); SpreadsheetEntry spreadsheet = service.getEntry(metafeedUrl, SpreadsheetEntry.class); URL listFeedUrl = ((WorksheetEntry) spreadsheet.getWorksheets().get(0)).getListFeedUrl(); // Print entries ListFeed feed = (ListFeed) service.getFeed(listFeedUrl, ListFeed.class); for(ListEntry entry : feed.getEntries()) { System.out.println("new row"); for(String tag : entry.getCustomElements().getTags()) { System.out.println(" "+tag + ": " + entry.getCustomElements().getValue(tag)); } } } }

There are multiple ways to get hold of and use an Http session with a Spring based web application. This is a summarization based on an experience with a recent project. Approach 1 Just inject in HttpSession where it is required. @Service public class ShoppingCartService { @Autowired private HttpSession httpSession; ... } Though surprising, since the service above is a singleton, this works well. Spring intelligently injects in a proxy to the actual HttpSession and this proxy knows how to internally delegate to the right session for the request. The catch with handling session this way though is that the object being retrieved and saved back in the session will have to be managed by the user: public void removeFromCart(long productId) { ShoppingCart shoppingCart = getShoppingCartInSession(); shoppingCart.removeItemFromCart(productId); updateCartInSession(shoppingCart); } Approach 2 Accept it as a parameter, this will work only in the web tier though: @Controller public class ShoppingCartController { @RequestMapping("/addToCart") public String addToCart(long productId, HttpSession httpSession) { //do something with the httpSession } } Approach 3 Create a bean and scope it to the session this way: @Component @Scope(proxyMode=ScopedProxyMode.TARGET_CLASS, value="session") public class ShoppingCart implements Serializable{ ... } Spring creates a proxy for a session scoped bean and makes the proxy available to services which inject in this bean. An advantage of using this approach is that any state changes on this bean are handled by Spring, it would take care of retrieving this bean from the session and propagating any changes to the bean back to the session. Further if the bean were to have any Spring lifecycle methods(say @PostConstruct or @PreDestroy annotated methods), they would get called appropriately. Approach 4 Annotating Spring MVC model attributes with @SessionAttribute annotation: @SessionAttributes("shoppingCart") public class OrderFlowController { public String step1(@ModelAttribute("shoppingCart") ShoppingCart shoppingCart) { } public String step2(@ModelAttribute("shoppingCart") ShoppingCart shoppingCart) { } public String step3(@ModelAttribute("shoppingCart") ShoppingCart shoppingCart, SessionStatus status) { status.setComplete(); } } The use case for using SessionAttributes annotation is very specific, to hold state during a flow like above Given these approaches, I personally prefer Approach 3 of using session scoped beans, this way depending on Spring to manage the underlying details of retrieving and storing the object into session. Other approaches have value though based on the scenario that you may be faced with, ranging from requiring more control over raw Http Sessions to needing to handle temporary state like in Approach 4 above.

wanted to go through some of the basic tenets, the technical terminology related to java ee. for many people, java ee/j2ee still mean servlets, jsps or maybe struts at best. no offence or pun intended! this is not a java ee 'bible' by any means. i am not capable enough of writing such a thing! so let us line up the 'keywords' related to java ee and then look at them one by one java ee java ee apis (specifications) containers services multitiered applications components let's try to elaborate on the above mentioned points. ok. so what is java ee? 'ee' stands for enterprise edition. that essentially makes java ee - java enterprise edition. if i had to summarize java ee in a couple of sentences, it would go something like this "java ee is a platform which defines 'standard specifications/apis' which are then implemented by vendors and used for development of enterprise (distributed, 'multi-tired', robust) 'applications'. these applications are composed of modules or 'components' which use java ee 'containers' as their run-time infrastructure." what is this 'standardized platform' based upon? what does it constitute? the platform revolves around 'standard' specifications or apis . think of these as contracts defined by a standard body e.g. enterprise java beans (ejb), java persistence api (jpa), java message service (jms) etc. these contracts/specifications/apis are implemented by different vendors e.g. glassfish, oracle weblogic, apache tomee etc alright. what about containers? containers can be visualized as 'virtual/logical partitions' . each container supports a subset of the apis/specifications defined by the java ee platform they provide run-time 'services' to the 'applications' which they host the java ee specification lists 4 types of containers ejb container web container application client container applet container java ee containers i am not going to dwell into details of these containers in this post. services?? well, 'services' are nothing but a result of the vendor implementations of the standard 'specifications' (mentioned above). examples of specifications are - jersey for jax-rs (restful services), tyrus (web sockets), eclipselink (jpa), weld (cdi) etc. the 'container' is the interface between the deployed application ('service' consumer) and the application server. here is a list of 'services' which are rendered by the 'container' to the underlying 'components' (this is not an exhaustive list) persistence - offered by the java persistence api (jpa) which drives object relational mapping (orm) and an abstraction for the database operations. messaging - the java message service (jms) provides asynchronous messaging between disparate parts of your applications. contexts & dependency injection - cdi provides loosely coupled and type safe injection of resources. web services - jaxrs and jaxws provide support for rest and soap style services respectively transaction - provided by the java transaction api (jta) implementation what is a typical java ee 'application'? what does it comprise of? applications are composed of different ' components ' which in turn are supported by their corresponding ' container ' supported 'component' types are: enterprise applications - make use of the specifications like ejb, jms, jpa etc and are executed within an ejb container web applications - they leverage the servlet api, jsp, jsf etc and are supported by a web container application client - executed in client side. they need an application client container which has a set of supported libraries and executes in a java se environment. applets - these are gui applications which execute in a web browser. how are java ee applications structured? as far as java ee 'application' architecture is concerned, they generally tend follow the n-tier model consisting of client tier, server tier and of course the database (back end) tier client tier - consists of web browsers or gui (swing, java fx) based clients. web browsers tend to talk to the 'web components' on the server tier while the gui clients interact directly with the 'business' layer within the server tier server tier - this tier comprises of the dynamic web components (jsp, jsf, servlets) and the business layer driven by ejbs, jms, jpa, jta specifications. database tier - contains 'enterprise information systems' backed by databases or even legacy data repositories. generic 3-tier java ee application architecture java ee - bare bones, basics.... as quickly and briefly as i possibly could. that's all for now! :-) stay tuned for more java ee content, specifically around the latest and greatest version of the java ee platform --> java ee 7 happy reading!

splunk vs. sumo logic vs. logstash vs. graylog vs. loggly vs. papertrails vs. splunk>storm splunk, sumo logic, logstash, graylog, loggly, papertrails - did i miss someone? i’m pretty sure i did. logs are like fossil fuels - we’ve been wanting to get rid of them for the past 20 years, but we’re not quite there yet. well, if that's the case i want a bmw! to deal with the growth of log data a host of log management & analysis tools have been built over the last few years to help developers and operations make sense of the growing data. i thought it’d be interesting to look at our options and what are each tools’ selling point, from a developer’s standpoint . splunk as the biggest tool in this space, i decided to put splunk in a category of its own. that’s not to say it’s the best tool for what you need, but more to give credit to a product who essentially created a new category. pros splunk is probably the most feature rich solution in the space. it’s got hundreds of apps (i counted 537 ) to make sense of almost every format of log data, from security to business analytics to infrastructure monitoring. splunk’s search and charting tools are feature rich to the point that there’s probably no set of data you can’t get to through its ui or apis. cons splunk has two major cons. the first, that is more subjective, is that it’s an on-premise solution which means that setup costs in terms of money and complexity are high. to deploy in a high-scale environment you will need to install and configure a dedicated cluster. as a developer, it’s usually something you can't or don’t want to do as your first choice. splunk’s second con is that it’s expensive. to support a real-world application you’re looking at tens of thousands of dollars, which most likely means you’ll need sign offs from high-ups in your organization, and the process is going to be slow. if you’ve got a new app and you want something fast that you can quickly spin up and ramp as things progress - keep reading. some more enterprise log analyzers can be found here . saas log analyzers sumo logic sumo was founded as a saas version of splunk, going so far as to imitate some of splunk’s features and visuals early on. having said that, sl has developed to a full fledged enterprise class log management solution. pros sl is chock-full of features to reduce, search and chart mass amounts of data. out of all the saas log analyzers, it’s probably the most feature rich. also, being a saas offering it inherently means setup and ongoing operation are easier. one of sumo logic’s main points of attraction is the ability to establish baselines and to actively notify you when key metrics change after an event such as a new version rollout or a breach attempt. cons this one is shared across all saas log analyzers, which is you need to get the data to the service to actually do something with it. this means that you’ll be looking at possible gbs (or more) uploaded from your servers. this can create issues on multiple fronts - as a developer, if you're logging sensitive or pii you need to make sure it’s redacted. there may be a lag between the time data is logged and the time it’s visible to to the service. there’s additional overhead on your machines transmitting gbs of data, which really depends on your logging throughput. sumo’s pricing is also not transparent , which means you might be looking at a buying process which is more complex than swiping your team’s credit card to get going. loggly loggly is also a robust log analyzer, focusing on simplicity and ease of use for a devops audience. pros whereas sumo logic has a strong enterprise and security focus, loggly is geared more towards helping devops find and fix operational problems. this makes it very developer-friendly. things like creating custom performance and devops dashboards are super-easy to do. pricing is also transparent, which makes start of use easier. cons don't expect loggly to scale into a full blown infrastructure, security or analytics solution. if you need forensics or infrastructure monitoring you’re in the wrong place. this is a tools mainly for devops to parse data coming from your app servers. anything beyond that you’ll have to build yourself. papertrails papertrails is a simple way to look and search through logs from multiple machines, in one consolidated easy-to-use interface. think of it like tailing your log in the cloud, and you won't be too far off. pros pt is what it is. a simple way to look at log files from multiple machines in a singular view in the cloud. the ux itself is very similar to looking at a log on your machine, and so are the search commands. it aims to do something simple and useful, and does it elegantly. it’s also very affordable . cons pt is mostly text based. looking for any advanced integrations, predictive or reporting capabilities? you're barking up the wrong tree. splunk>storm this is splunk’s little (some may say step) saas brother. it’s a pretty similar offering that’s hosted on splunk’s servers. pros storm lets you experiment with splunk without having to install the actual software on-premise, and contains much of the features available in the full version. cons this isn't really a commercial offering, and you're limited in the amount of data you can send. it seems to be more of an online limited version of splunk meant to help people test out the product without having to deploy first. a new service called splunk cloud is aimed at providing a full-blown splunk saas experience. open source analyzers logstash logstash is an open source tool for collecting and managing log files. it’s part of an open-source stack which includes elasticsearch for indexing and searching through data and kibana for charting and visualizing data. together they form a powerful log management solution. pros being an open-source solution means you're inherently getting a lot of a control and a very good price. logstash uses three mature and powerful components, all heavily maintained, to create a very robust and extensible package. for an open-source solution it’s also very easy to install and start using. we use logstash and love it. cons as logstash is essentially a stack, it means you're dealing with three different products. that means that extensibility also becomes complex. logstash filters are written in ruby, kibana is pure javascript and elasticsearch has its own rest api as well as json templates. when you move to production, you’ll also need to separate the three into different machines, which adds to the complexity. graylog2 a fairly new player in the space, gl2 is an open-source log analyzer backed by mongodb as well as elasticsearch (similar to logstash) for storing and searching through log errors. it’s mainly focused on helping developers detect and fix errors in their apps. also in this category you can find fluentd and kafka whose one of its main use-cases is also storing log data. phew, so many choices! takipi for logs while this post is not about takipi, i thought there’s one feature it has which you might find relevant to all of this. the biggest disadvantage in all log analyzers and log files in general, is that the right data has to be put there by you first. from a dev perspective, it means that if an exception isn’t logged, or the variable data you need to understand why it happened isn't there, no log file or analyzer in the world can help you. production debugging sucks. one of the things we’ve added to takipi is the ability to jump into a recorded debugging session straight from a log file error. this means that for every log error you can see the actual source code and variable values at the moment of error. you can learn more about it here . this is one post where i would love to hear from you guys about your experiences with some of the tools mentioned (and some that i didn’t). i’m sure there are things you would disagree with or would like to correct me on - so go ahead, the comment section is below and i would love to hear from you. originally posted on takipi blog

This is one annoying problem that happens sometimes to git users: the symptom is: git status command shows you some files as modified (you are sure that you had not modified that files), you revert all changes with a git checkout — . but the files stills are in modified state if you issue another git status. This is a real annoying problem, suppose you want to switch branch with git checkout branchname, you will find that git does not allow you to switch because of uncommitted changes. This problem is likely caused by the end-of-line normalization (I strongly suggest you to read all the details in Pro Git book or read the help of github). I do not want to enter into details of this feature, but I only want to help people to diagnose and avoid this kind of problem. To understand if you really have a Line Ending Issue you should run git diff -w command to verify what is really changed in files that git as modified with git status command. The -w options tells git to ignore whitespace and line endings, if this command shows no differences, you are probably victim of problem in Line Ending Normalization. This is especially true if you are working with git svn, connecting to a subversion repository where developers did not pay attention to line endings and it happens usually when you have files with mixed CRLF / CR / LF. If you work in mixed environment (Unix/Linux, Windows, Macintosh) it is better to find files that are listed as modified and manually (or with some tool) normalize Line Endings. If you do not work in mixed environment you can simply turn off eol normalizationfor the single repository where you experience the problem. To do this you can issue a git config –local core.autocrlf false but it works only for you and not for all the other developers that works to the project. Moreover some people reports that they still have problem even with core.autocrlf to false. Remember that git supports .gitattributes files, used to change settings for a single subdirectory. If you set core.autocrlf to false and still have line ending normalization problem, please search for .gitattribuges files in every subdirectory of your repository, and verify if it has a line where autocrlf is turned on: * text=auto now you can turn off in all .gitattributes files you find in your repository * text=off To be sure that every developer of the team works with autocrlf turned off, you should place a .gitattributes file in repository root with autocrlf turned off. Remember that it is a better option to normalize files and leave autocrlf turned on, but if you are working with legacy code imported from another VCS, or you work with git svn, git-tf or similar tools, probably it is better turn autocrlf to off if you start experiencing that kind of problems.

Get a rundown on the Tomcat NIO Connector as well as a tutorial on how to set it up.

Michael recently wrote a blog post showing how to create a time tree representing time down to the second using Neo4j’s Cypher query language, something I built on top of for a side project I’m working on. The domain I want to model is RSVPs to meetup invites – I want to understand how much in advance people respond and how likely they are to drop out at a later stage. For this problem I only need to measure time down to the day so my task is a bit easier than Michael’s. After a bit of fiddling around with leap years, I believe the following query will create a time tree representing all the days from 2011 – 2014, which covers the time the London Neo4j meetup has been running: WITH range(2011, 2014) AS years, range(1,12) as months FOREACH(year IN years | MERGE (y:Year {year: year}) FOREACH(month IN months | CREATE (m:Month {month: month}) MERGE (y)-[:HAS_MONTH]->(m) FOREACH(day IN (CASE WHEN month IN [1,3,5,7,8,10,12] THEN range(1,31) WHEN month = 2 THEN CASE WHEN year % 4 <> 0 THEN range(1,28) WHEN year % 100 <> 0 THEN range(1,29) WHEN year % 400 <> 0 THEN range(1,29) ELSE range(1,28) END ELSE range(1,30) END) | CREATE (d:Day {day: day}) MERGE (m)-[:HAS_DAY]->(d)))) The next step is to link adjacent days together so that we can easily traverse between adjacent days without needing to go back up and down the tree. For example we should have something like this: (jan31)-[:NEXT]->(feb1)-[:NEXT]->(feb2) We can build this by first collecting all the ‘day’ nodes in date order like so: MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day) WITH year,month,day ORDER BY year.year, month.month, day.day WITH collect(day) as days RETURN days And then iterating over adjacent nodes to create the ‘NEXT’ relationship: MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day) WITH year,month,day ORDER BY year.year, month.month, day.day WITH collect(day) as days FOREACH(i in RANGE(0, length(days)-2) | FOREACH(day1 in [days[i]] | FOREACH(day2 in [days[i+1]] | CREATE UNIQUE (day1)-[:NEXT]->(day2)))) Now if we want to find the previous 5 days from the 1st February 2014 we could write the following query: MATCH (y:Year {year: 2014})-[:HAS_MONTH]->(m:Month {month: 2})-[:HAS_DAY]->(:Day {day: 1})<-[:NEXT*0..5]-(day) RETURN y,m,day If we want to we can create the time tree and then connect the day nodes all in one query by using ‘WITH *’ like so: WITH range(2011, 2014) AS years, range(1,12) as months FOREACH(year IN years | MERGE (y:Year {year: year}) FOREACH(month IN months | CREATE (m:Month {month: month}) MERGE (y)-[:HAS_MONTH]->(m) FOREACH(day IN (CASE WHEN month IN [1,3,5,7,8,10,12] THEN range(1,31) WHEN month = 2 THEN CASE WHEN year % 4 <> 0 THEN range(1,28) WHEN year % 100 <> 0 THEN range(1,29) WHEN year % 400 <> 0 THEN range(1,29) ELSE range(1,28) END ELSE range(1,30) END) | CREATE (d:Day {day: day}) MERGE (m)-[:HAS_DAY]->(d)))) WITH * MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day) WITH year,month,day ORDER BY year.year, month.month, day.day WITH collect(day) as days FOREACH(i in RANGE(0, length(days)-2) | FOREACH(day1 in [days[i]] | FOREACH(day2 in [days[i+1]] | CREATE UNIQUE (day1)-[:NEXT]->(day2)))) Now I need to connect the RSVP events to the tree!

I was playing around with Neo4j’s Cypher last weekend and found myself accidentally running some queries against an earlier version of the Neo4j 2.0 series (2.0.0). My first query started with a map and I wanted to create a person from an identifier inside the map: WITH {person: {id: 1} AS params MERGE (p:Person {id: params.person.id}) RETURN p When I ran the query I got this error: ==> SyntaxException: Type mismatch: expected Map but was Boolean, Number, String or Collection (line 1, column 62) ==> "WITH {person: {id: 1} AS params MERGE (p:Person {id: params.person.id}) RETURN p" If we try the same query in 2.0.1 it works as we’d expect: ==> +---------------+ ==> | p | ==> +---------------+ ==> | Node[1]{id:} | ==> +---------------+ ==> 1 row ==> Nodes created: 1 ==> Properties set: 1 ==> Labels added: 1 ==> 47 ms My next query was the following which links topics of interest to a person: WITH {topics: [{name: "Java"}, {name: "Neo4j"}]} AS params MERGE (p:Person {id: 2}) FOREACH(t IN params.topics | MERGE (topic:Topic {name: t.name}) MERGE (p)-[:INTERESTED_IN]->(topic) ) RETURN p In 2.0.0 that query fails like so: ==> InternalException: Query not prepared correctly! but if we try it in 2.0.1 we’ll see that it works as well: ==> +---------------+ ==> | p | ==> +---------------+ ==> | Node[4]{id:2} | ==> +---------------+ ==> 1 row ==> Nodes created: 1 ==> Relationships created: 2 ==> Properties set: 1 ==> Labels added: 1 ==> 53 ms So if you’re seeing either of those errors, then get yourself upgraded to 2.0.1 as well!

Running Groovy scripts with GroovyShell is easy. We can for example incorporate a Domain Specific Language (DSL) in our application where the DSL is expressed in Groovy code and executed by GroovyShell. To limit the constructs that can be used in the DSL (which is Groovy code) we can apply a SecureASTCustomizer to the GroovyShell configuration. With the SecureASTCustomizer the Abstract Syntax Tree (AST) is inspected, we cannot define runtime checks here. We can for example disallow the definition of closures and methods in the DSL script. Or we can limit the tokens to be used to just a plus or minus token. To have even more control we can implement the StatementChecker andExpressionChecker interface to determine if a specific statement or expression is allowed or not. In the following sample we first use the properties of the SecureASTCustomizer class to define what is possible and not within the script: package com.mrhaki.blog import org.codehaus.groovy.control.customizers.SecureASTCustomizer import org.codehaus.groovy.control.CompilerConfiguration import org.codehaus.groovy.ast.stmt.* import org.codehaus.groovy.ast.expr.* import org.codehaus.groovy.control.MultipleCompilationErrorsException import static org.codehaus.groovy.syntax.Types.PLUS import static org.codehaus.groovy.syntax.Types.MINUS import static org.codehaus.groovy.syntax.Types.EQUAL // Define SecureASTCustomizer to limit allowed // language syntax in scripts. final SecureASTCustomizer astCustomizer = new SecureASTCustomizer( // Do not allow method creation. methodDefinitionAllowed: false, // Do not allow closure creation. closuresAllowed: false, // No package allowed. packageAllowed: false, // White or blacklists for imports. importsBlacklist: ['java.util.Date'], // or importsWhitelist staticImportsWhitelist: [], // or staticImportBlacklist staticStarImportsWhitelist: [], // or staticStarImportsBlacklist // Make sure indirect imports are restricted. indirectImportCheckEnabled: true, // Only allow plus and minus tokens. tokensWhitelist: [PLUS, MINUS, EQUAL], // or tokensBlacklist // Disallow constant types. constantTypesClassesWhiteList: [Integer, Object, String], // or constantTypesWhiteList // or constantTypesBlackList // or constantTypesClassesBlackList // Restrict method calls to whitelisted classes. // receiversClassesWhiteList: [], // or receiversWhiteList // or receiversClassesBlackList // or receiversBlackList // Ignore certain language statement by // whitelisting or blacklisting them. statementsBlacklist: [IfStatement], // or statementsWhitelist // Ignore certain language expressions by // whitelisting or blacklisting them. expressionsBlacklist: [MethodCallExpression] // or expresionsWhitelist ) // Add SecureASTCustomizer to configuration for shell. final conf = new CompilerConfiguration() conf.addCompilationCustomizers(astCustomizer) // Create shell with given configuration. final shell = new GroovyShell(conf) // All valid script. final result = shell.evaluate ''' def s1 = 'Groovy' def s2 = 'rocks' "$s1 $s2!" ''' assert result == 'Groovy rocks!' // Some invalid scripts. try { // Importing [java.util.Date] is not allowed shell.evaluate ''' new Date() ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('Indirect import checks prevents usage of expression') } try { // MethodCallExpression not allowed shell.evaluate ''' println "Groovy rocks!" ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('MethodCallExpressions are not allowed: this.println(Groovy rocks!)') } To have more fine-grained control on which statements and expression are allowed we can implement the StatementChecker andExpressionChecker interfaces. These interfaces have one method isAuthorized with a boolean return type. We return true if a statement or expression is allowed and false if not. package com.mrhaki.blog import org.codehaus.groovy.control.customizers.SecureASTCustomizer import org.codehaus.groovy.control.CompilerConfiguration import org.codehaus.groovy.ast.stmt.* import org.codehaus.groovy.ast.expr.* import org.codehaus.groovy.control.MultipleCompilationErrorsException import static org.codehaus.groovy.control.customizers.SecureASTCustomizer.ExpressionChecker import static org.codehaus.groovy.control.customizers.SecureASTCustomizer.StatementChecker // Define SecureASTCustomizer. final SecureASTCustomizer astCustomizer = new SecureASTCustomizer() // Define expression checker to deny // usage of variable names with length of 1. def smallVariableNames = { expr -> if (expr instanceof VariableExpression) { expr.variable.size() > 1 } else { true } } as ExpressionChecker astCustomizer.addExpressionCheckers smallVariableNames // In for loops the collection name // can only be 'names'. def forCollectionNames = { statement -> if (statement instanceof ForStatement) { statement.collectionExpression.variable == 'names' } else { true } } as StatementChecker astCustomizer.addStatementCheckers forCollectionNames // Add SecureASTCustomizer to configuration for shell. final CompilerConfiguration conf = new CompilerConfiguration() conf.addCompilationCustomizers(astCustomizer) // Create shell with given configuration. final GroovyShell shell = new GroovyShell(conf) // All valid script. final result = shell.evaluate ''' def names = ['Groovy', 'Grails'] for (name in names) { print "$name rocks! " } def s1 = 'Groovy' def s2 = 'rocks' "$s1 $s2!" ''' assert result == 'Groovy rocks!' // Some invalid scripts. try { // Variable s has length 1, which is not allowed. shell.evaluate ''' def s = 'Groovy rocks' s ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('Expression [VariableExpression] is not allowed: s') } try { // Only names as collection expression is allowed. shell.evaluate ''' def languages = ['Groovy', 'Grails'] for (name in languages) { println "$name rocks!" } ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('Statement [ForStatement] is not allowed') } Code written with Groovy 2.2.2.

It seems that I'm finally getting close to the end of this series of blogs on Error Tracking using Spring and for those who haven’t read any blogs in the series I’m writing a simple, but almost industrial strength, Spring application that scans for exceptions in log files and then generates a report. From the first blog in the series, these were my initial requirements: Search a given directory and its sub-directories (possibly) looking for files of a particular type. If a file is found then check its date: does it need to be searched for errors? If the file is young enough to be checked then validate it, looking for exceptions. If it contains exceptions, are they the ones we’re looking for or have they been excluded? If it contains the kind of exceptions we’re after, then add the details to a report. When all the files have been checked, format the report ready for publishing. Publish the report using email or some other technique. The whole thing will run at a given time every day This blog takes a look at meeting requirement number 8: "The whole thing will run at a given time every day" and this means implementing some kind of scheduling. Now, Java has been around for what seems like a very long time, which means that there are a number of ways of scheduling a task. These range from: Using a simple thread with a long sleep(...). Using Timer and TimerTask objects. Using a ScheduledExecutorService. Using Spring’s TaskExecutor and TaskScheduler classes. Using Spring’s @EnableScheduling and @Scheduled annotations (Spring 3.1 onwards). Using a more professional schedular. The more professional variety of schedulers range from Quartz (free) to Obsidian (seemingly much more advanced, but costs money). Spring, as you might expect, includes Quartz Scheduler support; in fact there are two ways of integrating the Quartz Scheduler into your Spring app and these are: Using a JobDetailBean Using a MethodInvokingJobDetailFactoryBean. For this application, I’m using the Spring’s Quartz integration together with a MethodInvokingJobDetailFactoryBean; the reason is that using Quartz allows me to configure my schedule using a a cron expression and MethodInvokingJobDetailFactoryBean can be configured quickly and simply using a few lines of XML. The cron expression technique used by Spring and Quartz has been shamelessly taken from Unix’s cron scheduler. For more information on how Quartz deals with cron expressions, take a look at the Quartz cron page. If you need help in creating your own cron expressions then you’ll find that Cron Maker is a really useful utility. The first thing to do when setting up Spring and Quartz is to include the following dependencies to your POM project file: org.springframework spring-context-support ${org.springframework-version} commons-logging commons-logging org.springframework spring-tx ${org.springframework-version} org.quartz-scheduler quartz 1.8.6 This is fairly straight forward with one tiny ’Gotcha’ at the end. Firstly Spring’s Quartz support is located in the spring-context-support-3.2.7.RELEASE.jar (substitute your Spring version number as applicable). Secondly, you also need to include the Spring transaction library - spring-td-3.2.7.RELEASE.jar. Lastly, you need to include a version of the Quartz scheduler; however, be careful as Spring 3.x and Quartz 2.x do not work together "out of the box" (although if you look around there are ad-hoc fixes to be found). I've used Quartz version 1.8.6, which does exactly what I need it to do. The next thing to do is to sort out the XML configuration and this involves three steps: Create an instance of a MethodInvokingJobDetailFactoryBean. This has two properties: the name of the bean that you want to call at a scheduled interval and the name of the method on that bean that you want to invoke. Couple the MethodInvokingJobDetailFactoryBean to a cron expression using a CronTriggerFactoryBean Finally, schedule the whole caboodle using a SchedulerFactoryBean Having configured these three beans, you get some XML that looks something like this: Note that I’ve use a place-holder for my cron expression. The actual cron expression can be found in the app.properties file: # run every morning at 2 AM cron.expression=0 0 2 * * ? # Use this to test the app (every minute) #cron.expression=0 0/1 * * * ? Here, I’ve got two expressions: one that schedules the job to run at 2AM every morning and another, commented out, that runs the job every minute. This is an instance of the app not quite being industrial strength. If there were a 'proper' app then I'd probably be using a different set of properties in every environment (DEV, UAT and production etc.). There are only a couple of steps left before this app can be released and the first one of these is creating an executable JAR file. More on that next time. The code for this blog is available on Github at: https://github.com/roghughe/captaindebug/tree/master/error-track. If you want to look at other blogs in this series take a look here... Tracking Application Exceptions With Spring Tracking Exceptions With Spring - Part 2 - Delegate Pattern Error Tracking Reports - Part 3 - Strategy and Package Private Tracking Exceptions - Part 4 - Spring's Mail Sender

Testing is crucial. While many different kinds and levels of testing exist, there’s good library support only for unit tests (the Python unittest package and its moral equivalents in other languages). However, unit testing does not cover all kinds of testing we may want to do – for example, all kinds of whole program tests and integration tests. This is where we usually end up with a custom "test runner" script. Having written my share of such custom test runners, I’ve recently gravitated towards a very convenient approach which I want to share here. In short, I’m actually using Python’s unittest, combined with the dynamic nature of the language, to run all kinds of tests. Let’s assume my tests are some sort of data files which have to be fed to a program. The output of the program is compared to some "expected results" file, or maybe is encoded in the data file itself in some way. The details of this are immaterial, but seasoned programmers usually encounter such testing rigs very frequently. It commonly comes up when the program under test is a data-transformation mechanism of some sort (compiler, encryptor, encoder, compressor, translator etc.) So you write a "test runner". A script that looks at some directory tree, finds all the "test files" there, runs each through the transformation, compares, reports, etc. I’m sure all these test runners share a lot of common infrastructure – I know that mine do. Why not employ Python’s existing "test runner" capabilities to do the same? Here’s a very short code snippet that can serve as a template to achieve this: import unittest class TestsContainer(unittest.TestCase): longMessage = True def make_test_function(description, a, b): def test(self): self.assertEqual(a, b, description) return test if __name__ == '__main__': testsmap = { 'foo': [1, 1], 'bar': [1, 2], 'baz': [5, 5]} for name, params in testsmap.iteritems(): test_func = make_test_function(name, params[0], params[1]) setattr(TestsContainer, 'test_{0}'.format(name), test_func) unittest.main() What happens here: The test class TestsContainer will contain dynamically generated test methods. make_test_function creates a test function (a method, to be precise) that compares its inputs. This is just a trivial template – it could do anything, or there can be multiple such "makers" fur multiple purposes. The loop creates test functions from the data description in testmap and attaches them to the test class. Keep in mind that this is a very basic example. I hope it’s obvious that testmap could really be test files found on disk, or whatever else. The main idea here is the dynamic test method creation. So what do we gain from this, you may ask? Quite a lot. unittest is powerful – armed to its teeth with useful tools for testing. You can now invoke tests from the command line, control verbosity, control "fast fail" behavior, easily filter which tests to run and which not to run, use all kinds of assertion methods for readability and reporting (why write your own smart list comparison assertions?). Moreover, you can build on top of any number of third-party tools for working with unittest results – HTML/XML reporting, logging, automatic CI integration, and so on. The possibilities are endless. One interesting variation on this theme is aiming the dynamic generation at a different testing "layer". unittest defines any number of "test cases" (classes), each with any number of "tests" (methods). In the code above, we generate a bunch of tests into a single test case. Here’s a sample invocation to see this in action: $ python dynamic_test_methods.py -v test_bar (__main__.TestsContainer) ... FAIL test_baz (__main__.TestsContainer) ... ok test_foo (__main__.TestsContainer) ... ok ====================================================================== FAIL: test_bar (__main__.TestsContainer) ---------------------------------------------------------------------- Traceback (most recent call last): File "dynamic_test_methods.py", line 8, in test self.assertEqual(a, b, description) AssertionError: 1 != 2 : bar ---------------------------------------------------------------------- Ran 3 tests in 0.001s FAILED (failures=1) As you can see, all data pairs in testmap are translated into distinctly named test methods within the single test case TestsContainer. Very easily, we can cut this a different way, by generating a whole test case for each data item: import unittest class DynamicClassBase(unittest.TestCase): longMessage = True def make_test_function(description, a, b): def test(self): self.assertEqual(a, b, description) return test if __name__ == '__main__': testsmap = { 'foo': [1, 1], 'bar': [1, 2], 'baz': [5, 5]} for name, params in testsmap.iteritems(): test_func = make_test_function(name, params[0], params[1]) klassname = 'Test_{0}'.format(name) globals()[klassname] = type(klassname, (DynamicClassBase,), {'test_gen_{0}'.format(name): test_func}) unittest.main() Most of the code here remains the same. The difference is in the lines within the loop: now instead of dynamically creating test methods and attaching them to the test case, we create whole test cases – one per data item, with a single test method. All test cases derive from DynamicClassBase and hence from unittest.TestCase, so they will be auto-discovered by the unittest machinery. Now an execution will look like this: $ python dynamic_test_classes.py -v test_gen_bar (__main__.Test_bar) ... FAIL test_gen_baz (__main__.Test_baz) ... ok test_gen_foo (__main__.Test_foo) ... ok ====================================================================== FAIL: test_gen_bar (__main__.Test_bar) ---------------------------------------------------------------------- Traceback (most recent call last): File "dynamic_test_classes.py", line 8, in test self.assertEqual(a, b, description) AssertionError: 1 != 2 : bar ---------------------------------------------------------------------- Ran 3 tests in 0.000s FAILED (failures=1) Why would you want to generate whole test cases dynamically rather than just single tests? It all depends on your specific needs, really. In general, test cases are better isolated and share less, than tests within one test case. Moreover, you may have a huge amount of tests and want to use tools that shard your tests for parallel execution – in this case you almost certainly need separate test cases. I’ve used this technique in a number of projects over the past couple of years and found it very useful; more than once, I replaced a whole complex test runner program with about 20-30 lines of code using this technique, and gained access to many more capabilities for free. Python’s built-in test discovery, reporting and running facilities are very powerful. Coupled with third-party tools they can be even more powerful. Leveraging all this power for any kind of testing, and not just unit testing, is possible with very little code, due to Python’s dynamism. I hope you find it useful too.

See how HashMap performance has been improved with new features of Java 8.

Recently needed to update the last inserted row of a table but didn't have anyway in knowing what the highest ID in the table was. I can easily do this by using the max() function to select the highest ID in the table. SELECT MAX(id) FROM table; Then I can use the result of this query in the UPDATE query to edit the record with the highest ID. But this is quite a easy query so I should be able to do this in one query by using a nested select query on the UPDATE. UPDATE table SET name='test_name' WHERE id = (SELECT max(id) FROM table) But the problem with this is that the MAX() function doesn't work inside a nested select so had to find another way of doing this. I found out that you can use an ORDER BY and a LIMIT in an UPDATE query therefore I can use a combination of these in the UPDATE query to make sure I only update the record with the highest ID, by doing a descendant order on the ID and limiting the return to only 1 record. UPDATE table SET name='test_name' ORDER BY id DESC LIMIT 1;

with the recent release of java 8, developers are still just beginning to asses the strengths and weaknesses of the new platform. the most pressing question is: does java 8 have the fastest jvm so far? a good way to asses the progress of java 8 is to test its ability to work with something that was new to java 7... fork/join. oleg shelajev uses the "infamous" java microbenchmark harness project (jmh) to create a benchmark test for the two most recent versions of java. but before implementing the benchmark, he takes the time to give a brief overview of fork/join and how it changes between java 7 and 8. here is a graph of the results : based on these results, oleg recommends taking a chance and upgrading to java 8, especially if you are working with mapreducing or fork/join. this is his interpretation of the data that lead him to that conclusion: one can see that the baseline results, which show the throughput of running the math directly in a single thread do not differ between the jdk 7 and 8. however, when we include the overhead of managing recursive tasks and going through a forkjoin execution then java 8 is much faster. the numbers for this simple benchmark suggest that the overhead of managing forkjoin tasks is around 35% more performant in the latest release. check out the full article ! it is very informative, has great visuals, and tackles complexity with clarity.