It's not always immediate why and in which part of the program is Hibernate generating a given SQL query, especially if we are dealing with code that we did not write ourselves. This post will go over how to configure Hibernate query logging, and use that together with other tricks to find out why and where in the program a given query is being executed. What does the Hibernate query log look like Hibernate has built-in query logging that looks like this: select /* load your.package.Employee */ this_.code, ... from employee this_ where this_.employee_id=? TRACE 12-04-2014@16:06:02 BasicBinder - binding parameter [1] as [NUMBER] - 1000 Why can't Hibernate log the actual query ? Notice that what is logged by Hibernate is the prepared statement sent by Hibernate to the JDBC driver plus it's parameters. The prepared statement has ? in the place of the query parameters, and the parameter values themselves are logged just bellow the prepared statement. This is not the same as the actual query sent to the database, as there is no way for Hibernate to log the actual query. The reason for this is that Hibernate only knows about the prepared statements and the parameters that it sends to the JDBC driver, and it's the driver that will build the actual queries and then send them to the database. In order to produce a log with the real queries, a tool like log4jdbc is needed, which will be the subject of another post. How to find out the origin of the query The logged query above contains a comment that allows to identify in most cases the origin of the query: if the query is due to a load by ID the comment is /* load your.entity.Name */, if it's a named query then the comment will contain the name of the query. If it's a one to many lazy initialization the comment will contain the name of the class and the property that triggered it, etc. In many cases the query comment created by is enough to identify the origin of the query. Setting up the Hibernate query log In order to obtain a query log, the following flags need to be set in the configuration of the session factory: ... true true true The example above is for Spring configuration of an entity manager factory. This is the meaning of the flags: show_sql enables query logging format_sql pretty prints the SQL use_sql_comments adds an explanatory comment In order to log the query parameters, the following log4j or equivalent configuration is needed: If everything else fails If the query comment added by the option use_sql_comments is not sufficient, then we can start by identifying the entity returned by the query based on the table names involved, and put a breakpoint in the constructor of the returned entity. If the entity does not have a constructor, then we can create one and put the breakpoint in the call to super(): @Entity public class Employee { public Employee() { super(); // put the breakpoint here } ... } When the breakpoint is hit, go to the IDE debug view containing the stack call of the program and go through it from top to bottom. The place where the query was made in the program will be there in the call stack.

Groovy closures are super cool. To fully understand them, I think it's really important to understand the meaning of this, owner and delegate. In general: this: refers to the instance of the class that the closure was defined in. owner: is the same as this, unless the closure was defined inside another closure in which case the owner refers to the outer closure. delegate: is the same as owner. But, it is the only one that can be programmatically changed, and it is the one that makes Groovy closures really powerful. Confused? Let's look at some code. class MyClass { def outerClosure = { println this.class.name // outputs MyClass println owner.class.name // outputs MyClass println delegate.class.name //outputs MyClass def nestedClosure = { println this.class.name // outputs MyClass println owner.class.name // outputs MyClass$_closure1 println delegate.class.name // outputs MyClass$_closure1 } nestedClosure() } } def closure = new MyClass().closure closure() With respect to above code: The this value always refers to the instance of the enclosing class. owner is always the same as this, except for nested closures. delegate is the same as owner by default. It can be changed and we will see that in a sec. So what is the point of this, owner, delegate? Well remember, that closures are not just anonymous functions. If they were we could just call them Lambdas and we wouldn't have to come up with another word, would we? Where closures go beyond lambdas is that they bind or "close over" variables that are not explicitly defined in the closure's scope. Again, let's take a look at some code. class MyClass { String myString = "myString1" def outerClosure = { println myString; // outputs myString1 def nestedClosure = { println myString; // outputs myString1 } nestedClosure() } } MyClass myClass = new MyClass() def closure = new MyClass().outerClosure closure() println myClass.myString Ok, so both the closure and the nestedClosure have access to variables on the instance of the class they were defined in. That's obvious. But, how exactly do they resolve the myString reference? Well it's like this. If the variable was not defined explicitly in the closure, the this scope is then checked, then the owner scope and then the delegatescope. In this example, myString is not defined in either of the closures, so groovy checks their this references and sees the myString is defined there and uses that. Ok, let's take a look at an example where it can't find a variable in the closure and can't find it on the closure's this scope, but it can find's it in the closure's owner scope. class MyClass { def outerClosure = { def myString = "outerClosure"; def nestedClosure = { println myString; // outputs outerClosure } nestedClosure() } } MyClass myClass = new MyClass() def closure = new MyClass().closure closure() In this case, Groovy can't find myString in the nestedClosure or in the this scope. It then checks the owner scope, which for the nestedClosure is the outerClosure. It finds myString there and uses that. Now, let's take a look at an example where Groovy can't find a variable in the closure, or on this or the owner scope but can find it in on closure'sdelegate scope. As discussed earlier the owner delegate scope is the same as the owner scope, unless it is explicitly changed. So, to make this a bit more interesting, let's change the delegate. class MyOtherClass { String myString = "I am over in here in myOtherClass" } class MyClass { def closure = { println myString } } MyClass myClass = new MyClass() def closure = new MyClass().closure closure.delegate = new MyOtherClass() closure() // outputs: "I am over in here in myOtherClass" The ability to have so much control over the lexical scope of closures in Groovy gives enormous power. Even when the delegate is set it can be change to something else, this means we can make the behavior of the closure super dynamic. class MyOtherClass { String myString = "I am over in here in myOtherClass" } class MyOtherClass2 { String myString = "I am over in here in myOtherClass2" } class MyClass { def closure = { println myString } } MyClass myClass = new MyClass() def closure = new MyClass().closure closure.delegate = new MyOtherClass() closure() // outputs: I am over in here in myOtherClass closure = new MyClass().closure closure.delegate = new MyOtherClass2() closure() // outputs: I am over in here in myOtherClass2 Ok, so it should be a bit clearer now what this, owner and delegate actually correspond to. As stated, the closure itself will be checked first, followed by the closure's this scope, than the closure's owner, then its delegate. However, Groovy is so flexible this strategy can be changed. Every closure has a property called resolvedStrategy. This can be set to: Closure.OWNER_FIRST Closure.DELEGATE_FIRST Closure.OWNER_ONLY Closure.DELEGATE_ONLY So where is a good example of a practical usage of the dynamic setting of the delegate property. Well you see in the GORM for Grails. Suppose we have the following domain class: class Author { String name static constraints = { name size: 10..15 } } In the Author class we can see a constraint defined using what looks like a DSL whereas in the Java / Hibernate world we would not being able to write an expressive DSL and instead use an annotation (which is better than XML but still not as neat as a DSL). So, how come we can use a DSL in Groovy then? Well it is because of the capabilities delegate setting on closures adds to Groovy's metaprogramming toolbox. In the Author GORM object, constraints is a closure, that invokes a name method with one parameter of name size which has the value of the range between 10 and 15. It could also be written less DSL'y as: class Author { String name static constraints = { name(size: 10..15) } } Either way, behind the scenes, Grails looks for a constraints closure and assigns its delegate to a special object that synthesizes the constraints logic. In pseudo code, it would be something like this... // Set the constraints delegate constraints.delegate = new ConstraintsBuilder(); // delegate is assigned before the closure is executed. class ConstraintsBuilder = { // // ... // In every Groovy object methodMissing() is invoked when a method that does not exist on the object is invoked // In this case, there is no name() method so methodMissing will be invoked. // ... def methodMissing(String methodName, args) { // We can get the name variable here from the method name // We can get that size is 10..15 from the args ... // Go and do stuff with hibernate to enforce constraints } } So there you have it. Closures are very powerful, they can delegate out to objects that can be set dynamically at runtime. That plays an important part in Groovy's meta programming capabilities which mean that Groovy can have some very expressive DSLs.

In a previous post, we had shared a small function that generated random string in Java. It turns out that similar functionality is available from a class in the extremely useful apache commons lang library. If you are using maven, download the jar using the following dependency: commons-lang commons-lang 20030203.000129 The class we are interested in is RandomStringUtils. Listed below are some functions you may find useful. Generate and print a random string of length 5 from all characters available System.out.println(RandomStringUtils.random(5)); Generate and print random string of length 10 from upper and lower case alphabets System.out.println(RandomStringUtils.randomAlphabetic(10)); Generate and print a random number of length 12 System.out.println(RandomStringUtils.randomNumeric(12)); Generate and print a random string of length 5 using only a, b, c and d characters System.out.println(RandomStringUtils.random(10,new char[]{'a','b','c','d'}));

I have been using Spring Scala for a toy project for the last few days and I have to say that it is a fantastic project, it simplifies Spring configuration even further when compared to the already simple configuration purely based on Spring Java Config. Let me demonstrate this by starting with the Cake Pattern based sample here: // ======================= // service interfaces trait OnOffDeviceComponent { val onOff: OnOffDevice trait OnOffDevice { def on: Unit def off: Unit } } trait SensorDeviceComponent { val sensor: SensorDevice trait SensorDevice { def isCoffeePresent: Boolean } } // ======================= // service implementations trait OnOffDeviceComponentImpl extends OnOffDeviceComponent { class Heater extends OnOffDevice { def on = println("heater.on") def off = println("heater.off") } } trait SensorDeviceComponentImpl extends SensorDeviceComponent { class PotSensor extends SensorDevice { def isCoffeePresent = true } } // ======================= // service declaring two dependencies that it wants injected trait WarmerComponentImpl { this: SensorDeviceComponent with OnOffDeviceComponent => class Warmer { def trigger = { if (sensor.isCoffeePresent) onOff.on else onOff.off } } } // ======================= // instantiate the services in a module object ComponentRegistry extends OnOffDeviceComponentImpl with SensorDeviceComponentImpl with WarmerComponentImpl { val onOff = new Heater val sensor = new PotSensor val warmer = new Warmer } // ======================= val warmer = ComponentRegistry.warmer warmer.trigger Cake pattern is a pure Scala way of specifying the dependencies. Now, if we were to specify this dependency using Spring's native Java config, but with Scala as the language, firs to define the components that need to be wired together: trait SensorDevice { def isCoffeePresent: Boolean } class PotSensor extends SensorDevice { def isCoffeePresent = true } trait OnOffDevice { def on: Unit def off: Unit } class Heater extends OnOffDevice { def on = println("heater.on") def off = println("heater.off") } class Warmer(s: SensorDevice, o: OnOffDevice) { def trigger = { if (s.isCoffeePresent) o.on else o.off } } and the configuration with Spring Java Config and a sample which makes use of this configuration: import org.springframework.context.annotation.Configuration import org.springframework.context.annotation.Bean @Configuration class WarmerConfig { @Bean def heater(): OnOffDevice = new Heater @Bean def potSensor(): SensorDevice = new PotSensor @Bean def warmer() = new Warmer(potSensor(), heater()) } import org.springframework.context.annotation.AnnotationConfigApplicationContext val ac = new AnnotationConfigApplicationContext(classOf[WarmerConfig]) val warmer = ac.getBean("warmer", classOf[Warmer]) warmer.trigger Taking this further to use Spring-Scala project to specify the dependencies, the configuration and a sample look like this: import org.springframework.context.annotation.Configuration import org.springframework.context.annotation.Bean @Configuration class WarmerConfig { @Bean def heater(): OnOffDevice = new Heater @Bean def potSensor(): SensorDevice = new PotSensor @Bean def warmer() = new Warmer(potSensor(), heater()) } import org.springframework.context.annotation.AnnotationConfigApplicationContext val ac = new AnnotationConfigApplicationContext(classOf[WarmerConfig]) val warmer = ac.getBean("warmer", classOf[Warmer]) warmer.trigger The essence of the Spring Scala project as explained in this wiki is the "bean" method derived from the `FunctionalConfiguration` trait, this method can be called to create a bean, passing in parameters to specify, if required, bean name, alias, scope and a function which returns the instantiated bean. This sample hopefully gives a good appreciation for how simple Spring Java Config is, and how much more simpler Spring-Scala project makes it for Scala based projects.

this post was originally written by muhammad irfan here on the percona mysql support team, we often see issues where a customer is complaining about replication delays – and many times the problem ends up being tied to mysql replication slave lag. this of course is nothing new for mysql users and we’ve had a few posts here on the mysql performance blog on this topic over the years (two particularly popular post in the past were: “ reasons for mysql replication lag ” and “ managing slave lag with mysql replication ,” both by percona ceo peter zaitsev) . in today’s post, however, i will share some new ways of identifying delays in replication – including possible causes of lagging slaves – and how to cure this problem. how to identify replication delay mysql replication works with two threads, io_thread & sql_thread. io_thread connects to a master, reads binary log events from the master as they come in and just copies them over to a local log file called relaylog . on the other hand, sql_thread reads events from a relay log stored locally on the replication slave (the file that was written by io thread) and then applies them as fast as possible. whenever replication delays, it’s important to discover first whether it’s delaying on slave io_thread or slave sql_thread. normally, i/o thread would not cause a huge replication delay as it is just reading the binary logs from the master. however, it depends on the network connectivity, network latency… how fast is that between the servers. the slave i/o thread could be slow because of high bandwidth usage. usually, when the slave io_thread is able to read binary logs quickly enough it copies and piles up the relay logs on the slave – which is one indication that the slave io_thread is not the culprit of slave lag. on the other hand, when the slave sql_thread is the source of replication delays it is probably because of queries coming from the replication stream are taking too long to execute on the slave. this is sometimes because of different hardware between master/slave, different schema indexes, workload. moreover, the slave oltp workload sometimes causes replication delays because of locking. for instance, if a long-running read against a myisam table blocks the sql thread, or any transaction against an innodb table creates an ix lock and blocks ddl in the sql thread. also, take into account that slave is single threaded prior to mysql 5.6, which would be another reason for delays on the slave sql_thread. let me show you via master status/slave status example to identify either slave is lagging on slave io_thread or slave sql_thread. mysql-master> show master status; +------------------+--------------+------------------+------------------------------------------------------------------+ | file | position | binlog_do_db | binlog_ignore_db | executed_gtid_set | +------------------+--------------+------------------+------------------------------------------------------------------+ | mysql-bin.018196 | 15818564 | | bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:1-2331947 | +------------------+--------------+------------------+------------------------------------------------------------------+ mysql-slave> show slave status\g *************************** 1. row *************************** slave_io_state: queueing master event to the relay log master_host: master.example.com master_user: repl master_port: 3306 connect_retry: 60 master_log_file: mysql-bin.018192 read_master_log_pos: 10050480 relay_log_file: mysql-relay-bin.001796 relay_log_pos: 157090 relay_master_log_file: mysql-bin.018192 slave_io_running: yes slave_sql_running: yes replicate_do_db: replicate_ignore_db: replicate_do_table: replicate_ignore_table: replicate_wild_do_table: replicate_wild_ignore_table: last_errno: 0 last_error: skip_counter: 0 exec_master_log_pos: 5395871 relay_log_space: 10056139 until_condition: none until_log_file: until_log_pos: 0 master_ssl_allowed: no master_ssl_ca_file: master_ssl_ca_path: master_ssl_cert: master_ssl_cipher: master_ssl_key: seconds_behind_master: 230775 master_ssl_verify_server_cert: no last_io_errno: 0 last_io_error: last_sql_errno: 0 last_sql_error: replicate_ignore_server_ids: master_server_id: 2 master_uuid: bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:2-973166 master_info_file: /var/lib/mysql/i1/data/master.info sql_delay: 0 sql_remaining_delay: null slave_sql_running_state: reading event from the relay log master_retry_count: 86400 master_bind: last_io_error_timestamp: last_sql_error_timestamp: master_ssl_crl: master_ssl_crlpath: retrieved_gtid_set: bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:2-973166 executed_gtid_set: bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:2-973166, ea75c885-c2c5-11e3-b8ee-5cf3fcfc9640:1-1370 auto_position: 1 this clearly suggests that the slave io_thread is lagging and obviously because of that the slave sql_thread is lagging, too, and it yields replication delays. as you can see the master log file is mysql-bin.018196 (file parameter from master status) and slave io_thread is on mysql-bin.018192 ( master_log_file from slave status ) which indicates slave io_thread is reading from that file, while on master it’s writing on mysql-bin.018196 , so the slave io_thread is behind by 4 binlogs. meanwhile, the slave sql_thread is reading from same file i.e. mysql-bin.01819 2 (relay_master_log_file from slave status) this indicates that the slave sql_thread is applying events fast enough, but it’s lagging too, which can be observed from the difference between read_master_log_pos & exec_master_log_pos from show slave status output. you can calculate slave sql_thread lag from read_master_log_pos – exec_master_log_pos in general as long as master_log_file parameter output from show slave status and relay_master_log_file parameter from show slave status output are the same. this will give you rough idea how fast slave sql_thread is applying events. as i mentioned above, the slave io_thread is lagging as in this example then off course slave sql_thread is behind too. you can read detailed description of show slave status output fields here. also, the seconds_behind_master parameter shows a huge delay in seconds. however, this can be misleading, because it only measures the difference between the timestamps of the relay log most recently executed, versus the relay log entry most recently downloaded by the io_thread. if there are more binlogs on the master, the slave doesn’t figure them into the calculation of seconds_behind_master. you can get a more accurate measure of slave lag using pt-heartbeat from percona toolkit. so, we learned how to check replication delays – either it’s slave io_thread or slave sql_thread. now, let me provide some tips and suggestions for what exactly causing this delay. tips and suggestions what causing replication delay & possible fixes usually, the slave io_thread is behind because of slow network between master/slave. most of the time, enabling slave_compressed_protocol helps to mitigate slave io_thread lag. one other suggestion is to disable binary logging on slave as it’s io intensive too unless you required it for point in time recovery. to minimize slave sql_thread lag, focus on query optimization. my recommendation is to enable the configuration option log_slow_slave_statements so that the queries executed by slave that take more than long_query_time will be logged to the slow log. to gather more information about query performance, i would also recommend setting the configuration option log_slow_verbosity to “full”. this way we can see if there are queries executed by slave sql_thread that are taking long time to complete. you can follow my previous post about how to enable slow query log for specific time period with mentioned options here . and as a reminder, log_slow_slave_statements as variable were first introduced in percona server 5.1 which is now part of vanilla mysql from version 5.6.11 in upstream version of mysql server log_slow_slave_statements were introduced as command line option. details can be found here while log_slow_verbosity is percona server specific feature. one another reason for delay on slave sql_thread if you use row based binlog format is that if your any database table missing primary key or unique key then it will scan all rows of the table for dml on slave and causes replication delays so make sure all your tables should have primary key or unique key. check this bug report for details http://bugs.mysql.com/bug.php?id=53375 you can use below query on slave to identify which of database tables missing primary or unique key. mysql> select t.table_schema,t.table_name,engine from information_schema.tables t inner join information_schema .columns c on t.table_schema=c.table_schema and t.table_name=c.table_name group by t.table_schema,t.table_name having sum(if(column_key in ('pri','uni'), 1,0)) =0; one improvement is made for this case in mysql 5.6, where in memory hash is used slave_rows_search_algorithms comes to the rescue. note that seconds_behind_master is not updated while we read huge rbr event, so, “lagging” may be related to just that – we had not completed reading of the event. for example, in row based replication huge transactions may cause delay on slave side e.g. if you have 10 million rows table and you do “delete from table where id < 5000000″ 5m rows will be sent to slave, each row separately which will be painfully slow. so, if you have to delete oldest rows time to time from huge table using partitioning might be good alternative for this for some kind of workloads where instead using delete use drop old partition may be good and only statement is replicated because it will be ddl operation. to explain it better, let suppose you have partition1 holding rows of id’s from 1 to 1000000 , partition2 – id’s from 1000001 to 2000000 and so on so instead of deleting via statement “delete from table where id<=1000000;” you can do “alter table drop partition1;” instead. for alter partitioning operations check manual – check this wonderful post too from my colleague roman explaining possible grounds for replication delays here pt-stalk is one of finest tool from percona toolkit which collects diagnostics data when problems occur. you can setup pt-stalk as follows so whenever there is a slave lag it can log diagnostic information which we can be later analyze to check to see what exactly causing the lag. here is how you can setup pt-stalk so that it captures diagnostic data when there is slave lag: ------- pt-plug.sh contents #!/bin/bash trg_plugin() { mysqladmin $ext_argv ping &> /dev/null mysqld_alive=$? if [[ $mysqld_alive == 0 ]] then seconds_behind_master=$(mysql $ext_argv -e "show slave status" --vertical | grep seconds_behind_master | awk '{print $2}') echo $seconds_behind_master else echo 1 fi } # uncomment below to test that trg_plugin function works as expected #trg_plugin ------- -- that's the pt-plug.sh file you would need to create and then use it as below with pt-stalk: $ /usr/bin/pt-stalk --function=/root/pt-plug.sh --variable=seconds_behind_master --threshold=300 --cycles=60 [email protected] --log=/root/pt-stalk.log --pid=/root/pt-stalk.pid --daemonize you can adjust the threshold, currently its 300 seconds, combining that with –cycles, it means that if seconds_behind_master value is >= 300 for 60 seconds or more then pt-stalk will start capturing data. adding –notify-by-email option will notify via email when pt-stalk captures data. you can adjust the pt-stalk thresholds accordingly so that’s how it triggers to collect diagnostic data during problem. conclusion a lagging slave is a tricky problem but a common issue in mysql replication. i’ve tried to cover most aspects of replication delays in this post. please share in the comments section if you know of any other reasons for replication delay.

This post will go through how to setup the Hibernate Second-Level and Query caches, how they work and what are their most common pitfalls.

So when I heard about a new feature in Java 8 that was supposed to help mitigate bugs, I was excited.

Tony Hoare called the invention of the null reference the “billion dollars mistake”. May be the use of primitives in Java could be called the million dollars mistake. Primitives where created for one reason: performance. Primitives have nothing to do in an Object language. Introduction of auto boxing/unboxing was a good thing, but much more should have been done. It probably will be done (it is sometimes said to be on the Java 10 road map). In the meanwhile, we have to deal with primitives, and this is a hassle, specially when using functions. Functions in Java 5/6/7 Before Java 8, one could create functions like this: public interface Function { U apply(T t); } Function addTax = new Function() { @Override public Integer apply(Integer x) { return x / 100 * (100 + 10); } }; System.out.println(addTax.apply(100)); This code produces the following result: 110 What Java 8 gives us is the Function interface and the lambda syntax. We do not need anymore to define our own functional interface, and we may use the following syntax: Function addTax = x -> x / 100 * (100 + 10); System.out.println(addTax.apply(100)); Note that in the first example, we used an anonymous class to create a named function. In the second example, using the lambda syntax does not change anything about this. There is still an anonymous class, and a named function. One interesting question is “What is the type of x?” The type was manifest in the first example. Here, it is inferred because of the type of the function. Java knows the function argument type is an Integer because the type of the function is explicitly Function. The first Integer is the type of the argument, and the second Integer is the return type. Boxing is automatically used to convert int to Integer and back as needed. More on this later. Could we use an anonymous function? Yes, but we would have a problem with type. This does not work: System.out.println((x -> x / 100 * (100 + 10)).apply(100)); which means we can't substitute the identifier addTax with its value (the addTax function). We have to restore the type information that is now missing because Java 8 is simply not able to infer the type in this case. The most visible thing which has no explicit type here is the identifier x. So we might try: System.out.println((Integer x) -> x / 100 * 100 + 10).apply(100)); After all, int the first example, we could have written: Function addTax = (Integer x) -> x / 100 * 100 + 10; so it should be enough for Java to infer the type. But this does not work. What we have to do is specifying the type of the function. Specifying the type of its argument is not enough, even if the return type may be inferred. And there is a serious reason for this: Java 8 does not know anything about functions. Functions are ordinary object with ordinary methods that we may call. Nothing more. So we have to specify the type like this: System.out.println(((Function) x -> x / 100 * 100 + 10).apply(100)); Otherwise, it could translate to: System.out.println(((Whatever) x -> x / 100 * 100 + 10).whatever(100)); So the lambda is only syntactic sugar to simplify the Function (or Whatever) interface implementation by an anonymous class. It has in fact absolutely nothing to do with functions. Should Java had only the Function interface with its apply method, this would not be a big deal. But what about primitives? The Function interface would be fine if Java was an object language. But it is not. It is only vaguely oriented toward the use of objects (hence the name Object Oriented). The most important types in Java are the primitives. And primitives do not fit well in OOP. Auto boxing has been introduced in Java 5 to help us deal with this problem, but auto boxing as severe limitations in terms of performance, and this is related to how thing are evaluated in Java. Java is a strict language, so eager evaluation is the rule. The consequence is that each time we have a primitive and need an object, the primitive has to be boxed. And each time we have an object and need a primitive, it has to be unboxed. If we rely upon automatic boxing an unboxing, we may end with much overhead for multiple boxing and unboxing. Other languages have solved this problem differently, allowing only objects and dealing with conversion in the background. They may have “value classes”, which are objects that are backed with primitives. With this functionality, programmers only use objects and the compiler only use primitives (this is over simplified, but it gives an idea of the principle). By allowing programmers to explicitly manipulate primitives, Java makes things much more difficult and much less safe, because programmers are encouraged to use primitives as business types, which is total nonsense either in OOP or in FP. (I will come back to this in another article.) Let's say it abruptly: we should not care about the overhead of boxing and unboxing. If Java programs using this feature are too slow, the language should be fixed. We should not use bad programming techniques to work around language weaknesses. By using primitives, we make the language work against us, and not for us. If this problem is not solved through fixing the language, we should just use another language. But we probably can't for lot of bad reasons, the most important being that we are payed to program in Java and not in any other language. The result is that instead of solving business problems, we find ourselves solving Java problems. And using primitives is a Java problem, and a big one. Lets rewrite our example using primitives instead of objects. Our function takes an argument of type Integer and returns an Integer. To replace this, Java has the type IntUnaryOperator. Wow, this smells! And guess what, it is defined as: public interface IntUnaryOperator { int applyAsInt(int operand); ... } It would probably have been too simple to call the method apply. So, our example using primitives may be rewritten as: IntUnaryOperator addTax = x -> x / 100 * (100 + 10); System.out.println(addTax.applyAsInt(100)); or, using an anonymous function: System.out.println(((IntUnaryOperator) x -> x / 100 * (100 + 10)).applyAsInt(100)); If only for functions of int returning int, this would be simple. But it is much more complex. Java 8 has 43 (functional) interfaces in the java.util.function package. In reality, they do not all represent functions. They can be grouped as follows: 21 one argument functions, among which 2 are functions of object returning object and 19 are various cases of object to primitive and primitive to object functions. One of the two object to object functions is for the specific case when both argument and return value are of the same type. 9 two arguments functions, among which 2 are functions of (object, object) to object, and 7 are various cases of (object, object) to primitive or (primitive, primitive) to primitive. 7 are effects, and not functions, since they do not return any value and are supposed to be used only for their side effect. (It's somewhat strange to call these “functional interfaces”.) 5 are “suppliers”, which means functions that do not take an argument but return a value. These could be functions. In the functional world, these are special functions called nullary functions (to indicate that their arity, or number of arguments, is zero). As functions, their return value may never change, so they allow treating constants as functions. In Java 8, their role is to depend upon mutable context to return variable values. So, they are not functions. What a mess! And furthermore, the methods of these interfaces have different names. Object functions have a method named apply, where methods returning numeric primitives have method name applyAsInt, applyAsLong, or applyAsDouble. Functions returning boolean have a method called test, and suppliers have methods called get, or getAsInt, getAsLong, getAsDouble, or getAsBoolean. (They did not dare calling BooleanSupplier “Predicate” with a test method taking no argument. I really wonder why!) One thing to note is that there are no functions for byte, char, short and float. Nor are there functions for arity greater that two. Needless to say, this is totally ridiculous. But we have to stick with it. As long as Java can infer the type, we may think we have no problem. However, if you want to manipulate functions in a functional way, you will soon face the problem of Java being unable to infer a type. Worst, Java will sometime infer the type and stay silent while using a type which is no the one you intended. How to help discovering the right type Let's say we want to use a three arguments function. As there are no such functional interfaces in Java 8, you are left with a choice: create you own functional interface, or use currying, as we have seen in a previous article (What's wrong with Java 8 part I ). Creating a three object arguments functional interface returning object is straightforward: interface Function { R apply(T, t, U, u, V, v); } However, we may face two problems. The first one is that we may need to process primitives. Parametric types will not help us for this. You may create special versions of the function using primitives instead of objects. After all, with eight type of primitives, three arguments and one return value, there are only 6 561 different versions of this function. Why do you think Oracle did not put TriFunction in Java 8? (To be precise, they only put a very limited number of BiFunction where arguments are Object and return type int, long or double, or when argument and return types are of the same type int, long or Object, leading to a total of 9 out of 729 possible.) A much better solution is to use autoboxing. Just use Integer, Long, Boolean and so on and let Java handle this. Doing whatever else would be the root of all evil, i.e. premature optimization (see http://c2.com/cgi/wiki?PrematureOptimization). Another way to go (beside creating three arguments functional interface) is to use currying. This is mandatory if the arguments may not be evaluated at the same time. Furthermore, it allows using only functions of one argument, which limits the number of possible functions to 81. If we restrict ourselves to boolean, int, long and double, the number falls to 25 (four primitive types plus Object in two places equals 5 x 5). The problem is that it may be somewhat difficult to use currying with functions returning primitives or taking primitives as their argument. As an example, here is the same example used in our previous article (What's wrong with Java 8 part I ), but using primitives: IntFunction> intToIntCalculation = x -> y -> z -> x + y * z; private IntStream calculate(IntStream stream, int a) { return stream.map(intToIntCalculation.apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); Note that the result is not “a stream containing the values 5, 8, 11, 14 and 17”, no more than the initial stream would have contained the value 1, 2, 3, 4 and 5. newStream in not evaluated at this stage, so it does not contain values. (We'll talk about this in a next article). To see the result, we have to evaluate the stream, which may be forced by binding it to a terminal operation. This may be done through a call to the collect method. But before doing this, we will bind the result to one more non terminal function using the method boxed. The boxed methods binds to the stream a function converting primitives to the corresponding objects. This will simplify evaluation: System.out.println(newStream.boxed().collect(toList())); This prints: [5, 8, 11, 14, 17] We could as well use an anonymous function. However, Java is not be able to infer the type, so we must help it: private IntStream calculate(IntStream stream, int a) { return stream.map(((IntFunction>) x -> y -> z -> x + y * z).apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); Currying in itself is very easy. Just remember, as I said in a previous article, that: (x, y, z) -> w translates to x -> y -> z -> w Finding the right type is slightly more complicated. You have to remember that each time you apply an argument, you are returning a function, so you need a function from the type of the argument to an object type (because functions are objects). Here, each argument is of type int, so we need to use IntFunction parameterized with the type of the returned function. As the final type is IntUnaryOperator (as required by the map method of the IntStream class), the result is: IntFunction>> Here, we are applying two of the three parameters and all parameters are of type int, so the type is: IntFunction> This may be compared to the version using autoboxing: Function>> If you have problems determining the right type, start with the version using autoboxing, just replacing the final type you know you need (since it is the type of the argument of map): Function> Note that you may perfectly use this type in your program: private IntStream calculate(IntStream stream, int a) { return stream.map(((Function>) x -> y -> z -> x + y * z).apply(b).apply(a)); } IntStream stream = IntStream.of(1, 2, 3, 4, 5); IntStream newStream = calculate(stream, 3); You may then replace each Function>) x -> y -> z -> x + y * z).apply(b).apply(a)); } and then to: private IntStream calculate(IntStream stream, int a) { return stream.map(((IntFunction>) x -> y -> z -> x + y * z).apply(b).apply(a)); } Note that all three versions compile and run. The only difference is whether autoboxing is used or not. When to be anonymous So, as we saw in the examples above, lambdas are very good at simplifying anonymous class creation, but there is rarely good reason not to name the instance that is created. Naming functions allows: function reuse function testing function replacement program maintenance program documentation Naming function plus currying will make your function completely independent from the environment (“referential transparency”), making you programs safer and more modular. There is however a difficulty. Using primitives makes it difficult to figure the type of curried function. And worst, primitive are not the right business types to use, so the compiler will not be able to help you in this area. To see why, look at this example: double tax = 10.24; double limit = 500.0; double delivery = 35.50; DoubleStream stream3 = DoubleStream.of(234.23, 567.45, 344.12, 765.00); DoubleStream stream4 = stream3.map(x -> { double total = x / 100 * (100 + tax); if ( total > limit) { total = total + delivery; } return total; }); To replace the anonymous “capturing” function by a named curried one, determining the correct type is not so difficult. There will be four arguments and it will return a DoubleUnaryOperator, so the type will be DoubleFunction>>. However, it is very easy to misplace the arguments: DoubleFunction>> computeTotal = x -> y -> z -> w -> { double total = w / 100 * (100 + x); if (total > y) { total = total + z; } return total; }; DoubleStream stream2 = stream.map(computeTotal.apply(tax).apply(limit).apply(delivery)); How can you be sure what x, y, z and w are ? There is in fact a simple rule: the arguments that are evaluated through the explicit use of the apply method come first, in the order they are applied, i.e. tax, limit, delivery, corresponding to x, y and z. The argument coming from the stream is applied last, so it corresponds to w. However, we are still having a problem: once the function is tested, we now that it is correct, but there is no way to be sure it will be used right. For example if we apply the parameters in the wrong order: DoubleStream stream2 = stream.map(computeTotal.apply(limit).apply(tax).apply(delivery)); we get [1440.8799999999999, 3440.2000000000003, 2100.2200000000003, 4625.5] instead of: [258.215152, 661.05688, 379.357888, 878.836] This means we have to test not only the function, but each use of it. Wouldn't it be nice if we could be sure that using the parameters in the wrong order would not compile? This is what using the right type system is about. Using primitives for business types is not good. It has never be. But now, with functions, we have one more reason not to do this. This will be the subject of another article. What's next We have seen how using primitives is somewhat more complicated that using objects. Functions using primitives are a real mess in Java 8. But the worst is to come. In a next article, we will talk about using primitives with streams.

Earlier I had blogged about using Scala with Spring Boot and how the combination just works. There was one issue with the previous approach though - the only way to run the earlier configuration was to build the project into a jar file and run the jar file. ./gradlew build java -jar build/libs/spring-boot-scala-web-0.1.0.jar Spring boot comes with a gradle based plugin which should have allowed the project to run with a "gradle bootRun" command, this unfortunately gives an error for scala based projects. A good workaround is to use sbt for building and running Spring-boot based projects. The catch though is that with gradle and maven, the versions of the dependencies would have been managed through a parent pom, now these have to be explicitly specified. This is how a sample sbt build file with the dependencies spelled out looks: name := "spring-boot-scala-web" version := "1.0" scalaVersion := "2.10.4" sbtVersion := "0.13.1" seq(webSettings : _*) libraryDependencies ++= Seq( "org.springframework.boot" % "spring-boot-starter-web" % "1.0.2.RELEASE", "org.springframework.boot" % "spring-boot-starter-data-jpa" % "1.0.2.RELEASE", "org.webjars" % "bootstrap" % "3.1.1", "org.webjars" % "jquery" % "2.1.0-2", "org.thymeleaf" % "thymeleaf-spring4" % "2.1.2.RELEASE", "org.hibernate" % "hibernate-validator" % "5.0.2.Final", "nz.net.ultraq.thymeleaf" % "thymeleaf-layout-dialect" % "1.2.1", "org.hsqldb" % "hsqldb" % "2.3.1", "org.springframework.boot" % "spring-boot-starter-tomcat" % "1.0.2.RELEASE" % "provided", "javax.servlet" % "javax.servlet-api" % "3.0.1" % "provided" ) libraryDependencies ++= Seq( "org.apache.tomcat.embed" % "tomcat-embed-core" % "7.0.53" % "container", "org.apache.tomcat.embed" % "tomcat-embed-logging-juli" % "7.0.53" % "container", "org.apache.tomcat.embed" % "tomcat-embed-jasper" % "7.0.53" % "container" ) Here I am also using xsbt-web-plugin which is plugin for building scala web applications. xsbt-web-plugin also comes with commands to start-up tomcat or jetty based containers and run the applications within these containers, however I had difficulty in getting these to work. What worked is the runMain command to start up the Spring-boot main program through sbt: runMain mvctest.SampleWebApplication and xsbt-web-plugin allows the project to be packaged as a war file using the "package" command, this war deploys and runs without any issues in a standalone tomcat container. Here is a github project with these changes: https://github.com/bijukunjummen/spring-boot-scala-web.git

Learn all about Jersey/Jax RS streaming with JSON.

The Open Session in View (OSIV) pattern gives rise to different opinions in the Java development community. Let's go over OSIV and some of the pros and cons of this pattern. The problem The problem that OSIV solves is a mismatch between the Hibernate concept of session and it's lifecycle and the way that many server-side view technologies work. In a typical Java frontend application the service layer starts by querying some of the data needed to build the view. The remaining data needed can be lazy-loaded later, with the condition that the Hibernate session remains open - and there lies the problem. Between the moment that the service layer method finishes it's execution and the moment that the view is rendered, Hibernate has already committed the transaction and closed the session. When the view tries to lazy load the extra data that it needs, if finds the Hibernate session closed, causing a LazyInitializationException. The OSIV solution OSIV tackles this problem by ensuring that the Hibernate session is kept open all the way up to the rendering of the view - hence the name of the pattern. Because the session is kept open, no more LazyInitializationExceptions occur. The session or entity manager is kept open by means of a filter that is added to the request processing chain. In the case of JPA the OpenEntityManagerInViewFilter will create an entity manager at the beginning of the request, and then bind it to the request thread. The service layer will then be executed and the business transaction committed or rolled back, but the transaction manager will not remove the entity manager from the thread after the commit. When the view rendering starts, the transaction manager will then check if there is already an entity manager binded to the thread, and if so use it instead of creating a new one. After the request is processed, the filter will then unbind the entity manager from the thread. The end result is that the same entity manager used to commit the business transaction was kept around in the request thread, allowing the view rendering code to lazy load the needed data. Going back to the original problem Let's step back a moment and go back to the initial problem: the LazyInitializationException. Is this exception really a problem? This exception can also be seen as a warning sign of a wrongly written query in the service layer. When building a view and it's backing services, the developer knows upfront what data is needed, and can make sure that the needed data is loaded before the rendering starts. Several relation types such as one-to-many use lazy-loading by default, but that default setting can be overridden if needed at query time using the following syntax: select p FROM Person p left join fetch p.invoices This means that the lazy loading can be turned off on a case by case basis depending on the data needed by the view. OSIV in projects I've worked In projects I have worked that used OSIV, we could see via query logging that the database was getting hit with a high number of SQL queries, sometimes to the point that developers had to turn off the Hibernate SQL logging. The performance of these application was impacted, but it was kept manageable using second-level caches, and due to the fact that these where intranet-based applications with a limited number of users. Pros of OSIV The main advantage of OSIV is that it makes working with ORM and the database more transparent: Less queries need to be manually written Less awareness is required about the Hibernate session and how to solve LazyInitializationExceptions. Cons of OSIV OSIV seems to be easy to misuse and can accidentally introduce N+1 performance problems in the application. On projects I've worked OSIV did not work out well in the long-term. The alternative of writing custom queries that eager fetch data depending on the use case is manageable and turned out well in other projects I've worked. Alternatives to OSIV Besides the application-level solution of writing custom queries to pre-fetch the needed data, there are other framework-level aproaches to OSIV. The Seam Framework was built by some of the same developers as Hibernate , and solves the problem by introducing the notion of conversation. Can you let me know in the comments bellow your thoughts and experiences with OSIV, thanks for reading.

System Requirements: Eclipse Kepler Service Release 2 JDK 1.5 or above Installed Google App Engine SDK on eclipse – this is required for second version of this example. Create a google spreadsheet – login to your google account and create a new spreadsheet, if you want to read existing one then put that url in SPREADSHEET_URL. Once you create a new spreadsheet our url will be like – https://docs.google.com/spreadsheets/d/1L8xtAJfOObsXL-XemliUV10wkDHQNxjn6jKS4XwzYZ8/ but don’t put this url in SPREADSHEET_URL, Use the below url simply change the bold part. public static final String SPREADSHEET_URL = “https://spreadsheets.google.com/feeds/spreadsheets/1L8xtAJfOObsXL-XemliUV10wkDHQNxjn6jKS4XwzYZ8“; // Fill in google spreadsheet URI package org.gopaldas.readsps; import java.io.IOException; import java.net.URL; import com.google.gdata.client.spreadsheet.SpreadsheetService; import com.google.gdata.data.spreadsheet.ListEntry; import com.google.gdata.data.spreadsheet.ListFeed; import com.google.gdata.data.spreadsheet.SpreadsheetEntry; import com.google.gdata.data.spreadsheet.WorksheetEntry; import com.google.gdata.util.ServiceException; public class ReadSpreadsheet { public static final String GOOGLE_ACCOUNT_USERNAME = "[email protected]"; // Fill in google account username public static final String GOOGLE_ACCOUNT_PASSWORD = "xxxx"; // Fill in google account password public static final String SPREADSHEET_URL = "https://spreadsheets.google.com/feeds/spreadsheets/1L8xtAJfOObsXL-XemliUV10wkDHQNxjn6jKS4XwzYZ8"; //Fill in google spreadsheet URI public static void main(String[] args) throws IOException, ServiceException { /** Our view of Google Spreadsheets as an authenticated Google user. */ SpreadsheetService service = new SpreadsheetService("Print Google Spreadsheet Demo"); // Login and prompt the user to pick a sheet to use. service.setUserCredentials(GOOGLE_ACCOUNT_USERNAME, GOOGLE_ACCOUNT_PASSWORD); // Load sheet URL metafeedUrl = new URL(SPREADSHEET_URL); SpreadsheetEntry spreadsheet = service.getEntry(metafeedUrl, SpreadsheetEntry.class); URL listFeedUrl = ((WorksheetEntry) spreadsheet.getWorksheets().get(0)).getListFeedUrl(); // Print entries ListFeed feed = (ListFeed) service.getFeed(listFeedUrl, ListFeed.class); for(ListEntry entry : feed.getEntries()) { System.out.println("new row"); for(String tag : entry.getCustomElements().getTags()) { System.out.println(" "+tag + ": " + entry.getCustomElements().getValue(tag)); } } } }

There are multiple ways to get hold of and use an Http session with a Spring based web application. This is a summarization based on an experience with a recent project. Approach 1 Just inject in HttpSession where it is required. @Service public class ShoppingCartService { @Autowired private HttpSession httpSession; ... } Though surprising, since the service above is a singleton, this works well. Spring intelligently injects in a proxy to the actual HttpSession and this proxy knows how to internally delegate to the right session for the request. The catch with handling session this way though is that the object being retrieved and saved back in the session will have to be managed by the user: public void removeFromCart(long productId) { ShoppingCart shoppingCart = getShoppingCartInSession(); shoppingCart.removeItemFromCart(productId); updateCartInSession(shoppingCart); } Approach 2 Accept it as a parameter, this will work only in the web tier though: @Controller public class ShoppingCartController { @RequestMapping("/addToCart") public String addToCart(long productId, HttpSession httpSession) { //do something with the httpSession } } Approach 3 Create a bean and scope it to the session this way: @Component @Scope(proxyMode=ScopedProxyMode.TARGET_CLASS, value="session") public class ShoppingCart implements Serializable{ ... } Spring creates a proxy for a session scoped bean and makes the proxy available to services which inject in this bean. An advantage of using this approach is that any state changes on this bean are handled by Spring, it would take care of retrieving this bean from the session and propagating any changes to the bean back to the session. Further if the bean were to have any Spring lifecycle methods(say @PostConstruct or @PreDestroy annotated methods), they would get called appropriately. Approach 4 Annotating Spring MVC model attributes with @SessionAttribute annotation: @SessionAttributes("shoppingCart") public class OrderFlowController { public String step1(@ModelAttribute("shoppingCart") ShoppingCart shoppingCart) { } public String step2(@ModelAttribute("shoppingCart") ShoppingCart shoppingCart) { } public String step3(@ModelAttribute("shoppingCart") ShoppingCart shoppingCart, SessionStatus status) { status.setComplete(); } } The use case for using SessionAttributes annotation is very specific, to hold state during a flow like above Given these approaches, I personally prefer Approach 3 of using session scoped beans, this way depending on Spring to manage the underlying details of retrieving and storing the object into session. Other approaches have value though based on the scenario that you may be faced with, ranging from requiring more control over raw Http Sessions to needing to handle temporary state like in Approach 4 above.

wanted to go through some of the basic tenets, the technical terminology related to java ee. for many people, java ee/j2ee still mean servlets, jsps or maybe struts at best. no offence or pun intended! this is not a java ee 'bible' by any means. i am not capable enough of writing such a thing! so let us line up the 'keywords' related to java ee and then look at them one by one java ee java ee apis (specifications) containers services multitiered applications components let's try to elaborate on the above mentioned points. ok. so what is java ee? 'ee' stands for enterprise edition. that essentially makes java ee - java enterprise edition. if i had to summarize java ee in a couple of sentences, it would go something like this "java ee is a platform which defines 'standard specifications/apis' which are then implemented by vendors and used for development of enterprise (distributed, 'multi-tired', robust) 'applications'. these applications are composed of modules or 'components' which use java ee 'containers' as their run-time infrastructure." what is this 'standardized platform' based upon? what does it constitute? the platform revolves around 'standard' specifications or apis . think of these as contracts defined by a standard body e.g. enterprise java beans (ejb), java persistence api (jpa), java message service (jms) etc. these contracts/specifications/apis are implemented by different vendors e.g. glassfish, oracle weblogic, apache tomee etc alright. what about containers? containers can be visualized as 'virtual/logical partitions' . each container supports a subset of the apis/specifications defined by the java ee platform they provide run-time 'services' to the 'applications' which they host the java ee specification lists 4 types of containers ejb container web container application client container applet container java ee containers i am not going to dwell into details of these containers in this post. services?? well, 'services' are nothing but a result of the vendor implementations of the standard 'specifications' (mentioned above). examples of specifications are - jersey for jax-rs (restful services), tyrus (web sockets), eclipselink (jpa), weld (cdi) etc. the 'container' is the interface between the deployed application ('service' consumer) and the application server. here is a list of 'services' which are rendered by the 'container' to the underlying 'components' (this is not an exhaustive list) persistence - offered by the java persistence api (jpa) which drives object relational mapping (orm) and an abstraction for the database operations. messaging - the java message service (jms) provides asynchronous messaging between disparate parts of your applications. contexts & dependency injection - cdi provides loosely coupled and type safe injection of resources. web services - jaxrs and jaxws provide support for rest and soap style services respectively transaction - provided by the java transaction api (jta) implementation what is a typical java ee 'application'? what does it comprise of? applications are composed of different ' components ' which in turn are supported by their corresponding ' container ' supported 'component' types are: enterprise applications - make use of the specifications like ejb, jms, jpa etc and are executed within an ejb container web applications - they leverage the servlet api, jsp, jsf etc and are supported by a web container application client - executed in client side. they need an application client container which has a set of supported libraries and executes in a java se environment. applets - these are gui applications which execute in a web browser. how are java ee applications structured? as far as java ee 'application' architecture is concerned, they generally tend follow the n-tier model consisting of client tier, server tier and of course the database (back end) tier client tier - consists of web browsers or gui (swing, java fx) based clients. web browsers tend to talk to the 'web components' on the server tier while the gui clients interact directly with the 'business' layer within the server tier server tier - this tier comprises of the dynamic web components (jsp, jsf, servlets) and the business layer driven by ejbs, jms, jpa, jta specifications. database tier - contains 'enterprise information systems' backed by databases or even legacy data repositories. generic 3-tier java ee application architecture java ee - bare bones, basics.... as quickly and briefly as i possibly could. that's all for now! :-) stay tuned for more java ee content, specifically around the latest and greatest version of the java ee platform --> java ee 7 happy reading!

splunk vs. sumo logic vs. logstash vs. graylog vs. loggly vs. papertrails vs. splunk>storm splunk, sumo logic, logstash, graylog, loggly, papertrails - did i miss someone? i’m pretty sure i did. logs are like fossil fuels - we’ve been wanting to get rid of them for the past 20 years, but we’re not quite there yet. well, if that's the case i want a bmw! to deal with the growth of log data a host of log management & analysis tools have been built over the last few years to help developers and operations make sense of the growing data. i thought it’d be interesting to look at our options and what are each tools’ selling point, from a developer’s standpoint . splunk as the biggest tool in this space, i decided to put splunk in a category of its own. that’s not to say it’s the best tool for what you need, but more to give credit to a product who essentially created a new category. pros splunk is probably the most feature rich solution in the space. it’s got hundreds of apps (i counted 537 ) to make sense of almost every format of log data, from security to business analytics to infrastructure monitoring. splunk’s search and charting tools are feature rich to the point that there’s probably no set of data you can’t get to through its ui or apis. cons splunk has two major cons. the first, that is more subjective, is that it’s an on-premise solution which means that setup costs in terms of money and complexity are high. to deploy in a high-scale environment you will need to install and configure a dedicated cluster. as a developer, it’s usually something you can't or don’t want to do as your first choice. splunk’s second con is that it’s expensive. to support a real-world application you’re looking at tens of thousands of dollars, which most likely means you’ll need sign offs from high-ups in your organization, and the process is going to be slow. if you’ve got a new app and you want something fast that you can quickly spin up and ramp as things progress - keep reading. some more enterprise log analyzers can be found here . saas log analyzers sumo logic sumo was founded as a saas version of splunk, going so far as to imitate some of splunk’s features and visuals early on. having said that, sl has developed to a full fledged enterprise class log management solution. pros sl is chock-full of features to reduce, search and chart mass amounts of data. out of all the saas log analyzers, it’s probably the most feature rich. also, being a saas offering it inherently means setup and ongoing operation are easier. one of sumo logic’s main points of attraction is the ability to establish baselines and to actively notify you when key metrics change after an event such as a new version rollout or a breach attempt. cons this one is shared across all saas log analyzers, which is you need to get the data to the service to actually do something with it. this means that you’ll be looking at possible gbs (or more) uploaded from your servers. this can create issues on multiple fronts - as a developer, if you're logging sensitive or pii you need to make sure it’s redacted. there may be a lag between the time data is logged and the time it’s visible to to the service. there’s additional overhead on your machines transmitting gbs of data, which really depends on your logging throughput. sumo’s pricing is also not transparent , which means you might be looking at a buying process which is more complex than swiping your team’s credit card to get going. loggly loggly is also a robust log analyzer, focusing on simplicity and ease of use for a devops audience. pros whereas sumo logic has a strong enterprise and security focus, loggly is geared more towards helping devops find and fix operational problems. this makes it very developer-friendly. things like creating custom performance and devops dashboards are super-easy to do. pricing is also transparent, which makes start of use easier. cons don't expect loggly to scale into a full blown infrastructure, security or analytics solution. if you need forensics or infrastructure monitoring you’re in the wrong place. this is a tools mainly for devops to parse data coming from your app servers. anything beyond that you’ll have to build yourself. papertrails papertrails is a simple way to look and search through logs from multiple machines, in one consolidated easy-to-use interface. think of it like tailing your log in the cloud, and you won't be too far off. pros pt is what it is. a simple way to look at log files from multiple machines in a singular view in the cloud. the ux itself is very similar to looking at a log on your machine, and so are the search commands. it aims to do something simple and useful, and does it elegantly. it’s also very affordable . cons pt is mostly text based. looking for any advanced integrations, predictive or reporting capabilities? you're barking up the wrong tree. splunk>storm this is splunk’s little (some may say step) saas brother. it’s a pretty similar offering that’s hosted on splunk’s servers. pros storm lets you experiment with splunk without having to install the actual software on-premise, and contains much of the features available in the full version. cons this isn't really a commercial offering, and you're limited in the amount of data you can send. it seems to be more of an online limited version of splunk meant to help people test out the product without having to deploy first. a new service called splunk cloud is aimed at providing a full-blown splunk saas experience. open source analyzers logstash logstash is an open source tool for collecting and managing log files. it’s part of an open-source stack which includes elasticsearch for indexing and searching through data and kibana for charting and visualizing data. together they form a powerful log management solution. pros being an open-source solution means you're inherently getting a lot of a control and a very good price. logstash uses three mature and powerful components, all heavily maintained, to create a very robust and extensible package. for an open-source solution it’s also very easy to install and start using. we use logstash and love it. cons as logstash is essentially a stack, it means you're dealing with three different products. that means that extensibility also becomes complex. logstash filters are written in ruby, kibana is pure javascript and elasticsearch has its own rest api as well as json templates. when you move to production, you’ll also need to separate the three into different machines, which adds to the complexity. graylog2 a fairly new player in the space, gl2 is an open-source log analyzer backed by mongodb as well as elasticsearch (similar to logstash) for storing and searching through log errors. it’s mainly focused on helping developers detect and fix errors in their apps. also in this category you can find fluentd and kafka whose one of its main use-cases is also storing log data. phew, so many choices! takipi for logs while this post is not about takipi, i thought there’s one feature it has which you might find relevant to all of this. the biggest disadvantage in all log analyzers and log files in general, is that the right data has to be put there by you first. from a dev perspective, it means that if an exception isn’t logged, or the variable data you need to understand why it happened isn't there, no log file or analyzer in the world can help you. production debugging sucks. one of the things we’ve added to takipi is the ability to jump into a recorded debugging session straight from a log file error. this means that for every log error you can see the actual source code and variable values at the moment of error. you can learn more about it here . this is one post where i would love to hear from you guys about your experiences with some of the tools mentioned (and some that i didn’t). i’m sure there are things you would disagree with or would like to correct me on - so go ahead, the comment section is below and i would love to hear from you. originally posted on takipi blog

This is one annoying problem that happens sometimes to git users: the symptom is: git status command shows you some files as modified (you are sure that you had not modified that files), you revert all changes with a git checkout — . but the files stills are in modified state if you issue another git status. This is a real annoying problem, suppose you want to switch branch with git checkout branchname, you will find that git does not allow you to switch because of uncommitted changes. This problem is likely caused by the end-of-line normalization (I strongly suggest you to read all the details in Pro Git book or read the help of github). I do not want to enter into details of this feature, but I only want to help people to diagnose and avoid this kind of problem. To understand if you really have a Line Ending Issue you should run git diff -w command to verify what is really changed in files that git as modified with git status command. The -w options tells git to ignore whitespace and line endings, if this command shows no differences, you are probably victim of problem in Line Ending Normalization. This is especially true if you are working with git svn, connecting to a subversion repository where developers did not pay attention to line endings and it happens usually when you have files with mixed CRLF / CR / LF. If you work in mixed environment (Unix/Linux, Windows, Macintosh) it is better to find files that are listed as modified and manually (or with some tool) normalize Line Endings. If you do not work in mixed environment you can simply turn off eol normalizationfor the single repository where you experience the problem. To do this you can issue a git config –local core.autocrlf false but it works only for you and not for all the other developers that works to the project. Moreover some people reports that they still have problem even with core.autocrlf to false. Remember that git supports .gitattributes files, used to change settings for a single subdirectory. If you set core.autocrlf to false and still have line ending normalization problem, please search for .gitattribuges files in every subdirectory of your repository, and verify if it has a line where autocrlf is turned on: * text=auto now you can turn off in all .gitattributes files you find in your repository * text=off To be sure that every developer of the team works with autocrlf turned off, you should place a .gitattributes file in repository root with autocrlf turned off. Remember that it is a better option to normalize files and leave autocrlf turned on, but if you are working with legacy code imported from another VCS, or you work with git svn, git-tf or similar tools, probably it is better turn autocrlf to off if you start experiencing that kind of problems.

Get a rundown on the Tomcat NIO Connector as well as a tutorial on how to set it up.

Michael recently wrote a blog post showing how to create a time tree representing time down to the second using Neo4j’s Cypher query language, something I built on top of for a side project I’m working on. The domain I want to model is RSVPs to meetup invites – I want to understand how much in advance people respond and how likely they are to drop out at a later stage. For this problem I only need to measure time down to the day so my task is a bit easier than Michael’s. After a bit of fiddling around with leap years, I believe the following query will create a time tree representing all the days from 2011 – 2014, which covers the time the London Neo4j meetup has been running: WITH range(2011, 2014) AS years, range(1,12) as months FOREACH(year IN years | MERGE (y:Year {year: year}) FOREACH(month IN months | CREATE (m:Month {month: month}) MERGE (y)-[:HAS_MONTH]->(m) FOREACH(day IN (CASE WHEN month IN [1,3,5,7,8,10,12] THEN range(1,31) WHEN month = 2 THEN CASE WHEN year % 4 <> 0 THEN range(1,28) WHEN year % 100 <> 0 THEN range(1,29) WHEN year % 400 <> 0 THEN range(1,29) ELSE range(1,28) END ELSE range(1,30) END) | CREATE (d:Day {day: day}) MERGE (m)-[:HAS_DAY]->(d)))) The next step is to link adjacent days together so that we can easily traverse between adjacent days without needing to go back up and down the tree. For example we should have something like this: (jan31)-[:NEXT]->(feb1)-[:NEXT]->(feb2) We can build this by first collecting all the ‘day’ nodes in date order like so: MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day) WITH year,month,day ORDER BY year.year, month.month, day.day WITH collect(day) as days RETURN days And then iterating over adjacent nodes to create the ‘NEXT’ relationship: MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day) WITH year,month,day ORDER BY year.year, month.month, day.day WITH collect(day) as days FOREACH(i in RANGE(0, length(days)-2) | FOREACH(day1 in [days[i]] | FOREACH(day2 in [days[i+1]] | CREATE UNIQUE (day1)-[:NEXT]->(day2)))) Now if we want to find the previous 5 days from the 1st February 2014 we could write the following query: MATCH (y:Year {year: 2014})-[:HAS_MONTH]->(m:Month {month: 2})-[:HAS_DAY]->(:Day {day: 1})<-[:NEXT*0..5]-(day) RETURN y,m,day If we want to we can create the time tree and then connect the day nodes all in one query by using ‘WITH *’ like so: WITH range(2011, 2014) AS years, range(1,12) as months FOREACH(year IN years | MERGE (y:Year {year: year}) FOREACH(month IN months | CREATE (m:Month {month: month}) MERGE (y)-[:HAS_MONTH]->(m) FOREACH(day IN (CASE WHEN month IN [1,3,5,7,8,10,12] THEN range(1,31) WHEN month = 2 THEN CASE WHEN year % 4 <> 0 THEN range(1,28) WHEN year % 100 <> 0 THEN range(1,29) WHEN year % 400 <> 0 THEN range(1,29) ELSE range(1,28) END ELSE range(1,30) END) | CREATE (d:Day {day: day}) MERGE (m)-[:HAS_DAY]->(d)))) WITH * MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day) WITH year,month,day ORDER BY year.year, month.month, day.day WITH collect(day) as days FOREACH(i in RANGE(0, length(days)-2) | FOREACH(day1 in [days[i]] | FOREACH(day2 in [days[i+1]] | CREATE UNIQUE (day1)-[:NEXT]->(day2)))) Now I need to connect the RSVP events to the tree!

I was playing around with Neo4j’s Cypher last weekend and found myself accidentally running some queries against an earlier version of the Neo4j 2.0 series (2.0.0). My first query started with a map and I wanted to create a person from an identifier inside the map: WITH {person: {id: 1} AS params MERGE (p:Person {id: params.person.id}) RETURN p When I ran the query I got this error: ==> SyntaxException: Type mismatch: expected Map but was Boolean, Number, String or Collection (line 1, column 62) ==> "WITH {person: {id: 1} AS params MERGE (p:Person {id: params.person.id}) RETURN p" If we try the same query in 2.0.1 it works as we’d expect: ==> +---------------+ ==> | p | ==> +---------------+ ==> | Node[1]{id:} | ==> +---------------+ ==> 1 row ==> Nodes created: 1 ==> Properties set: 1 ==> Labels added: 1 ==> 47 ms My next query was the following which links topics of interest to a person: WITH {topics: [{name: "Java"}, {name: "Neo4j"}]} AS params MERGE (p:Person {id: 2}) FOREACH(t IN params.topics | MERGE (topic:Topic {name: t.name}) MERGE (p)-[:INTERESTED_IN]->(topic) ) RETURN p In 2.0.0 that query fails like so: ==> InternalException: Query not prepared correctly! but if we try it in 2.0.1 we’ll see that it works as well: ==> +---------------+ ==> | p | ==> +---------------+ ==> | Node[4]{id:2} | ==> +---------------+ ==> 1 row ==> Nodes created: 1 ==> Relationships created: 2 ==> Properties set: 1 ==> Labels added: 1 ==> 53 ms So if you’re seeing either of those errors, then get yourself upgraded to 2.0.1 as well!

Running Groovy scripts with GroovyShell is easy. We can for example incorporate a Domain Specific Language (DSL) in our application where the DSL is expressed in Groovy code and executed by GroovyShell. To limit the constructs that can be used in the DSL (which is Groovy code) we can apply a SecureASTCustomizer to the GroovyShell configuration. With the SecureASTCustomizer the Abstract Syntax Tree (AST) is inspected, we cannot define runtime checks here. We can for example disallow the definition of closures and methods in the DSL script. Or we can limit the tokens to be used to just a plus or minus token. To have even more control we can implement the StatementChecker andExpressionChecker interface to determine if a specific statement or expression is allowed or not. In the following sample we first use the properties of the SecureASTCustomizer class to define what is possible and not within the script: package com.mrhaki.blog import org.codehaus.groovy.control.customizers.SecureASTCustomizer import org.codehaus.groovy.control.CompilerConfiguration import org.codehaus.groovy.ast.stmt.* import org.codehaus.groovy.ast.expr.* import org.codehaus.groovy.control.MultipleCompilationErrorsException import static org.codehaus.groovy.syntax.Types.PLUS import static org.codehaus.groovy.syntax.Types.MINUS import static org.codehaus.groovy.syntax.Types.EQUAL // Define SecureASTCustomizer to limit allowed // language syntax in scripts. final SecureASTCustomizer astCustomizer = new SecureASTCustomizer( // Do not allow method creation. methodDefinitionAllowed: false, // Do not allow closure creation. closuresAllowed: false, // No package allowed. packageAllowed: false, // White or blacklists for imports. importsBlacklist: ['java.util.Date'], // or importsWhitelist staticImportsWhitelist: [], // or staticImportBlacklist staticStarImportsWhitelist: [], // or staticStarImportsBlacklist // Make sure indirect imports are restricted. indirectImportCheckEnabled: true, // Only allow plus and minus tokens. tokensWhitelist: [PLUS, MINUS, EQUAL], // or tokensBlacklist // Disallow constant types. constantTypesClassesWhiteList: [Integer, Object, String], // or constantTypesWhiteList // or constantTypesBlackList // or constantTypesClassesBlackList // Restrict method calls to whitelisted classes. // receiversClassesWhiteList: [], // or receiversWhiteList // or receiversClassesBlackList // or receiversBlackList // Ignore certain language statement by // whitelisting or blacklisting them. statementsBlacklist: [IfStatement], // or statementsWhitelist // Ignore certain language expressions by // whitelisting or blacklisting them. expressionsBlacklist: [MethodCallExpression] // or expresionsWhitelist ) // Add SecureASTCustomizer to configuration for shell. final conf = new CompilerConfiguration() conf.addCompilationCustomizers(astCustomizer) // Create shell with given configuration. final shell = new GroovyShell(conf) // All valid script. final result = shell.evaluate ''' def s1 = 'Groovy' def s2 = 'rocks' "$s1 $s2!" ''' assert result == 'Groovy rocks!' // Some invalid scripts. try { // Importing [java.util.Date] is not allowed shell.evaluate ''' new Date() ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('Indirect import checks prevents usage of expression') } try { // MethodCallExpression not allowed shell.evaluate ''' println "Groovy rocks!" ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('MethodCallExpressions are not allowed: this.println(Groovy rocks!)') } To have more fine-grained control on which statements and expression are allowed we can implement the StatementChecker andExpressionChecker interfaces. These interfaces have one method isAuthorized with a boolean return type. We return true if a statement or expression is allowed and false if not. package com.mrhaki.blog import org.codehaus.groovy.control.customizers.SecureASTCustomizer import org.codehaus.groovy.control.CompilerConfiguration import org.codehaus.groovy.ast.stmt.* import org.codehaus.groovy.ast.expr.* import org.codehaus.groovy.control.MultipleCompilationErrorsException import static org.codehaus.groovy.control.customizers.SecureASTCustomizer.ExpressionChecker import static org.codehaus.groovy.control.customizers.SecureASTCustomizer.StatementChecker // Define SecureASTCustomizer. final SecureASTCustomizer astCustomizer = new SecureASTCustomizer() // Define expression checker to deny // usage of variable names with length of 1. def smallVariableNames = { expr -> if (expr instanceof VariableExpression) { expr.variable.size() > 1 } else { true } } as ExpressionChecker astCustomizer.addExpressionCheckers smallVariableNames // In for loops the collection name // can only be 'names'. def forCollectionNames = { statement -> if (statement instanceof ForStatement) { statement.collectionExpression.variable == 'names' } else { true } } as StatementChecker astCustomizer.addStatementCheckers forCollectionNames // Add SecureASTCustomizer to configuration for shell. final CompilerConfiguration conf = new CompilerConfiguration() conf.addCompilationCustomizers(astCustomizer) // Create shell with given configuration. final GroovyShell shell = new GroovyShell(conf) // All valid script. final result = shell.evaluate ''' def names = ['Groovy', 'Grails'] for (name in names) { print "$name rocks! " } def s1 = 'Groovy' def s2 = 'rocks' "$s1 $s2!" ''' assert result == 'Groovy rocks!' // Some invalid scripts. try { // Variable s has length 1, which is not allowed. shell.evaluate ''' def s = 'Groovy rocks' s ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('Expression [VariableExpression] is not allowed: s') } try { // Only names as collection expression is allowed. shell.evaluate ''' def languages = ['Groovy', 'Grails'] for (name in languages) { println "$name rocks!" } ''' } catch (MultipleCompilationErrorsException e) { assert e.message.contains('Statement [ForStatement] is not allowed') } Code written with Groovy 2.2.2.