After attending Sam Newman’s microservice talks at Geecon last week I started to think more about what is most likely an essential feature of service-oriented/microservice platforms for monitoring, reporting and diagnostics: correlation ids. Correlation ids allow distributed tracing within complex service oriented platforms, where a single request into the application can often be dealt with by multiple downstream service. Without the ability to correlate downstream service requests it can be very difficult to understand how requests are being handled within your platform. I’ve seen the benefit of correlation ids in several recent SOA projects I have worked on, but as Sam mentioned in his talks, it’s often very easy to think this type of tracing won’t be needed when building the initial version of the application, but then very difficult to retrofit into the application when you do realise the benefits (and the need for!). I’ve not yet found the perfect way to implement correlation ids within a Java/Spring-based application, but after chatting to Sam via email he made several suggestions which I have now turned into a simple project using Spring Boot to demonstrate how this could be implemented. Why? During both of Sam’s Geecon talks he mentioned that in his experience correlation ids were very useful for diagnostic purposes. Correlation ids are essentially an id that is generated and associated with a single (typically user-driven) request into the application that is passed down through the stack and onto dependent services. In SOA or microservice platforms this type of id is very useful, as requests into the application typically are ‘fanned out’ or handled by multiple downstream services, and a correlation id allows all of the downstream requests (from the initial point of request) to be correlated or grouped based on the id. So called ‘distributed tracing’ can then be performed using the correlation ids by combining all the downstream service logs and matching the required id to see the trace of the request throughout your entire application stack (which is very easy if you are using a centralised logging framework such as logstash) The big players in the service-oriented field have been talking about the need for distributed tracing and correlating requests for quite some time, and as such Twitter have created their open source Zipkin framework (which often plugs into their RPC framework Finagle), and Netflix has open-sourced their Karyon web/microservice framework, both of which provide distributed tracing. There are of course commercial offering in this area, one such product being AppDynamics, which is very cool, but has a rather hefty price tag. Creating a proof-of-concept in Spring Boot As great as Zipkin and Karyon are, they are both relatively invasive, in that you have to build your services on top of the (often opinionated) frameworks. This might be fine for some use cases, but no so much for others, especially when you are building microservices. I’ve been enjoying experimenting with Spring Boot of late, and this framework builds on the much known and loved (at least by me :-) ) Spring framework by providing lots of preconfigured sensible defaults. This allows you to build microservices (especially ones that communicate via RESTful interfaces) very rapidly. The remainder of this blog pos explains how I implemented a (hopefully) non-invasive way of implementing correlation ids. Goals Allow a correlation id to be generated for a initial request into the application Enable the correlation id to be passed to downstream services, using as method that is as non-invasive into the code as possible Implementation I have created two projects on GitHub, one containing an implementation where all requests are being handled in a synchronous style (i.e. the traditional Spring approach of handling all request processing on a single thread), and also one for when an asynchronous (non-blocking) style of communication is being used (i.e., using the Servlet 3 asynchronous support combined with Spring’s DeferredResult and Java’s Futures/Callables). The majority of this article describes the asynchronous implementation, as this is more interesting: Spring Boot asynchronous (DeferredResult + Futures) communication correlation id Github repo The main work in both code bases is undertaken by the CorrelationHeaderFilter, which is a standard Java EE Filter that inspects the HttpServletRequest header for the presence of a correlationId. If one is found then we set a ThreadLocal variable in the RequestCorrelation Class (discussed later). If a correlation id is not found then one is generated and added to the RequestCorrelation Class: public class CorrelationHeaderFilter implements Filter { //... @Override public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException { final HttpServletRequest httpServletRequest = (HttpServletRequest) servletRequest; String currentCorrId = httpServletRequest.getHeader(RequestCorrelation.CORRELATION_ID_HEADER); if (!currentRequestIsAsyncDispatcher(httpServletRequest)) { if (currentCorrId == null) { currentCorrId = UUID.randomUUID().toString(); LOGGER.info("No correlationId found in Header. Generated : " + currentCorrId); } else { LOGGER.info("Found correlationId in Header : " + currentCorrId); } RequestCorrelation.setId(currentCorrId); } filterChain.doFilter(httpServletRequest, servletResponse); } //... private boolean currentRequestIsAsyncDispatcher(HttpServletRequest httpServletRequest) { return httpServletRequest.getDispatcherType().equals(DispatcherType.ASYNC); } The only thing is this code that may not instantly be obvious is the conditional checkcurrentRequestIsAsyncDispatcher(httpServletRequest), but this is here to guard against the correlation id code being executed when the Async Dispatcher thread is running to return the results (this is interesting to note, as I initially didn’t expect the Async Dispatcher to trigger the execution of the filter again?) Here is the RequestCorrelation Class, which contains a simple ThreadLocal static variable to hold the correlation id for the current Thread of execution (set via the CorrelationHeaderFilter above) public class RequestCorrelation { public static final String CORRELATION_ID = "correlationId"; private static final ThreadLocal id = new ThreadLocal(); public static String getId() { return id.get(); } public static void setId(String correlationId) { id.set(correlationId); } } Once the correlation id is stored in the RequestCorrelation Class it can be retrieved and added to downstream service requests (or data store access etc) as required by calling the static getId() method within RequestCorrelation. It is probably a good idea to encapsulate this behaviour away from your application services, and you can see an example of how to do this in a RestClient Class I have created, which composes Spring’s RestTemplate and handles the setting of the correlation id within the header transparently from the calling Class. @Component public class CorrelatingRestClient implements RestClient { private RestTemplate restTemplate = new RestTemplate(); @Override public String getForString(String uri) { String correlationId = RequestCorrelation.getId(); HttpHeaders httpHeaders = new HttpHeaders(); httpHeaders.set(RequestCorrelation.CORRELATION_ID, correlationId); LOGGER.info("start REST request to {} with correlationId {}", uri, correlationId); //TODO: error-handling and fault-tolerance in production ResponseEntity response = restTemplate.exchange(uri, HttpMethod.GET, new HttpEntity(httpHeaders), String.class); LOGGER.info("completed REST request to {} with correlationId {}", uri, correlationId); return response.getBody(); } } //... calling Class public String exampleMethod() { RestClient restClient = new CorrelatingRestClient(); return restClient.getForString(URI_LOCATION); //correlation id handling completely abstracted to RestClient impl } Making this work for asynchronous requests… The code included above works fine when you are handling all of your requests synchronously, but it is often a good idea in a SOA/microservice platform to handle requests in a non-blocking asynchronous manner. In Spring this can be achieved by using the DeferredResult Class in combination with the Servlet 3 asynchronous support. The problem with using ThreadLocal variables within the asynchronous approach is that the Thread that initially handles the request (and creates the DeferredResult/Future) will not be the Thread doing the actual processing. Accordingly, a bit of glue code is needed to ensure that the correlation id is propagated across the Threads. This can be achieved by extending Callable with the required functionality: (don’t worry if example Calling Class code doesn’t look intuitive – this adaption between DeferredResults and Futures is a necessary evil within Spring, and the full code including the boilerplate ListenableFutureAdapter is in my GitHub repo): public class CorrelationCallable implements Callable { private String correlationId; private Callable callable; public CorrelationCallable(Callable targetCallable) { correlationId = RequestCorrelation.getId(); callable = targetCallable; } @Override public V call() throws Exception { RequestCorrelation.setId(correlationId); return callable.call(); } } //... Calling Class @RequestMapping("externalNews") public DeferredResult externalNews() { return new ListenableFutureAdapter<>(service.submit(new CorrelationCallable<>(externalNewsService::getNews))); } And there we have it – the propagation of correlation id regardless of the synchronous/asynchronous nature of processing! You can clone the Github report containing my asynchronous example, and execute the application by running mvn spring-boot:run at the command line. If you access http://localhost:8080/externalNewsin your browser (or via curl) you will see something similar to the following in your Spring Boot console, which clearly demonstrates a correlation id being generated on the initial request, and then this being propagated through to a simulated external call (have a look in the ExternalNewsServiceRest Class to see how this has been implemented): [nio-8080-exec-1] u.c.t.e.c.w.f.CorrelationHeaderFilter : No correlationId found in Header. Generated : d205991b-c613-4acd-97b8-97112b2b2ad0 [pool-1-thread-1] u.c.t.e.c.w.c.CorrelatingRestClient : start REST request to http://localhost:8080/news with correlationId d205991b-c613-4acd-97b8-97112b2b2ad0 [nio-8080-exec-2] u.c.t.e.c.w.f.CorrelationHeaderFilter : Found correlationId in Header : d205991b-c613-4acd-97b8-97112b2b2ad0 [pool-1-thread-1] u.c.t.e.c.w.c.CorrelatingRestClient : completed REST request to http://localhost:8080/news with correlationId d205991b-c613-4acd-97b8-97112b2b2ad0 Conclusion I’m quite happy with this simple prototype, and it does meet the two goals I listed above. Future work will include writing some tests for this code (shame on me for not TDDing!), and also extend this functionality to a more realistic example. I would like to say a massive thanks to Sam, not only for sharing his knowledge at the great talks at Geecon, but also for taking time to respond to my emails. If you’re interested in microservices and related work I can highly recommend Sam’s Microservice book which is available in Early Access at O’Reilly. I’ve enjoyed reading the currently available chapters, and having implemented quite a few SOA projects recently I can relate to a lot of the good advice contained within. I’ll be following the development of this book with keen interest! If you have any comments or thoughts then please do share them via the comment below, or feel free to get in touch via the usual mechanisms! References I used Tomasz Nurkiewicz’s excellent blog several times for learning how best to wire up all of the DeferredResult/Future code in Spring: http://www.nurkiewicz.com/2013/03/deferredresult-asynchronous-processing.html

neo4j is quite awesome for pretty much everything . here is an extract i found from this blog whenever someone gives you a problem, think graphs . they are the most fundamental and flexible way of representing any kind of a relationship, so it’s about a 50-50 shot that any interesting design problem has a graph involved in it. make absolutely sure you can’t think of a way to solve it using graphs before moving on to other solution types. as a baby step, let us try to visualize an xml as a graph. we can start with a simple xml here, belgian waffles $5.95 two of our famous belgian waffles with plenty of real maple syrup 650 it is a simple ‘breakfast menu’ with just one item, ‘ belgian waffles ‘. neo4j has a neat interface which lets us visualize the graph we created. the graph for the above xml looks slick.. the above xml has 6 tags and we have 6 nodes in our graph. the nested tags/nodes has a ‘child-of’ relationship with the parent tag/node. in the graph above, node numbered 0 is the root node – corresponds to the tag node numbered 1 is the immediate child of the root – corresponds to the tag nodes numbered from 2-4 are the tags which come under tag, viz , , and transforming xml.. let us parse the xml into a data structure which can be easily persisted using neo4j . personally, i would prefer sax parser as there are serious memory constraints when creating a dom object (really painful if you talking big data as xml). additionally sax parsing gives you unlimited freedom to do whatever you want to do with the xml. this is how i represent a node, using the object xmlelement . each xmlelement is identified by an id. this is basically an integer or long. only thing to make sure is that the number has to be unique across all xmlelements. publicclassxmlelement { privatestring tagname; privatestring tagvalue; privatemap attributes = newhashmap(); privatehierarchyidentifier hierarchyidentifier; privateintparentid; privatebooleanpersisted; //setters and getters for the members publicstring getatrributestring(){ objectmapper jsonmapper = newobjectmapper(); try{ returnjsonmapper.writevalueasstring(this.attributes); } catch(jsonprocessingexception e) { logger.severe(e.getmessage()); e.printstacktrace(); } returnnull; } } the xml tag name and value are stored as string and the attributes are stored as a map. also, there is another member object, hierarchyidentifier publicclasshierarchyidentifier { privateintdepth; privateintwidth; privateintid; //getters and setters for member element } the hierarchyidentifier class contains the id class which is used to identify an xmlelement , which translates to identifying an xml tag. each xmlelement has the id of it’s parent stored. and after the parse, the xml should be represented as a map of and each xmlelement identified by the corresponding integer id. you can see the sax parser here .. so we pass the map to the graphwriter object. we create a node with the following properties value – the value of the xml tag id – the id of the current node parent – the id of the parent tag attributes – the string representation of the tag attributes also, the node has the following labels node – for all nodes parent – for the seed ( highest parent ) node xml tag name – say, the node for the tag will have label food currently, there is only a single type of relationship, ‘child_of’ , between immediate nested tags. see the below the example for a bigger xml belgian waffles $5.95 two of our famous belgian waffles with plenty of real maple syrup 650 masala dosa $10.95 south india's famous slim pancake with mashed potatoes 650 25 i have added a second item to the ‘breakfast_menu’.. the legendary masala dosa . also, the second food item has a new child tag, . and the resultant graph looks prettier the sets of tags,relationships and properties can be seen in the neo4j local server interface. the entire codebase can be found here in github .. visualizing as a graphgist an easy representation can be done in graphgist. i have created the graph of this xml. here is the link to the gist . also, the gist script can be found in the github gist here ...

Originally written by John Hugg I’d like to take a quick moment to address some myths and misconceptions about VoltDB. Many people selling products who view VoltDB as competition seem to be repeating them. As you’ll read, much of what’s said is just plain FUD. VoltDB is an in-memory database that has benchmarked at over 3 million transactions a second on bare metal, and recently crushed previous performance records in the cloud, posting eye-popping YCSB (Yahoo! Cloud Service Benchmark) numbers on AWS, Amazon’s cloud platform. It’s fully transactional, supports full disk durability, and has very low latency and fully native high availability. It’s not surprising competing interests look for FUD. Note the comments below apply to the current shipping version of VoltDB at the time of this posting (4.2), unless otherwise noted. Myth #1: “VoltDB requires stored procedures.” This was true for 1.0, but no one seems to notice it’s been false since we shipped 1.1 in 2010. VoltDB supports unforeseen SQL without any stored procedure use. We have users in production who have never used a single stored procedure. SQL queries can be run through: JDBC Our command line SQL console A web portal built into every server The HTTP/JSON interface Native client drivers written in C++, C#, PHP, Python, Node.js and even Java. Community-provided language drivers ODBC drivers (in development to ship in Q2) There is one restriction: VoltDB doesn’t support external transaction control; VoltDB is permanently in auto-commit mode. Note this offers more transactionality than most NoSQL systems, and is a shared restriction of several NewSQL or “In-Memory” SQL systems. For those systems that do offer external transaction control, I’m not aware of any that publish benchmarks where transactions require multiple round trips to the client. This is because it’s generally understood to be a bad thing in a performance-critical system. In addition to ad-hoc SQL, VoltDB auto-generates optimized procedures for various CRUD operations (create, read, update, delete). For example, to fetch a row by primary key from table FOO, I can call a procedure named “FOO.select” with the key value as a parameter. Myth #2: “VoltDB doesn’t support ad-hoc SQL.” This is just a rephrasing of Myth #1 and is still false. Myth #3: “VoltDB is slow unless I use stored procedures.” Well, no. VoltDB can run faster with stored procedures, but it’s still fast if they are not used. In our internal benchmarks on pretty cheap single-socket hardware, we can run about 50k write statements per second, per host with full durability. That’s a bit lower than we can achieve with stored procedures, so we’re doing some engineering work to close the gap. Still, on a 3-node cluster costing about $3,000 for hardware, we can run well over 100k SQL updates per second with durability and double redundancy. If your current system can run that fast, let us know. Also, the auto-generated CRUD procedures mentioned above run at the full speed of pre-optimized procedures. Using just CRUD, VoltDB is the fastest durable Key-Value store I’m aware of. Myth #4: “I have to know Java to use VoltDB.” As of VoltDB 3.0, released over a year ago, (we’re on V4.2 today), a user can build VoltDB apps and run the server without ever directly interacting with the Java CLI tools or any Java code. Client apps can also be written without Java. As mentioned previously, VoltDB supports client drivers in many languages, with community-contributed drivers in others. If you do want to leverage VoltDB’s ability to perform multi-statement, transactional logic, only the most basic understanding of Java is needed to build stored procedures. We provide example applications in our kit that can be copied from and/or modified. We even support declaring procedures in Groovy right in your DDL. Myth #5: “VoltDB has garbage collection problems because it is written in Java.” The short answer is this is just not true. First, VoltDB uses native C++ code for all of its data storage and SQL execution paths. This is done to avoid putting long-lived data on the Java heap, but also to allow very fine control of memory use. VoltDB’s special native data structures use less memory than competing systems and actively fight allocation fragmentation. What’s left on the Java heap is either near-immortal configuration and setup data, or very short-lived per-transaction data. This is almost the perfect workload for the Java garbage collector, so there is minimal impact to the user’s workload. For more detail, read Ariel Weisberg’s excellent blog post on the topic. Myth #6: “VoltDB’s SQL is very limited.” This misconception is based on historical truth, but is way out of date in 2014. The original H-Store project, upon which VoltDB is based, started with very limited, OLTP-focused SQL, but much has changed in the past years. VoltDB is rapidly approaching SQL-92 functionality for DML and DQL SQL. One of the final holdouts, subselects, shipped in part in 4.2; more functionality will ship in upcoming versions. In addition to this ANSI SQL baseline, we’ve added materialized view support, function-based index support, native JSON functionality and more. We have a lot more coming in 2014. We’re proud to announce all of this is fully Unicode-compliant and is in production in several uses in countries with multi-byte character sets. Myth #7: “Yes, VoltDB supports cross-partition transactions, but they are too slow to use.” Yes we do, and no they aren’t too slow to use. Keep in mind that NoSQL systems often don’t offer cross-partition transactions and many other NewSQL systems have functionality or performance limitations. Few benchmarks stress this functionality, so I’ll be as clear as possible here. Traditionally, VoltDB’s cross-partition transactions have been slower than our partitioned operations. Additional coordination needs to happen to make these transactions ACID at the “serializable” isolation level at which all VoltDB transactions run. Furthermore, these operations don’t scale with cluster size and sometimes can vary in performance, depending on your networking performance. But are they too slow to use? I’ll let you be the judge. Cross-partition writes run on the order of 1,000 per second,regardless of cluster size. This number can vary by an order of magnitude depending on hardware. Cross partition reads run on the order of 50,000 per second, regardless of cluster size. Again, there is some variability, but less than with writes. It’s possible to send writes to each partition individually using the “Send Everywhere” pattern. These writes are transactional at each partition, but individual partitions might fail and rollback independently of others. A common pattern we see is a mix of single-partition writes and cross-partition reads. Cross-partition writes are often used to update infrequently changing lookup tables. Of course at 1000 per second, they can still change fairly frequently. Note that if your read operation is scanning millions of tuples, you won’t see the same throughput, but if it’s populating a schema supported leaderboard or computing a time window statistic using a materialized view, you will. The bottom line is that most of our customers not only use cross-partition transactions, but also these kinds of operations are one of the key benefits VoltDB offers. We are always working to improve the performance of cross-partition operations, and in recent releases we have made a lot of progress. In 4.0 we introduced read optimizations that got us to the 50k/second number. We also have write optimizations in the pipeline. In Conclusion Many of these myths and misconceptions are firmly rooted in the world of traditional RDBMSs. VoltDB is a NewSQL database. When evaluating VoltDB, consider the non-traditional features that set us head and shoulders above traditional RDBMSs, NoSQL and other NewSQL options: Intelligent, asynchronous clients allow tremendous throughput per-client, as well as terrific management. Native export connectors push tuples into downstream systems like message queues, analytic data-warehouses and Hadoop Active-Active intra-cluster replication with sub-millisecond latency ensures perfectly consistent high availability and redundant durability. Seamless node failure and replacement without impacting serializable ACID semantics. Transactional cluster expansion with low performance impact and without impacting serializable ACID semantics provides near-limitless scalability. 1500 active, connected clients per server node with minimal latency impact meets the needs of most high-velocity, high transaction volume businesses. VoltDB is open source (https://github.com/VoltDB/voltdb) If you want to hear more, or if you have a use case you want to talk to us about, contact us. We’d love to hear from you. If you’d like to try VoltDB out yourself, you can download a free trial here.

After attending Sam Newman’s microservice talks at Geecon last week I started to think more about what is most likely an essential feature of service-oriented/microservice platforms for monitoring, reporting and diagnostics: correlation ids. Correlation ids allow distributed tracing within complex service oriented platforms, where a single request into the application can often be dealt with by multiple downstream service. Without the ability to correlate downstream service requests it can be very difficult to understand how requests are being handled within your platform. I’ve seen the benefit of correlation ids in several recent SOA projects I have worked on, but as Sam mentioned in his talks, it’s often very easy to think this type of tracing won’t be needed when building the initial version of the application, but then very difficult to retrofit into the application when you do realise the benefits (and the need for!). I’ve not yet found the perfect way to implement correlation ids within a Java/Spring-based application, but after chatting to Sam via email he made several suggestions which I have now turned into a simple project using Spring Boot to demonstrate how this could be implemented. Why? During both of Sam’s Geecon talks he mentioned that in his experience correlation ids were very useful for diagnostic purposes. Correlation ids are essentially an id that is generated and associated with a single (typically user-driven) request into the application that is passed down through the stack and onto dependent services. In SOA or microservice platforms this type of id is very useful, as requests into the application typically are ‘fanned out’ or handled by multiple downstream services, and a correlation id allows all of the downstream requests (from the initial point of request) to be correlated or grouped based on the id. So called ‘distributed tracing’ can then be performed using the correlation ids by combining all the downstream service logs and matching the required id to see the trace of the request throughout your entire application stack (which is very easy if you are using a centralised logging framework such as logstash) The big players in the service-oriented field have been talking about the need for distributed tracing and correlating requests for quite some time, and as such Twitter have created their open source Zipkin framework (which often plugs into their RPC framework Finagle), and Netflix has open-sourced their Karyon web/microservice framework, both of which provide distributed tracing. There are of course commercial offering in this area, one such product being AppDynamics, which is very cool, but has a rather hefty price tag. Creating a proof-of-concept in Spring Boot As great as Zipkin and Karyon are, they are both relatively invasive, in that you have to build your services on top of the (often opinionated) frameworks. This might be fine for some use cases, but no so much for others, especially when you are building microservices. I’ve been enjoying experimenting with Spring Boot of late, and this framework builds on the much known and loved (at least by me :-) ) Spring framework by providing lots of preconfigured sensible defaults. This allows you to build microservices (especially ones that communicate via RESTful interfaces) very rapidly. The remainder of this blog pos explains how I implemented a (hopefully) non-invasive way of implementing correlation ids. Goals Allow a correlation id to be generated for a initial request into the application Enable the correlation id to be passed to downstream services, using as method that is as non-invasive into the code as possible Implementation I have created two projects on GitHub, one containing an implementation where all requests are being handled in a synchronous style (i.e. the traditional Spring approach of handling all request processing on a single thread), and also one for when an asynchronous (non-blocking) style of communication is being used (i.e., using the Servlet 3 asynchronous support combined with Spring’s DeferredResult and Java’s Futures/Callables). The majority of this article describes the asynchronous implementation, as this is more interesting: Spring Boot asynchronous (DeferredResult + Futures) communication correlation id Github repo The main work in both code bases is undertaken by the CorrelationHeaderFilter, which is a standard Java EE Filter that inspects the HttpServletRequest header for the presence of a correlationId. If one is found then we set a ThreadLocal variable in the RequestCorrelation Class (discussed later). If a correlation id is not found then one is generated and added to the RequestCorrelation Class: public class CorrelationHeaderFilter implements Filter { //... @Override public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException { final HttpServletRequest httpServletRequest = (HttpServletRequest) servletRequest; String currentCorrId = httpServletRequest.getHeader(RequestCorrelation.CORRELATION_ID_HEADER); if (!currentRequestIsAsyncDispatcher(httpServletRequest)) { if (currentCorrId == null) { currentCorrId = UUID.randomUUID().toString(); LOGGER.info("No correlationId found in Header. Generated : " + currentCorrId); } else { LOGGER.info("Found correlationId in Header : " + currentCorrId); } RequestCorrelation.setId(currentCorrId); } filterChain.doFilter(httpServletRequest, servletResponse); } //... private boolean currentRequestIsAsyncDispatcher(HttpServletRequest httpServletRequest) { return httpServletRequest.getDispatcherType().equals(DispatcherType.ASYNC); } The only thing is this code that may not instantly be obvious is the conditional check currentRequestIsAsyncDispatcher(httpServletRequest), but this is here to guard against the correlation id code being executed when the Async Dispatcher thread is running to return the results (this is interesting to note, as I initially didn’t expect the Async Dispatcher to trigger the execution of the filter again?) Here is the RequestCorrelation Class, which contains a simple ThreadLocal static variable to hold the correlation id for the current Thread of execution (set via the CorrelationHeaderFilter above) public class RequestCorrelation { public static final String CORRELATION_ID = "correlationId"; private static final ThreadLocal id = new ThreadLocal(); public static String getId() { return id.get(); } public static void setId(String correlationId) { id.set(correlationId); } } Once the correlation id is stored in the RequestCorrelation Class it can be retrieved and added to downstream service requests (or data store access etc) as required by calling the static getId() method within RequestCorrelation. It is probably a good idea to encapsulate this behaviour away from your application services, and you can see an example of how to do this in a RestClient Class I have created, which composes Spring’s RestTemplate and handles the setting of the correlation id within the header transparently from the calling Class. @Component public class CorrelatingRestClient implements RestClient { private RestTemplate restTemplate = new RestTemplate(); @Override public String getForString(String uri) { String correlationId = RequestCorrelation.getId(); HttpHeaders httpHeaders = new HttpHeaders(); httpHeaders.set(RequestCorrelation.CORRELATION_ID, correlationId); LOGGER.info("start REST request to {} with correlationId {}", uri, correlationId); //TODO: error-handling and fault-tolerance in production ResponseEntity response = restTemplate.exchange(uri, HttpMethod.GET, new HttpEntity(httpHeaders), String.class); LOGGER.info("completed REST request to {} with correlationId {}", uri, correlationId); return response.getBody(); } } //... calling Class public String exampleMethod() { RestClient restClient = new CorrelatingRestClient(); return restClient.getForString(URI_LOCATION); //correlation id handling completely abstracted to RestClient impl } Making this work for asynchronous requests… The code included above works fine when you are handling all of your requests synchronously, but it is often a good idea in a SOA/microservice platform to handle requests in a non-blocking asynchronous manner. In Spring this can be achieved by using the DeferredResult Class in combination with the Servlet 3 asynchronous support. The problem with using ThreadLocal variables within the asynchronous approach is that the Thread that initially handles the request (and creates the DeferredResult/Future) will not be the Thread doing the actual processing. Accordingly, a bit of glue code is needed to ensure that the correlation id is propagated across the Threads. This can be achieved by extending Callable with the required functionality: (don’t worry if example Calling Class code doesn’t look intuitive – this adaption between DeferredResults and Futures is a necessary evil within Spring, and the full code including the boilerplate ListenableFutureAdapter is in my GitHub repo): public class CorrelationCallable implements Callable { private String correlationId; private Callable callable; public CorrelationCallable(Callable targetCallable) { correlationId = RequestCorrelation.getId(); callable = targetCallable; } @Override public V call() throws Exception { RequestCorrelation.setId(correlationId); return callable.call(); } } //... Calling Class @RequestMapping("externalNews") public DeferredResult externalNews() { return new ListenableFutureAdapter<>(service.submit(new CorrelationCallable<>(externalNewsService::getNews))); } And there we have it – the propagation of correlation id regardless of the synchronous/asynchronous nature of processing! You can clone the Github report containing my asynchronous example, and execute the application by running mvn spring-boot:run at the command line. If you access http://localhost:8080/externalNews in your browser (or via curl) you will see something similar to the following in your Spring Boot console, which clearly demonstrates a correlation id being generated on the initial request, and then this being propagated through to a simulated external call (have a look in the ExternalNewsServiceRest Class to see how this has been implemented): [nio-8080-exec-1] u.c.t.e.c.w.f.CorrelationHeaderFilter : No correlationId found in Header. Generated : d205991b-c613-4acd-97b8-97112b2b2ad0 [pool-1-thread-1] u.c.t.e.c.w.c.CorrelatingRestClient : start REST request to http://localhost:8080/news with correlationId d205991b-c613-4acd-97b8-97112b2b2ad0 [nio-8080-exec-2] u.c.t.e.c.w.f.CorrelationHeaderFilter : Found correlationId in Header : d205991b-c613-4acd-97b8-97112b2b2ad0 [pool-1-thread-1] u.c.t.e.c.w.c.CorrelatingRestClient : completed REST request to http://localhost:8080/news with correlationId d205991b-c613-4acd-97b8-97112b2b2ad0 Conclusion I’m quite happy with this simple prototype, and it does meet the two goals I listed above. Future work will include writing some tests for this code (shame on me for not TDDing!), and also extend this functionality to a more realistic example. I would like to say a massive thanks to Sam, not only for sharing his knowledge at the great talks at Geecon, but also for taking time to respond to my emails. If you’re interested in microservices and related work I can highly recommend Sam’s Microservice book which is available in Early Access at O’Reilly. I’ve enjoyed reading the currently available chapters, and having implemented quite a few SOA projects recently I can relate to a lot of the good advice contained within. I’ll be following the development of this book with keen interest! If you have any comments or thoughts then please do share them via the comment below, or feel free to get in touch via the usual mechanisms! References I used Tomasz Nurkiewicz’s excellent blog several times for learning how best to wire up all of the DeferredResult/Future code in Spring: http://www.nurkiewicz.com/2013/03/deferredresult-asynchronous-processing.html

Hibernate provides SQL logging out of the box, but such logging only shows prepared statements, and not the actual SQL queries sent to the database. It also does not log the execution time of each query, which is useful for performance troubleshooting. This blog post will go over how to setup Hibernate query logging, and then compare it to the logging that can be obtained with log4jdbc . The Hibernate query logging functionality Hibernate does not log the real SQL queries sent to the database. This is because Hibernate interacts with the database via the JDBC driver, to which it sends prepared statements but not the actual queries. So Hibernate can only log the prepared statements and the values of their binding parameters, but not the actual SQL queries themselves. This is how a query looks like when logged by Hibernate: select /* load your.package.Employee */ this_.code, ... from employee this_ where this_.employee_id=? TRACE 12-04-2014@16:06:02 BasicBinder - binding parameter [1] as [NUMBER] - 1000 See this post Hibernate Debugging - Finding the origin of a Query for how to setup this type of logging. Using log4jdbc For a developer it's useful to be able to copy paste a query from the log and be able to execute the query directly in an SQL client, but the variable placeholders ? make that unfeasible. Log4jdbc in an open source tool that allows to do just that, and more. Log4jdbc is a spy driver that will wrap itself around the real JDBC driver, logging queries as they go through it. The version linked from this post provides Spring integration, unlike several other log4jdbc forks. Setting up log4jdbc First include the log4jdbc-remix library in your pom.xml. This library is a fork of the original log4jdbc: org.lazyluke log4jdbc-remix0.2.7 Next, find in the Spring configuration the definition of the data source. As an example, when using the JNDI lookup element this is how the data source looks like: After finding the data source definition, rename it to the following name: Then define a new log4jdbc data source that wraps the real data source, and give it the original name: With this configuration, the query logging should already be working. It's possible to customize the logging level of the several log4jdbc loggers available. The original log4jdbc documentation provides more information on the available loggers: jdbc.sqlonly: Logs only SQL jdbc.sqltiming: Logs the SQL, post-execution, including timing execution statistics jdbc.audit: Logs ALL JDBC calls except for ResultSets jdbc.resultset: all calls to ResultSet objects are logged jdbc.connection: Logs connection open and close events The jdbc.audit logger is especially useful to validate the scope of transactions, as it logs the begin/commit/rollback events of a database transaction. This is the proposed log4j configuration that will print only the SQL queries together with their execution time: Conclusion Using log4jdbc does imply some initial setup, but once it's in place it's really convenient to have. Having a true query log is also useful for performance troubleshooting (to be described in a future post).

Introduction When I am looking for a forum post related to SQL Server, one of the junior professional is asking how to use a DO…WHILE loop is MS SQL Server. Several people wrote their opinion related to it. Everyone is saying to use WHILE loop and some of them suggesting with T-SQL structure of CURSOR with WHILE LOOP. Obviously, when a junior professional is learning MS SQL server, the question in mind arises: is there DO… WHILE, REPEAT … UNTIL loop present in MS SQL Server as there is in C or C++ etc? No one is answering directly on the forum whether we can use DO… WHILE or REPEAT … UNTIL in MS SQL Server or NOT. If yes, how can we implement them? DO… WHILE in MS SQL Sever First we look at the algorithm of DO… WHILE. SET X = 1 DO PRINT X SET X = X + 1 WHILE X <= 10 Now we try to implement it in MS SQL Server. DECLARE @X INT=1; WAY: --> Here the DO statement PRINT @X; SET @X += 1; IF @X<=10 GOTO WAY; --> Here the WHILE @X<=1 REPEAT… UNTIL First we look at the algorithm of REPEAT... UNTIL SET X = 1 REPEAT PRINT X SET X = X + 1 UNTIL X > 10 Now we try to implement it in MS SQL Server DECLARE @X INT = 1; WAY: -- Here the REPEAT statement PRINT @X; SET @X += 1; IFNOT(@X >1 0) GOTO WAY; -- Here the UNTIL @X>10 So we see that it is possible, but a little complicated… So most developers prefer the WHILE loop in MS SQL Server.

Cloudfoundry's java buildpack is supporting some popular jvm based applications. This article is oriented to the audiences already with experience of cloudfoundry/heroku buildpack who want to have more understanding of how buildpack and cloudfoundry works internally. cf push app -p app.war -b build-pack-url The above command demonstrates the usage of pushing a war file to cloudfoundry by using a custom buildpack (E.g. https://github.com/cloudfoundry/java-buildpack). However, what exactly happens inside, or how cloudfoundry bootstrap the war file with tomcat? There are three contracts phase that bridge communication between buildpack and cloudfoundry. The three phases are detect, compile and release, which are three ruby shell scripts: Java buildpack has multiple sub components, while each of them has all of these three phases (E.g. tomcat is one of the sub components, while it contained another layer of sub components). Detect Phase: detect phase is to check whether a particular buildpack/component applies to the deployed application. Take the war file example, tomcat applies only when https://github.com/cloudfoundry/java-buildpack/blob/master/lib/java_buildpack/container/tomcat.rb is true: def supports? web_inf? && !JavaBuildpack::Util::JavaMainUtils.main_class(@application) end The above code means, the tomcat applies when the application has a WEB-INF folder andthisisnot a main class bootstrapped application. Compile Phase: Compile phase would be the major/comprehensive work for a customized buildpack, while it is trying to build a file system on a lxc container. Take the example of our war application and tomcat example. In https://github.com/cloudfoundry/java-buildpack/blob/master/lib/java_buildpack/container/tomcat/tomcat_instance.rb def compile download(@version, @uri) { |file| expand file } link_to(@application.root.children, root) @droplet.additional_libraries << tomcat_datasource_jar if tomcat_datasource_jar.exist? @droplet.additional_libraries.link_to web_inf_lib end def expand(file) with_timing "Expanding Tomcat to #{@droplet.sandbox.relative_path_from(@droplet.root)}" do FileUtils.mkdir_p @droplet.sandbox shell "tar xzf #{file.path} -C #{@droplet.sandbox} --strip 1 --exclude webapps 2>&1" @droplet.copy_resources end The above code is all about preparing the tomcat and link the application files, so the application files will be available for the tomcat classpath. Before going to the code, we have to understand the working directory when the above code executes: . => working directory .app => @application, contains the extracted war archive .buildpack/tomcat => @droplet.sandbox .buildpack/jdk .buildpack/other needed components Inside compile method: download method will download tomcat binary file (specified here: https://github.com/cloudfoundry/java-buildpack/blob/master/config/tomcat.yml), and then extract the archive file to @droplet.sandbox directory. Then copy the resources folder's files to https://github.com/cloudfoundry/java-buildpack/tree/master/resources/tomcat/conf to @droplet.sandbox/conf Symlink the @droplet.sandbox/webapps/ROOT to .app/ Symlink additional libraries (comes from other component rather than application) to the WEB-INF/lib Note: All the symlinks use relative path, since when the container deployed to DEA, the absolute paths would be different. RELEASE PHASE: Release phase is to setup instructions of how to start tomcat. Look at the code in :https://github.com/cloudfoundry/java-buildpack/blob/master/lib/java_buildpack/container/tomcat.rb def command @droplet.java_opts.add_system_property 'http.port', '$PORT' [ @droplet.java_home.as_env_var, @droplet.java_opts.as_env_var, "$PWD/#{(@droplet.sandbox + 'bin/catalina.sh').relative_path_from(@droplet.root)}", 'run' ].flatten.compact.join(' ') end The above code does: Add java system properties http.port (referenced in tomcat server.xml) with environment properties ($PORT), this is the port on the DEA bridging to the lxc container already setup when the container was provisioned. instruction of how to run the tomcat Eg. "./bin/catalina.sh run"

Cyclop is a web-based tool for querying Cassandra databases with features like syntax highlighting and query completion.

It's not always immediate why and in which part of the program is Hibernate generating a given SQL query, especially if we are dealing with code that we did not write ourselves. This post will go over how to configure Hibernate query logging, and use that together with other tricks to find out why and where in the program a given query is being executed. What does the Hibernate query log look like Hibernate has built-in query logging that looks like this: select /* load your.package.Employee */ this_.code, ... from employee this_ where this_.employee_id=? TRACE 12-04-2014@16:06:02 BasicBinder - binding parameter [1] as [NUMBER] - 1000 Why can't Hibernate log the actual query ? Notice that what is logged by Hibernate is the prepared statement sent by Hibernate to the JDBC driver plus it's parameters. The prepared statement has ? in the place of the query parameters, and the parameter values themselves are logged just bellow the prepared statement. This is not the same as the actual query sent to the database, as there is no way for Hibernate to log the actual query. The reason for this is that Hibernate only knows about the prepared statements and the parameters that it sends to the JDBC driver, and it's the driver that will build the actual queries and then send them to the database. In order to produce a log with the real queries, a tool like log4jdbc is needed, which will be the subject of another post. How to find out the origin of the query The logged query above contains a comment that allows to identify in most cases the origin of the query: if the query is due to a load by ID the comment is /* load your.entity.Name */, if it's a named query then the comment will contain the name of the query. If it's a one to many lazy initialization the comment will contain the name of the class and the property that triggered it, etc. In many cases the query comment created by is enough to identify the origin of the query. Setting up the Hibernate query log In order to obtain a query log, the following flags need to be set in the configuration of the session factory: ... true true true The example above is for Spring configuration of an entity manager factory. This is the meaning of the flags: show_sql enables query logging format_sql pretty prints the SQL use_sql_comments adds an explanatory comment In order to log the query parameters, the following log4j or equivalent configuration is needed: If everything else fails If the query comment added by the option use_sql_comments is not sufficient, then we can start by identifying the entity returned by the query based on the table names involved, and put a breakpoint in the constructor of the returned entity. If the entity does not have a constructor, then we can create one and put the breakpoint in the call to super(): @Entity public class Employee { public Employee() { super(); // put the breakpoint here } ... } When the breakpoint is hit, go to the IDE debug view containing the stack call of the program and go through it from top to bottom. The place where the query was made in the program will be there in the call stack.

Take another look at @OneToOne with a shared primary key.

this post was originally written by muhammad irfan here on the percona mysql support team, we often see issues where a customer is complaining about replication delays – and many times the problem ends up being tied to mysql replication slave lag. this of course is nothing new for mysql users and we’ve had a few posts here on the mysql performance blog on this topic over the years (two particularly popular post in the past were: “ reasons for mysql replication lag ” and “ managing slave lag with mysql replication ,” both by percona ceo peter zaitsev) . in today’s post, however, i will share some new ways of identifying delays in replication – including possible causes of lagging slaves – and how to cure this problem. how to identify replication delay mysql replication works with two threads, io_thread & sql_thread. io_thread connects to a master, reads binary log events from the master as they come in and just copies them over to a local log file called relaylog . on the other hand, sql_thread reads events from a relay log stored locally on the replication slave (the file that was written by io thread) and then applies them as fast as possible. whenever replication delays, it’s important to discover first whether it’s delaying on slave io_thread or slave sql_thread. normally, i/o thread would not cause a huge replication delay as it is just reading the binary logs from the master. however, it depends on the network connectivity, network latency… how fast is that between the servers. the slave i/o thread could be slow because of high bandwidth usage. usually, when the slave io_thread is able to read binary logs quickly enough it copies and piles up the relay logs on the slave – which is one indication that the slave io_thread is not the culprit of slave lag. on the other hand, when the slave sql_thread is the source of replication delays it is probably because of queries coming from the replication stream are taking too long to execute on the slave. this is sometimes because of different hardware between master/slave, different schema indexes, workload. moreover, the slave oltp workload sometimes causes replication delays because of locking. for instance, if a long-running read against a myisam table blocks the sql thread, or any transaction against an innodb table creates an ix lock and blocks ddl in the sql thread. also, take into account that slave is single threaded prior to mysql 5.6, which would be another reason for delays on the slave sql_thread. let me show you via master status/slave status example to identify either slave is lagging on slave io_thread or slave sql_thread. mysql-master> show master status; +------------------+--------------+------------------+------------------------------------------------------------------+ | file | position | binlog_do_db | binlog_ignore_db | executed_gtid_set | +------------------+--------------+------------------+------------------------------------------------------------------+ | mysql-bin.018196 | 15818564 | | bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:1-2331947 | +------------------+--------------+------------------+------------------------------------------------------------------+ mysql-slave> show slave status\g *************************** 1. row *************************** slave_io_state: queueing master event to the relay log master_host: master.example.com master_user: repl master_port: 3306 connect_retry: 60 master_log_file: mysql-bin.018192 read_master_log_pos: 10050480 relay_log_file: mysql-relay-bin.001796 relay_log_pos: 157090 relay_master_log_file: mysql-bin.018192 slave_io_running: yes slave_sql_running: yes replicate_do_db: replicate_ignore_db: replicate_do_table: replicate_ignore_table: replicate_wild_do_table: replicate_wild_ignore_table: last_errno: 0 last_error: skip_counter: 0 exec_master_log_pos: 5395871 relay_log_space: 10056139 until_condition: none until_log_file: until_log_pos: 0 master_ssl_allowed: no master_ssl_ca_file: master_ssl_ca_path: master_ssl_cert: master_ssl_cipher: master_ssl_key: seconds_behind_master: 230775 master_ssl_verify_server_cert: no last_io_errno: 0 last_io_error: last_sql_errno: 0 last_sql_error: replicate_ignore_server_ids: master_server_id: 2 master_uuid: bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:2-973166 master_info_file: /var/lib/mysql/i1/data/master.info sql_delay: 0 sql_remaining_delay: null slave_sql_running_state: reading event from the relay log master_retry_count: 86400 master_bind: last_io_error_timestamp: last_sql_error_timestamp: master_ssl_crl: master_ssl_crlpath: retrieved_gtid_set: bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:2-973166 executed_gtid_set: bb11b389-d2a7-11e3-b82b-5cf3fcfc8f58:2-973166, ea75c885-c2c5-11e3-b8ee-5cf3fcfc9640:1-1370 auto_position: 1 this clearly suggests that the slave io_thread is lagging and obviously because of that the slave sql_thread is lagging, too, and it yields replication delays. as you can see the master log file is mysql-bin.018196 (file parameter from master status) and slave io_thread is on mysql-bin.018192 ( master_log_file from slave status ) which indicates slave io_thread is reading from that file, while on master it’s writing on mysql-bin.018196 , so the slave io_thread is behind by 4 binlogs. meanwhile, the slave sql_thread is reading from same file i.e. mysql-bin.01819 2 (relay_master_log_file from slave status) this indicates that the slave sql_thread is applying events fast enough, but it’s lagging too, which can be observed from the difference between read_master_log_pos & exec_master_log_pos from show slave status output. you can calculate slave sql_thread lag from read_master_log_pos – exec_master_log_pos in general as long as master_log_file parameter output from show slave status and relay_master_log_file parameter from show slave status output are the same. this will give you rough idea how fast slave sql_thread is applying events. as i mentioned above, the slave io_thread is lagging as in this example then off course slave sql_thread is behind too. you can read detailed description of show slave status output fields here. also, the seconds_behind_master parameter shows a huge delay in seconds. however, this can be misleading, because it only measures the difference between the timestamps of the relay log most recently executed, versus the relay log entry most recently downloaded by the io_thread. if there are more binlogs on the master, the slave doesn’t figure them into the calculation of seconds_behind_master. you can get a more accurate measure of slave lag using pt-heartbeat from percona toolkit. so, we learned how to check replication delays – either it’s slave io_thread or slave sql_thread. now, let me provide some tips and suggestions for what exactly causing this delay. tips and suggestions what causing replication delay & possible fixes usually, the slave io_thread is behind because of slow network between master/slave. most of the time, enabling slave_compressed_protocol helps to mitigate slave io_thread lag. one other suggestion is to disable binary logging on slave as it’s io intensive too unless you required it for point in time recovery. to minimize slave sql_thread lag, focus on query optimization. my recommendation is to enable the configuration option log_slow_slave_statements so that the queries executed by slave that take more than long_query_time will be logged to the slow log. to gather more information about query performance, i would also recommend setting the configuration option log_slow_verbosity to “full”. this way we can see if there are queries executed by slave sql_thread that are taking long time to complete. you can follow my previous post about how to enable slow query log for specific time period with mentioned options here . and as a reminder, log_slow_slave_statements as variable were first introduced in percona server 5.1 which is now part of vanilla mysql from version 5.6.11 in upstream version of mysql server log_slow_slave_statements were introduced as command line option. details can be found here while log_slow_verbosity is percona server specific feature. one another reason for delay on slave sql_thread if you use row based binlog format is that if your any database table missing primary key or unique key then it will scan all rows of the table for dml on slave and causes replication delays so make sure all your tables should have primary key or unique key. check this bug report for details http://bugs.mysql.com/bug.php?id=53375 you can use below query on slave to identify which of database tables missing primary or unique key. mysql> select t.table_schema,t.table_name,engine from information_schema.tables t inner join information_schema .columns c on t.table_schema=c.table_schema and t.table_name=c.table_name group by t.table_schema,t.table_name having sum(if(column_key in ('pri','uni'), 1,0)) =0; one improvement is made for this case in mysql 5.6, where in memory hash is used slave_rows_search_algorithms comes to the rescue. note that seconds_behind_master is not updated while we read huge rbr event, so, “lagging” may be related to just that – we had not completed reading of the event. for example, in row based replication huge transactions may cause delay on slave side e.g. if you have 10 million rows table and you do “delete from table where id < 5000000″ 5m rows will be sent to slave, each row separately which will be painfully slow. so, if you have to delete oldest rows time to time from huge table using partitioning might be good alternative for this for some kind of workloads where instead using delete use drop old partition may be good and only statement is replicated because it will be ddl operation. to explain it better, let suppose you have partition1 holding rows of id’s from 1 to 1000000 , partition2 – id’s from 1000001 to 2000000 and so on so instead of deleting via statement “delete from table where id<=1000000;” you can do “alter table drop partition1;” instead. for alter partitioning operations check manual – check this wonderful post too from my colleague roman explaining possible grounds for replication delays here pt-stalk is one of finest tool from percona toolkit which collects diagnostics data when problems occur. you can setup pt-stalk as follows so whenever there is a slave lag it can log diagnostic information which we can be later analyze to check to see what exactly causing the lag. here is how you can setup pt-stalk so that it captures diagnostic data when there is slave lag: ------- pt-plug.sh contents #!/bin/bash trg_plugin() { mysqladmin $ext_argv ping &> /dev/null mysqld_alive=$? if [[ $mysqld_alive == 0 ]] then seconds_behind_master=$(mysql $ext_argv -e "show slave status" --vertical | grep seconds_behind_master | awk '{print $2}') echo $seconds_behind_master else echo 1 fi } # uncomment below to test that trg_plugin function works as expected #trg_plugin ------- -- that's the pt-plug.sh file you would need to create and then use it as below with pt-stalk: $ /usr/bin/pt-stalk --function=/root/pt-plug.sh --variable=seconds_behind_master --threshold=300 --cycles=60 [email protected] --log=/root/pt-stalk.log --pid=/root/pt-stalk.pid --daemonize you can adjust the threshold, currently its 300 seconds, combining that with –cycles, it means that if seconds_behind_master value is >= 300 for 60 seconds or more then pt-stalk will start capturing data. adding –notify-by-email option will notify via email when pt-stalk captures data. you can adjust the pt-stalk thresholds accordingly so that’s how it triggers to collect diagnostic data during problem. conclusion a lagging slave is a tricky problem but a common issue in mysql replication. i’ve tried to cover most aspects of replication delays in this post. please share in the comments section if you know of any other reasons for replication delay.

This post will go through how to setup the Hibernate Second-Level and Query caches, how they work and what are their most common pitfalls.

The Open Session in View (OSIV) pattern gives rise to different opinions in the Java development community. Let's go over OSIV and some of the pros and cons of this pattern. The problem The problem that OSIV solves is a mismatch between the Hibernate concept of session and it's lifecycle and the way that many server-side view technologies work. In a typical Java frontend application the service layer starts by querying some of the data needed to build the view. The remaining data needed can be lazy-loaded later, with the condition that the Hibernate session remains open - and there lies the problem. Between the moment that the service layer method finishes it's execution and the moment that the view is rendered, Hibernate has already committed the transaction and closed the session. When the view tries to lazy load the extra data that it needs, if finds the Hibernate session closed, causing a LazyInitializationException. The OSIV solution OSIV tackles this problem by ensuring that the Hibernate session is kept open all the way up to the rendering of the view - hence the name of the pattern. Because the session is kept open, no more LazyInitializationExceptions occur. The session or entity manager is kept open by means of a filter that is added to the request processing chain. In the case of JPA the OpenEntityManagerInViewFilter will create an entity manager at the beginning of the request, and then bind it to the request thread. The service layer will then be executed and the business transaction committed or rolled back, but the transaction manager will not remove the entity manager from the thread after the commit. When the view rendering starts, the transaction manager will then check if there is already an entity manager binded to the thread, and if so use it instead of creating a new one. After the request is processed, the filter will then unbind the entity manager from the thread. The end result is that the same entity manager used to commit the business transaction was kept around in the request thread, allowing the view rendering code to lazy load the needed data. Going back to the original problem Let's step back a moment and go back to the initial problem: the LazyInitializationException. Is this exception really a problem? This exception can also be seen as a warning sign of a wrongly written query in the service layer. When building a view and it's backing services, the developer knows upfront what data is needed, and can make sure that the needed data is loaded before the rendering starts. Several relation types such as one-to-many use lazy-loading by default, but that default setting can be overridden if needed at query time using the following syntax: select p FROM Person p left join fetch p.invoices This means that the lazy loading can be turned off on a case by case basis depending on the data needed by the view. OSIV in projects I've worked In projects I have worked that used OSIV, we could see via query logging that the database was getting hit with a high number of SQL queries, sometimes to the point that developers had to turn off the Hibernate SQL logging. The performance of these application was impacted, but it was kept manageable using second-level caches, and due to the fact that these where intranet-based applications with a limited number of users. Pros of OSIV The main advantage of OSIV is that it makes working with ORM and the database more transparent: Less queries need to be manually written Less awareness is required about the Hibernate session and how to solve LazyInitializationExceptions. Cons of OSIV OSIV seems to be easy to misuse and can accidentally introduce N+1 performance problems in the application. On projects I've worked OSIV did not work out well in the long-term. The alternative of writing custom queries that eager fetch data depending on the use case is manageable and turned out well in other projects I've worked. Alternatives to OSIV Besides the application-level solution of writing custom queries to pre-fetch the needed data, there are other framework-level aproaches to OSIV. The Seam Framework was built by some of the same developers as Hibernate , and solves the problem by introducing the notion of conversation. Can you let me know in the comments bellow your thoughts and experiences with OSIV, thanks for reading.

wanted to go through some of the basic tenets, the technical terminology related to java ee. for many people, java ee/j2ee still mean servlets, jsps or maybe struts at best. no offence or pun intended! this is not a java ee 'bible' by any means. i am not capable enough of writing such a thing! so let us line up the 'keywords' related to java ee and then look at them one by one java ee java ee apis (specifications) containers services multitiered applications components let's try to elaborate on the above mentioned points. ok. so what is java ee? 'ee' stands for enterprise edition. that essentially makes java ee - java enterprise edition. if i had to summarize java ee in a couple of sentences, it would go something like this "java ee is a platform which defines 'standard specifications/apis' which are then implemented by vendors and used for development of enterprise (distributed, 'multi-tired', robust) 'applications'. these applications are composed of modules or 'components' which use java ee 'containers' as their run-time infrastructure." what is this 'standardized platform' based upon? what does it constitute? the platform revolves around 'standard' specifications or apis . think of these as contracts defined by a standard body e.g. enterprise java beans (ejb), java persistence api (jpa), java message service (jms) etc. these contracts/specifications/apis are implemented by different vendors e.g. glassfish, oracle weblogic, apache tomee etc alright. what about containers? containers can be visualized as 'virtual/logical partitions' . each container supports a subset of the apis/specifications defined by the java ee platform they provide run-time 'services' to the 'applications' which they host the java ee specification lists 4 types of containers ejb container web container application client container applet container java ee containers i am not going to dwell into details of these containers in this post. services?? well, 'services' are nothing but a result of the vendor implementations of the standard 'specifications' (mentioned above). examples of specifications are - jersey for jax-rs (restful services), tyrus (web sockets), eclipselink (jpa), weld (cdi) etc. the 'container' is the interface between the deployed application ('service' consumer) and the application server. here is a list of 'services' which are rendered by the 'container' to the underlying 'components' (this is not an exhaustive list) persistence - offered by the java persistence api (jpa) which drives object relational mapping (orm) and an abstraction for the database operations. messaging - the java message service (jms) provides asynchronous messaging between disparate parts of your applications. contexts & dependency injection - cdi provides loosely coupled and type safe injection of resources. web services - jaxrs and jaxws provide support for rest and soap style services respectively transaction - provided by the java transaction api (jta) implementation what is a typical java ee 'application'? what does it comprise of? applications are composed of different ' components ' which in turn are supported by their corresponding ' container ' supported 'component' types are: enterprise applications - make use of the specifications like ejb, jms, jpa etc and are executed within an ejb container web applications - they leverage the servlet api, jsp, jsf etc and are supported by a web container application client - executed in client side. they need an application client container which has a set of supported libraries and executes in a java se environment. applets - these are gui applications which execute in a web browser. how are java ee applications structured? as far as java ee 'application' architecture is concerned, they generally tend follow the n-tier model consisting of client tier, server tier and of course the database (back end) tier client tier - consists of web browsers or gui (swing, java fx) based clients. web browsers tend to talk to the 'web components' on the server tier while the gui clients interact directly with the 'business' layer within the server tier server tier - this tier comprises of the dynamic web components (jsp, jsf, servlets) and the business layer driven by ejbs, jms, jpa, jta specifications. database tier - contains 'enterprise information systems' backed by databases or even legacy data repositories. generic 3-tier java ee application architecture java ee - bare bones, basics.... as quickly and briefly as i possibly could. that's all for now! :-) stay tuned for more java ee content, specifically around the latest and greatest version of the java ee platform --> java ee 7 happy reading!

Michael recently wrote a blog post showing how to create a time tree representing time down to the second using Neo4j’s Cypher query language, something I built on top of for a side project I’m working on. The domain I want to model is RSVPs to meetup invites – I want to understand how much in advance people respond and how likely they are to drop out at a later stage. For this problem I only need to measure time down to the day so my task is a bit easier than Michael’s. After a bit of fiddling around with leap years, I believe the following query will create a time tree representing all the days from 2011 – 2014, which covers the time the London Neo4j meetup has been running: WITH range(2011, 2014) AS years, range(1,12) as months FOREACH(year IN years | MERGE (y:Year {year: year}) FOREACH(month IN months | CREATE (m:Month {month: month}) MERGE (y)-[:HAS_MONTH]->(m) FOREACH(day IN (CASE WHEN month IN [1,3,5,7,8,10,12] THEN range(1,31) WHEN month = 2 THEN CASE WHEN year % 4 <> 0 THEN range(1,28) WHEN year % 100 <> 0 THEN range(1,29) WHEN year % 400 <> 0 THEN range(1,29) ELSE range(1,28) END ELSE range(1,30) END) | CREATE (d:Day {day: day}) MERGE (m)-[:HAS_DAY]->(d)))) The next step is to link adjacent days together so that we can easily traverse between adjacent days without needing to go back up and down the tree. For example we should have something like this: (jan31)-[:NEXT]->(feb1)-[:NEXT]->(feb2) We can build this by first collecting all the ‘day’ nodes in date order like so: MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day) WITH year,month,day ORDER BY year.year, month.month, day.day WITH collect(day) as days RETURN days And then iterating over adjacent nodes to create the ‘NEXT’ relationship: MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day) WITH year,month,day ORDER BY year.year, month.month, day.day WITH collect(day) as days FOREACH(i in RANGE(0, length(days)-2) | FOREACH(day1 in [days[i]] | FOREACH(day2 in [days[i+1]] | CREATE UNIQUE (day1)-[:NEXT]->(day2)))) Now if we want to find the previous 5 days from the 1st February 2014 we could write the following query: MATCH (y:Year {year: 2014})-[:HAS_MONTH]->(m:Month {month: 2})-[:HAS_DAY]->(:Day {day: 1})<-[:NEXT*0..5]-(day) RETURN y,m,day If we want to we can create the time tree and then connect the day nodes all in one query by using ‘WITH *’ like so: WITH range(2011, 2014) AS years, range(1,12) as months FOREACH(year IN years | MERGE (y:Year {year: year}) FOREACH(month IN months | CREATE (m:Month {month: month}) MERGE (y)-[:HAS_MONTH]->(m) FOREACH(day IN (CASE WHEN month IN [1,3,5,7,8,10,12] THEN range(1,31) WHEN month = 2 THEN CASE WHEN year % 4 <> 0 THEN range(1,28) WHEN year % 100 <> 0 THEN range(1,29) WHEN year % 400 <> 0 THEN range(1,29) ELSE range(1,28) END ELSE range(1,30) END) | CREATE (d:Day {day: day}) MERGE (m)-[:HAS_DAY]->(d)))) WITH * MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day) WITH year,month,day ORDER BY year.year, month.month, day.day WITH collect(day) as days FOREACH(i in RANGE(0, length(days)-2) | FOREACH(day1 in [days[i]] | FOREACH(day2 in [days[i+1]] | CREATE UNIQUE (day1)-[:NEXT]->(day2)))) Now I need to connect the RSVP events to the tree!

I was playing around with Neo4j’s Cypher last weekend and found myself accidentally running some queries against an earlier version of the Neo4j 2.0 series (2.0.0). My first query started with a map and I wanted to create a person from an identifier inside the map: WITH {person: {id: 1} AS params MERGE (p:Person {id: params.person.id}) RETURN p When I ran the query I got this error: ==> SyntaxException: Type mismatch: expected Map but was Boolean, Number, String or Collection (line 1, column 62) ==> "WITH {person: {id: 1} AS params MERGE (p:Person {id: params.person.id}) RETURN p" If we try the same query in 2.0.1 it works as we’d expect: ==> +---------------+ ==> | p | ==> +---------------+ ==> | Node[1]{id:} | ==> +---------------+ ==> 1 row ==> Nodes created: 1 ==> Properties set: 1 ==> Labels added: 1 ==> 47 ms My next query was the following which links topics of interest to a person: WITH {topics: [{name: "Java"}, {name: "Neo4j"}]} AS params MERGE (p:Person {id: 2}) FOREACH(t IN params.topics | MERGE (topic:Topic {name: t.name}) MERGE (p)-[:INTERESTED_IN]->(topic) ) RETURN p In 2.0.0 that query fails like so: ==> InternalException: Query not prepared correctly! but if we try it in 2.0.1 we’ll see that it works as well: ==> +---------------+ ==> | p | ==> +---------------+ ==> | Node[4]{id:2} | ==> +---------------+ ==> 1 row ==> Nodes created: 1 ==> Relationships created: 2 ==> Properties set: 1 ==> Labels added: 1 ==> 53 ms So if you’re seeing either of those errors, then get yourself upgraded to 2.0.1 as well!

Recently needed to update the last inserted row of a table but didn't have anyway in knowing what the highest ID in the table was. I can easily do this by using the max() function to select the highest ID in the table. SELECT MAX(id) FROM table; Then I can use the result of this query in the UPDATE query to edit the record with the highest ID. But this is quite a easy query so I should be able to do this in one query by using a nested select query on the UPDATE. UPDATE table SET name='test_name' WHERE id = (SELECT max(id) FROM table) But the problem with this is that the MAX() function doesn't work inside a nested select so had to find another way of doing this. I found out that you can use an ORDER BY and a LIMIT in an UPDATE query therefore I can use a combination of these in the UPDATE query to make sure I only update the record with the highest ID, by doing a descendant order on the ID and limiting the return to only 1 record. UPDATE table SET name='test_name' ORDER BY id DESC LIMIT 1;

There are many instances where we will have to write custom queries with hibernate. I always hated writing custom queries due to following reasons Hibernate returns List of Object arrays (List getSalaryByDepartment() { Session session = HibernateUtil.getSessionFactory().openSession(); try { List salaryByDepartments = new ArrayList(); for (Object object : contactList) { Object[] result = (Object[]) object; SalaryByDepartment salaryByDepartment = new SalaryByDepartment(); salaryByDepartment.setDeptId((Integer) result[0]); salaryByDepartment.setDepartmentName((String) result[1]); salaryByDepartment.setSalary((Double) result[2]); salaryByDepartments.add(salaryByDepartment); } return salaryByDepartments; } catch (HibernateException e) { e.printStackTrace(); return null; } finally { session.close(); } } There is a better mechanism for writing the custom queries called 'select new' but it is not widely used for some reason. It might be an overkill to write all custom queries using this mechanism, but is good for queries which return more than 2 columns. public List getNewSalaryByDepartment() { Session session = HibernateUtil.getSessionFactory().openSession(); try { List salaryByDept = session.createQuery("select " + "new hsqldb.results.SalaryByDepartment(department.id, department.departmentName, sum(employee.salary)) " + "from Employee employee, Department department " + "where employee.department.id = department.id group by department.id") .list(); return salaryByDept; } catch (HibernateException e) { e.printStackTrace(); return null; } finally { session.close(); } }

After reading the interesting blog post of my colleague Rik van Bruggen on “Media, Politics and Graphs” I thought it would be really cool to render it as a GrapGist. Especially, as he already shared all the queries as a GitHub Gist. Unfortunately the dataset was a bit large for a sensible GraphGist representation, so I thought about means of extracting a smaller sample of his raw data that he made available (see his blog post for the link). Considering my last blog post on creating data from sampling a cross product, this should be much easier. We know we want to have all nodes with the labels PARTY, SHOW and GENDER in our graph as well as a sample of GUEST nodes with their relationships. The first part is easy: MATCH (n) WHERE n:PARTY OR n:SHOW OR n:GENDER RETURN n; The second part uses something that was not helpful in my last exploration, namely that random sampling when applied directly to a match, is used to filter the first node-pattern in the match and then still traverse all relationships/paths emanating from that node. MATCH(n:GUEST)-[r]->() WHERE rand() < 0.1 RETURN n,r; The number you compare rand() to is the percentage you want to get back, in this example 10%. Now I have two nice queries, that can get me the data, how can I bring them together? With UNION ALL MATCH (n) WHERE n:PARTY OR n:SHOW OR n:GENDER RETURN n, null as r UNION ALL MATCH(n:GUEST)-[r]->() WHERE rand() < 0.1 RETURN n,r; And where do I get the Cypher statements from, that I can use to populate my GraphGist database setup? Fortunately my dump command made it into the Neo4j-Shell, so that we can just run it on the command-line and redirect the output into a file: bin/neo4j-shell -path talkshow/graph.db \ -c 'dump MATCH (n) WHERE n:PARTY OR n:SHOW OR n:GENDER RETURN n, null as r UNION ALL MATCH(n:GUEST)-[r]->() WHERE rand() < 0.1 RETURN n,r;' \ > talkshow/sample.cql Don’t forget the semicolon at the end! Looking at sample.cql we see something like: begin create (_0:`SHOW` {`Modularity Name`:"B&vD", `id`:"B&vD", `label`:"B&vD", `modularity_class`:3, `weighted outdegree`:0.000000}) create (_1:`SHOW` {`Modularity Name`:"P&W", `id`:"P&W", `label`:"P&W", `modularity_class`:4, `weighted outdegree`:0.000000}) create (_2:`SHOW` {`Modularity Name`:"DWDD", `id`:"DWDD", `label`:"DWDD", `modularity_class`:5, `weighted outdegree`:0.000000}) ... ... create _509-[:`VISITED` {`quantity`:1}]->_5 create _509-[:`VISITED` {`quantity`:1}]->_2 create _509-[:`VISITED` {`quantity`:1}]->_1 create _509-[:`VISITED` {`quantity`:1}]->_0 ; commit Which we can now use to populate our database for our GraphGist, and here it is in all its beauty – GraphGist: “Media, Politics and Graphs”. But actually I chose not to use Rik’s GitHub Gist with the queries, but to copy the nice text and pictures from his blog post into the GraphGist. You might notice that some of the parties go without connections. That would need some tweaking of the sampling which I leave as exercise for you. Have fun Michael

What's New in this Release? Aspose development team is happy to announce the monthly release of Aspose.Words for Java &.NET 14.3.0. Aspose.Words now supports insertion of OLE objects such as another Microsoft Word document or a Microsoft Excel chart. A new public method, InsertOleObject, has been introduced in the DocumentBuilder class. This method can be used to insert an embedded or linked OLE object from a file into a Word document. Aspose.Words’ rendering engine now partially supports EUDC (End-User-Defined-Characters) fonts. Please find below the description of how EUDC fonts works on Windows. In this first implementation, Aspose.Words uses a single EUDC font. When rendering a document to fixed-page formats, this font is searched among the specified font sources by “EUDC” family name. Starting from Aspose.Words 14.3.0, Best Fit position of data labels in pie charts is partially supported. In previous versions labels with best fit position were rendered as if they had the inside end position. Currently we use a modified Open Office algorithm to set the best fit position of data labels. The list of new and improved features in this release are listed below Public API for insertion of OLE objects both linked and embedded Outline, Shadow, Reflection, Glow and Fill text effects for rendering text inside DrawingML shapes EUDC fonts rendering partially supported “PDF Logical Structure” export reworked, significantly improving memory usage Support OLE embedding of documents and files Feature Write an article about how to work with Table of contents in Aspose.Words Add tag support Support BestFit position of Pie chart's data labels. Support style:text-decoration attribute of Paragraph tag. Preserve private characters in EUDC.TTE during rendering to Pdf Support HTML table row borders in HTML import Support text outline effect. Support text fill effect. Support text shadow effect. Support text reflection effect Support text glow effect. Support text effects applied to text in Dml Shape or in SmartArt. Implement reading text effects from rPr in DML. Support inheriting styles from parent objects. Add image compression options for different image types Add a link to the online documentation in the DLLs only release Default run properties lose lang attribute value on DOCX to DOC conversions Padding for image in table is lost when rendering to Pdf Consider using "Don't vertically align cells containing floating objects" compatibility option when exporting to HTML Shape in table's cell is improperly horizontally aligned to center after export to HTML is now fixed Floating shape is now properly vertically positioned after export to HTML Warning : Unkno/wn ProgId value 'Visio.Drawing.11'. This might cause inaccessible OLE embedding Charts (DrawingML) issue fixed and now render correctly in Pdf file Aspose.Words now properly work in Jdeveloper IDE OLE object cannot be edited after re saving the document Diagram connectors are inverted/flipped after conversion from Docx to pdf WordArt letters are condensing issue is resolved Condensed character spacing is lost is fixed Relative hyperlink with Unicode is now properly saved to Pdf Add pre-built Document Explorer JAR to Java release. Aspose.Words does not take in account style set in . Table now looks correctly while converting html to doc. Hyperlinks split into multiple fragments/links in output PDF Horizontal table position is corrected. Document.UpdateFields Does not Update TOC in DOCX Content in the output html is overlapped at many places in Html Different table justify alignment for different compatibilityMode values. CSS selectors now work for , , and elements Chart Legend/Series now render correctly in output Pdf file Data labels with best fit position are rendering at correct places in Chart Offset of the hyperlink text line is off by a few pixels in output Pdf Hyperlinks split into multiple fragments/links in output PDF is fixed Document.UpdateFields is enhanced and update SUM formula field Paragraph's first line indent increases is fixed when exported to HTM Space before a paragraph following a floating element is too large after HTML to DOCX conversion is now fixd Block-level SVG image become inline after HTML to DOCX conversion Problem with vertical paragraph spacing is resolved when importing HTML using InsertHtml method Shape rotation is fixed after conversion from Doc to Pdf Text color is changed after conversion from Docx to WordML/Doc Word Table indentation is now corrected when is placed inside Arrows on the Lines gets distorted when converting to Pdf is now fixed A space character is exported to PDF output between Japanese and Numeric characters is fixed Line Shape from Header is merged with the top border of Table in Body Docx to WordML conversion issue resolved with content control Junk text is rendered in fixed page formats Positions of some DrawingML circles are now preserved during rendering A row and some content is rendering at the bottom of previous page is fixed Header table rows and images are now preserved in PDF List items issue fixed, now line up correctly after conversion from RTF to HTML Conversion from Docm to Doc creates corrupted document now fixed Hyperlink for an icon is now preserved during HTML to PDF conversion Text overlapping is fixed after conversion from Docx to Pdf Text position change is fixed after conversion from Docx to Pdf Other most recent bug fixes are also included in this release Newly added documentation pages and articles Some new tips and articles have now been added into Aspose.Words for .NET documentation that may guide you briefly how to use Aspose.Words for performing different tasks like the followings. How Aspose.Words Uses True Type Fonts How to Extract Images from a Document Overview: Aspose.Words Aspose.Words is a word processing component that enables .NET, Java & Android applications to read, write and modify Word documents without using Microsoft Word. Other useful features include document creation, content and formatting manipulation, mail merge abilities, reporting features, TOC updated/rebuilt, Embedded OOXML, Footnotes rendering and support of DOCX, DOC, WordprocessingML, HTML, XHTML, TXT and PDF formats (requires Aspose.Pdf). It supports both 32-bit and 64-bit operating systems. You can even use Aspose.Words for .NET to build applications with Mono. More about Aspose.Words Homepage Aspose.Words for .NET Homepage Java Word Library Download Aspose.Words for .NET Download Aspose.Words for Java Demos for Aspose.Words Contact Information Aspose Pty Ltd Suite 163, 79 Longueville Road Lane Cove, NSW, 2066 Australia Aspose - Your File Format Experts [email protected] Phone: 888.277.6734 Fax: 866.810.9465