This article will provide you with a tutorial allowing you to determine how much and where Java heap space is retained from your active application Java threads. A true case study from an Oracle Weblogic 10.0 production environment will be presented in order for you to better understand the analysis process. We will also attempt to demonstrate that excessive garbage collection or Java heap space memory footprint problems are often not caused by true memory leaks but instead due to thread execution patterns and high amount of short lived objects. Background As you may have seen from my past JVM overview article, Java threads are part of the JVM fundamentals. Your Java heap space memory footprint is driven not only by static and long lived objects but also by short lived objects. OutOfMemoryError problems are often wrongly assumed to be due to memory leaks. We often overlook faulty thread execution patterns and short lived objects they “retain” on the Java heap until their executions are completed. In this problematic scenario: Your “expected” application short lived / stateless objects (XML, JSON data payload etc.) become retained by the threads for too long (thread lock contention, huge data payload, slow response time from remote system etc.) Eventually such short lived objects get promoted to the long lived object space e.g. OldGen/tenured space by the garbage collector As a side effect, this is causing the OldGen space to fill up rapidly, increasing the Full GC (major collections) frequency Depending of the severity of the situation this can lead to excessive GC garbage collection, increased JVM paused time and ultimately OutOfMemoryError: Java heap space Your application is now down, you are now puzzled on what is going on Finally, you are thinking to either increase the Java heap or look for memory leaks…are you really on the right track? In the above scenario, you need to look at the thread execution patterns and determine how much memory each of them retain at a given time. OK I get the picture but what about the thread stack size? It is very important to avoid any confusion between thread stack size and Java memory retention. The thread stack size is a special memory space used by the JVM to store each method call. When a thread calls method A, it “pushes” the call onto the stack. If method A calls method B, it gets also pushed onto the stack. Once the method execution completes, the call is “popped” off the stack. The Java objects created as a result of such thread method calls are allocated on the Java heap space. Increasing the thread stack size will definitely not have any effect. Tuning of the thread stack size is normally required when dealing with java.lang.stackoverflowerror or OutOfMemoryError: unable to create new native thread problems. Case study and problem context The following analysis is based on a true production problem we investigated recently. Severe performance degradation was observed from a Weblogic 10.0 production environment following some changes to the user web interface (using Google Web Toolkit and JSON as data payload) Initial analysis did reveal several occurrences of OutOfMemoryError: Java heap space errors along with excessive garbage collection. Java heap dump files were generated automatically (-XX:+HeapDumpOnOutOfMemoryError) following OOM events Analysis of the verbose:gc logs did confirm full depletion of the 32-bit HotSpot JVM OldGen space (1 GB capacity) Thread dump snapshots were also generated before and during the problem The only problem mitigation available at that time was to restart the affected Weblogic server when problem was observed A rollback of the changes was eventually performed which did resolve the situation The team first suspected a memory leak problem from the new code introduced. Thread dump analysis: looking for suspects… The first analysis step we did was to perform an analysis of the generated thread dump data. Thread dump will often show you the culprit threads allocating memory on the Java heap. It will also reveal any hogging or stuck thread attempting to send and receive data payload from a remote system. The first pattern we noticed was a good correlation between OOM events and STUCK threads observed from the Weblogic managed servers (JVM processes). Find below the primary thread pattern found: <10-Dec-2012 1:27:59 o'clock PM EST> <[STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "672" seconds working on the request which is more than the configured time of "600" seconds. As you can see, the above thread appears to be STUCK or taking very long time to read and receive the JSON response from the remote server. Once we found that pattern, the next step was to correlate this finding with the JVM heap dump analysis and determine how much memory these stuck threads were taking from the Java heap. Heap dump analysis: retained objects exposed! The Java heap dump analysis was performed using MAT. We will now list the different analysis steps which did allow us to pinpoint the retained memory size and source. 1. Load the HotSpot JVM heap dump 2. Select the HISTOGRAM view and filter by “ExecuteThread” * ExecuteThread is the Java class used by the Weblogic kernel for thread creation & execution * As you can see, this view was quite revealing. We can see a total of 210 Weblogic threads created. The total retained memory footprint from these threads is 806 MB. This is pretty significant for a 32-bit JVM process with 1 GB OldGen space. This view alone is telling us that the core of the problem and memory retention originates from the threads themselves. 3. Deep dive into the thread memory footprint analysis The next step was to deep dive into the thread memory retention. To do this, simply right click over the ExecuteThread class and select: List objects > with outgoing references. As you can see, we were able to correlate STUCK threads from the thread dump analysis with high memory retention from the heap dump analysis. The finding was quite surprising. 3. Thread Java Local variables identification The final analysis step did require us to expand a few thread samples and understand the primary source of memory retention. As you can see, this last analysis step did reveal huge JSON response data payload at the root cause. That pattern was also exposed earlier via the thread dump analysis where we found a few threads taking very long time to read & receive the JSON response; a clear symptom of huge data payload footprint. It is crucial to note that short lived objects created via local method variables will show up in the heap dump analysis. However, some of those will only be visible from their parent threads since they are not referenced by other objects, like in this case. You will also need to analyze the thread stack trace in order to identify the true caller, followed by a code review to confirm the root cause. Following this finding, our delivery team was able to determine that the recent JSON faulty code changes were generating, under some scenarios, huge JSON data payload up to 45 MB+. Given the fact that this environment is using a 32-bit JVM with only 1 GB of OldGen space, you can understand that only a few threads were enough to trigger severe performance degradation. This case study is clearly showing the importance of proper capacity planning and Java heap analysis, including the memory retained from your active application & Java EE container threads. Learning is experience. Everything else is just information I hope this article has helped you understand how you can pinpoint the Java heap memory footprint retained by your active threads by combining thread dump and heap dump analysis. Now, this article will remain just words if you don’t experiment so I highly recommend that you take some time to learn this analysis process yourself for your application(s). Please feel free to post any comment or question.

in this article i deal with a different nosql database called mongodb a mature nosql engine born outside the .net world to clarify the concept of id in a typical no sql database. installation of mongo is really simple, just download, uncompress, locate the bin folder, and type this from an administrator console prompt to install mongo as service 1: mongod --install --logpath c:xxxx --dbpath c:yyyy you can find plenty of installation guide on the internet, but with the above install command you create a windows service that will automatically start mongodb on your machine using specified datafolder. now you should download the c# driver to connect from .net code, but if you like using linq you can install fluent mongo directly with nuget. figure 1: install fluent mongo with nuget. fluent mongo is a library that gave little linq capability over standard drivers, but adding a nuget reference to fluentmongo you get automatically a reference to the official drivers. now you are ready to insert your first record in mongodb with the above code. mongoserver server = mongoserver.create(); mongodatabase databasetest = server.getdatabase("test"); var untyped = databasetest.getcollection("untyped"); untyped.save(new bsondocument { { "name", "untyped1" } }); bsondocument seconddocument = bsondocument.parse("{name: 'untyped2', blabla: 'bla bla value'}"); untyped.save(seconddocument); in line1 i create a connection to mongodb server passing no parameter to connect to local mongodb server, then i obtain a reference to a mongodatabase object called “test” with mongoserver::getdatabase() method and finally i get a reference to a collection named “untyped” with the mongngodatabase::getcollection () method. this is quite similar to a sql server or other sql database, you have a server, the server contains several databases, and each database is composed by tables; in the same way mongo is divided into server/database/collection where a collection contains document. mongodb stores data in json format and to insert data inside a collection you can simply create a bsondocument, an object defined by c# driver assembly that is capable to represent a document composed by a series of key-value pair. to initialize a bsondocument you can pass a icollection (line 4) or if you feel more confortable with string json representation, you can user bsondocument.parse() to specify the document directly with a json string. after you inserted the above documents you can use mongovue to see what is contained inside the database. figure 2: use mongovue to see what is inside the database the interesting aspect is that each document has an unique id, even if i did not specified any special property in the code. this is a standard behavior for nosql databases, if you did not specify any id property the database engine will create unique id on his own to idendify the docuemnt. the id is a key factor for mongo and other nosql storage, if you try to store a document directly inside the collection specifying json content you will get an error. untyped.save("{name: 'json', attribute:'attribute content'}"); the mongocollection object contains a save object that accepts a string, but the above call will fail with the error subclass must implement getdocumentid . previous code works because one of the specific functionality that a bsondocument implements is the ability to manage id generation, but plain json does not have this capability. if you need to know the id generated by the database, you can query the bsondocument for its unique id after it was saved in a mongo collection . (remember that the id is not available until you save the document). bsondocument seconddocument = bsondocument.parse("{name: 'untyped2', blabla: 'bla bla value'}"); object id; type idtype; iidgenerator generator; untyped.save(seconddocument); seconddocument.getdocumentid(out id, out idtype, out generator); basically you are asking to your bsondocument to return you the generated id, as well as the type of the id and the generator that mongo used to generate that specific id. the result is represented into this snippet figure 3: the three object that you got with a call to getdocumentid: id, idtype and the generator. as you can see the id is an instance of type mongodb.bson.objectid, based on bsonvalue base class and the generator is an instance of objectidgenerator. this type of id is specific to mongo, and the documentation states that an objectid is a bson objectid is a 12-byte value consisting of a 4-byte timestamp (seconds since epoch), a 3-byte machine id, a 2-byte process id, and a 3-byte counter. note that the timestamp and counter fields must be stored big endian unlike the rest of bson. this is because they are compared byte-by-byte and we want to ensure a mostly increasing order. if you want to have a generator that creates integer id, like identity column in sql server , you will find that it is simply not available out of the box, because an int value is not guarantee to be unique if you use sharding. sharding is a technique that permits to partition data into different physical instances, so each instance should generates ids that are unique across all instances and this prevents the use of a simple int32 id. clearly in .net world a guid is guarantee to be unique and is more .net oriented, so mongo db has a guid id generator, that can be specified with the above snippet of code.. bsondocument thirddocument = bsondocument.parse("{name: 'untyped3', anotherproperty: 'xxxxxxxxxxxxxxxxxxxxxxx'}"); var id2 = mongodb.bson.serialization.idgenerators.guidgenerator.instance.generateid(untyped, thirddocument); thirddocument.setdocumentid(id2); the key is using the guidgenerator (in the mongodb.bson.serialization.idgenerators namespace) to generate a valid mongoid guid value, then call the setdocumentid method of bsondocument to manually set the id and not relay on automatic id generation. if you look at the db you will find that the document with guid id has really a different id type. figure 4: the document with guid id is represented in a different way in mongovue, but as you can verify there is no problem in having documents with different id types in the same collection. this demonstrates that a no sql database has a concept of document id that is similar to the concept of id of a standard sql server, you can use a native id generation of the engine that generates a valid id during insertion or you can assign your own id to the document, but basically the whole concept of id is more engine-related and has no business meaning, so i strongly discourage to use anything that has a business meaning as id of a document. noone prevents you to insert in the document a property called “myid” or something else that has a business meaning and can be used as logical id and let the engine handle the internal id by itself.

Take a look at the date and calendar extensions in Groovy JDK.

Overview Devoxx, the biggest vendor-independent Java conference in the world, took place in Atwerp, Belgium on 12 - 16 November. This year it was bigger yet, reaching 3400 attendees from 40 different countries. As last year, I and a small group of colleagues from SAP were there and enjoyed it a lot. After the impressive dance of Nao robots and the opening keynotes, more than 200 conference sessions explored a variety of different technology areas, ranging from Java SE to methodology and robotics. One of the most interesting topics for me was the evolution of the Java language and platform in JDK 8. My interest was driven partly by the fact that I was already starting work on Wordcounter, and finishing work on another concurrent Java library named Evictor, about which I will be blogging in a future post. In this blog series, I would like to share somewhat more detailed summaries of the sessions on this topic which I attended. These three sessions all took place in the same day, in the same room, one after the other, and together provided three different perspectives on lambdas, parallel collections, and parallelism in general in Java 8. On the road to JDK 8: Lambda, parallel libraries, and more by Joe Darcy Closures and Collections - the World After Eight by Maurice Naftalin Fork / Join, lambda & parallel() : parallel computing made (too ?) easy by Jose Paumard In this post, I will cover the first session, with the other two coming soon. On the road to JDK 8: Lambda, parallel libraries, and more In the first session, Joe Darcy, a lead engineer of several projects at Oracle, introduced the key changes to the language coming in JDK 8, such as lambda expressions and default methods, summarized the implementation approach, and examined the parallel libraries and their new programming model. The slides from this session are available here. Evolving the Java platform Joe started by talking a bit about the context and concerns related to evolving the language. The general evolution policy for OpenJDK is: Don't break binary compatibility Avoid introducing source incompatibilities. Manage behavioral compatibility changes The above list also extends to the language evolution. These rules mean that old classfiles will be always recognized, the cases when currently legal code stops compiling are limited, and changes in the generated code that introduce behavioral changes are also avoided. The goals of this policy are to keep existing binaries linking and running, and to keep existing sources compiling. This has also influenced the sets of features chosen to be implemented in the language itself, as well as how they were implemented. Such concerns were also in effect when adding closures to Java. Interfaces, for example, are a double-edged sword. With the language features that we have today, they cannot evolve compatibly over time. However, in reality APIs age, as people's expectations how to use them evolve. Adding closures to the language results in a really different programming model, which implies it would be really helpful if interfaces could be evolved compatibly. This resulted in a change affecting both the language and the VM, known as default methods. Project Lambda Project Lambda introduces a coordinated language, library, and VM change. In the language, there are lambda expressions and default methods. In the libraries, there are bulk operations on collections and additional support for parallelism. In the VM, besides the default methods, there are also enhancements to the invokedynamic functionality. This is the biggest change to the language ever done, bigger than other significant changes such as generics. What is a lambda expression? A lambda expression is an anonymous method having an argument list, a return type, and a body, and able to refer to values from the enclosing scope: (Object o) -> o.toString() (Person p) -> p.getName().equals(name) Besides lambda expressions, there is also the method reference syntax: Object::toString() The main benefit of lambdas is that it allows the programmer to treat code as data, store it in variables and pass it to methods. Some history When Java was first introduced in 1995 not many languages had closures, but they are present in pretty much every major language today, even C++. For Java, it has been a long and winding road to get support for closures, until Project Lambda finally started in Dec 2009. The current status is that JSR 335 is in early draft review, there are binary builds available, and it's expected to become very soon part of the mainline JDK 8 builds. Internal and external iteration There are two ways to do iteration - internal and external. In external iteration you bring the data to the code, whereas in internal iteration you bring the code to the data. External iteration is what we have today, for example: for (Shape s : shapes) { if (s.getColor() == RED) s.setColor(BLUE); } There are several limitations with this approach. One of them is that the above loop is inherently sequential, even though there is no fundamental reason it couldn't be executed by multiple threads. Re-written to use internal iteration with lambda, the above code would be: shapes.forEach(s -> { if (s.getColor() == RED) s.setColor(BLUE); }) This is not just a syntactic change, since now the library is in control of how the iteration happens. Written in this way, the code expresses much more what and less how, the how being left to the library. The library authors are free to use parallelism, out-of-order execution, laziness, and all kinds of other techniques. This allows the library to abstract over behavior, which is a fundamentally more powerful way of doing things. Functional Interfaces Project Lambda avoided adding new types, instead reusing existing coding practices. Java programmers are familiar with and have long used interfaces with one method, such as Runnable, Comparator, or ActionListener. Such interfaces are now called functional interfaces. There will be also new functional interfaces, such as Predicate and Block. A lambda expression evaluates to an instance of a functional interface, for example: Predicate isEmpty = s -> s.isEmpty(); Predicate isEmpty = String::isEmpty; Runnable r = () -> { System.out.println(“Boo!”) }; So existing libraries are forward-compatible with lambdas, which results in an "automatic upgrade", maintaining the significant investment in those libraries. Default Methods The above example used a new method on Collection, forEach. However, adding a method to an existing interface is a no-go in Java, as it would result in a runtime exception when a client calls the new method on an old class in which it is not implemented. A default method is an interface method that has an implementation, which is woven-in by the VM at link time. In a sense, this is multiple inheritance, but there's no reason to panic, since this is multiple inheritance of behavior, not state. The syntax looks like this: interface Collection { ... default void forEach(Block action) { for (T t : this) action.apply(t); } } There are certain inheritance rules to resolve conflicts between multiple supertypes: Rule 1 – prefer superclass methods to interface methods ("Class wins") Rule 2 – prefer more specific interfaces to less ("Subtype wins") Rule 3 – otherwise, act as if the method is abstract. In the case of conflicting defaults, the concrete class must provide an implementation. In summary, conflicts are resolved by looking for a unique, most specific default-providing interface. With these rules, "diamonds" are not a problem. In the worst case, when there isn't a unique most specific implementation of the method, the subclass must provide one, or there will be a compiler error. If this implementation needs to call to one of the inherited implementations, the new syntax for this is A.super.m(). The primary goal of default methods is API evolution, but they are useful as an inheritance mechanism on their own as well. One other way to benefit from them is optional methods. For example, most implementations of Iterator don't provide a useful remove(), so it can be declared "optional" as follows: interface Iterator { ... default void remove() { throw new UnsupportedOperationException(); } } Bulk operations on collections Bulk operations on collections also enable a map / reduce style of programming. For example, the above code could be further decomposed by getting a stream from the shapes collection, filtering the red elements, and then iterating only over the filtered elements: shapes.stream().filter(s -> s.getColor() == RED).forEach(s -> { s.setColor(BLUE); }); The above code corresponds even more closely to the problem statement of what you actually want to get done. There also other useful bulk operations such as map, into, or sum. The main advantages of this programming model are: More composability Clarity - each stage does one thing The library can use parallelism, out-of-order, laziness for performance, etc. The stream is the basic new abstraction being added to the platform. It encapsulates laziness as a better alternative to "lazy" collections such as LazyList. It is a facility that allows getting a sequence of elements out of it, its source being a collection, array, or a function. The basic programming model with streams is that of a pipeline, such as collection-filter-map-sum or array-map-sorted-forEach. Since streams are lazy, they only compute as elements are needed, which pays off big in cases like filter-map-findFirst. Another advantage of streams is that they allow to take advantage of fork/join parallelism, by having libraries use fork/join behind the scenes to ease programming and avoid boilerplate. Implementation technique In the last part of his talk, Joe described the advantages and disadvantages of the possible implementation techniques for lambda expressions. Different options such as inner classes and method handles were considered, but not accepted due to their shortcomings. The best solution would involve adding a level of indirection, by letting the compiler emit a declarative recipe, rather than imperative code, for creating a lambda, and then letting the runtime execute that recipe however it deems fit (and make sure it's fast). This sounded like a job for invokedynamic, a new invocation mode introduced with Java SE 7 for an entirely different reason - support for dynamic languages on the JVM. It turned out this feature is not just for dynamic languages any more, as it provides a suitable implementation mechanism for lambdas, and is also much better in terms of performance. Conclusion Project Lambda is a large, coordinated update across the Java language and platform. It enables much more powerful programming model for collections and takes advantage of new features in the VM. You can evaluate these new features by downloading the JDK8 build with lambda support. IDE support is also already available in NetBeans builds with Lambda support and IntelliJ IDEA 12 EAP builds with Lambda support. I already made my own experiences with lambdas in Java in Wordcounter. As I already wrote, I am convinced that this style of programming will quickly become pervasive in Java, so if you don't yet have experience with it, I do encourage you to try it out. Published on DZone by Stoyan Rachev (source).

In between other jobs I have recently been reviewing the WADL specification with a view to fixing some documentation problems and to producing an updated version. One of the things that was apparent was the lack of any grammar support for languages other than XML - yes you can use a mapping from JSON<->XML Schema but this would be less than pleasant for a JSON purist. So I began to look at how one would go about attaching a JSON-Schema grammar of a JSON document in a WADL description of a service. This isn't a specification yet; but a proposal of how it might work consistently. Now I work with Jersey mostly, so lets consider what Jersey will currently generate for a service that returns both XML and JSON. So the service here is implemented using the JAX-B binding so they both use a similar structure as defined by the XML-Schema reference by the include. So the first thing we considered was re-using the existing element property, which is defined as a QName, on the representation element to reference an imported JSON-Schema. It is shown here both with another an arbitrary namespace so it can be told apart from XML elements without a namespace. Or xmlns:json="http://wadl.dev.java.net/2009/02/json" The problem is that the JSON-Schema specification as it stands doesn't have a concept of a "name" property, so each JSON-Schema is uniquely identified by its URI. Also from my read of the specification, each JSON-Schema contains the definition for at most one document - not the multiple types / documents that can be contained in XML-Schema. So the next best suggestion would be to just use the "filename" part of the URI as a proxy for the URI; but of course that won't necessarily be unique. I could see for example the US government and Yahoo both publishing there own "address" micro format. The better solution to this problem is to introduce a new attribute, luckily the WADL spec was designed with this in mind, that is a type of URI that can be used to directly reference the JSON-Schema definitions. So rather than the direct import in the previous example we have a URI property on the element itself. The "describedby" attribute name comes from the JSON-Schema proposal and is consistent with the rel used on atom links in the spec. xmlns:json="http://wadl.dev.java.net/2009/02/json-schema" xmlns:m="urn:message" The secondary advantage is that this format is backward-compatible with tooling that was relying on the XML-Schema grammar. Although this is probably only of interest to people who work in tooling / testing tools like myself. Once you have the JSON-Schema definition then some users are going to want to do away with the XML all together, so finally here is a simple mapping of the WADL to a JSON document that contains just the JSON-Schema information. It has been suggested by Sergey Breyozkin that the JSON mapping would only show the json grammars and I am coming around to that way of thinking. I would be interested to hear of a usecase for the JSON mapping that would want access to the XML Schema. { "doc":{ "@generatedBy":"Jersey: 1.16-SNAPSHOT 10/26/2012 09:28 AM" }, "resources":{ "@base":"http://localhost/", "resource":{ "@path":"/root", "method":{ "@id":"hello", "@name":"PUT", "request":{ "representation":[ { "@mediaType":"application/json", "@describedby":"application.wadl/requestMessage" } ] }, "response":{ "representation":[ { "@mediaType":"application/json", "@describedby":"application.wadl/responseMessage" } ] } } } } } I am currently using the mime type of "application/vnd.sun.wadl+json" for this mapping to be consistent with the default WADL mime type. I suspect we would want to change this in the future; but it will do for starters. So this is all very interesting but you can't play with it unless you have an example implementation. I have something working for both the server side and for a Java client generator in Jersey and wadl2java respectively, and that will be the topic of my next post. I have been working with Pavel Bucek and the Jersey team on these implementations and the WADL proposal. Thanks very much to him for putting up with me.

Decorator pattern is one of the widely used structural patterns. This pattern dynamically changes the functionality of an object at runtime without impacting the existing functionality of the objects. In short this pattern adds additional functionalities to the object by wrapping it. Problem statement: Imagine a scenario where we have a pizza which is already baked with tomato and cheese. After that you just recall that you need to put some additional topping at customer’s choice. So you will need to give some additional toppings like chicken and pepper on the go. Intent: Add or remove additional functionalities or responsibilities from the object dynamically without impacting the original object. At times it is required when addition of functionalities is not possible by subclassing as it might create loads of subclasses. Solution: So in this case we are not using inheritance to add additional functionalities to the object i.e. pizza, instead we are using composition. This pattern is useful when we don’t want to use inheritance and rather use composition. Structure Decorator Design Pattern Structure Following are the participants of the Decorator Design pattern: Component – this is the wrapper which can have additional responsibilities associated with it at runtime. Concrete component- is the original object to which the additional functionalities are added. Decorator-this is an abstract class which contains a reference to the component object and also implements the component interface. Concrete decorator-they extend the decorator and builds additional functionality on top of the Component class. Example: Decorator Design Pattern Example In the above example the Pizza class acts as the Component and BasicPizza is the concrete component which needs to be decorated. The PizzaDecorator acts as a Decorator abstract class which contains a reference to the Pizza class. The ChickenTikkaPizza is the ConcreteDecorator which builds additional functionality to the Pizza class. Let’s summarize the steps to implement the decorator design pattern: Create an interface to the BasicPizza(Concrete Component) that we want to decorate. Create an abstract class PizzaDecorator that contains reference field of Pizza(decorated) interface. Note: The decorator(PizzaDecorator) must extend same decorated(Pizza) interface. We will need to now pass the Pizza object that you want to decorate in the constructor of decorator. Let us create Concrete Decorator(ChickenTikkaPizza) which should provide additional functionalities of additional topping. The Concrete Decorator(ChickenTikkaPizza) should extend the PizzaDecorator abstract class. Redirect methods of decorator (bakePizza()) to decorated class’s core implementation. Override methods(bakePizza()) where you need to change behavior e.g. addition of the Chicken Tikka topping. Let the client class create the Component type (Pizza) object by creating a Concrete Decorator(ChickenTikkaPizza) with help from Concrete Component(BasicPizza). To remember in short : New Component = Concrete Component + Concrete Decorator Pizza pizza = new ChickenTikkaPizza(new BasicPizza()); Code Example: BasicPizza.java public String bakePizza() { return "Basic Pizza"; } Pizza.java public interface Pizza { public String bakePizza(); } PizzaDecorator.java public abstract class PizzaDecorator implements Pizza { Pizza pizza; public PizzaDecorator(Pizza newPizza) { this.pizza = newPizza; } @Override public String bakePizza() { return pizza.bakePizza(); } } ChickenTikkaPizza.java public class ChickenTikkaPizza extends PizzaDecorator { public ChickenTikkaPizza(Pizza newPizza) { super(newPizza); } public String bakePizza() { return pizza.bakePizza() + " with Chicken topping added"; } } Client.java public static void main(String[] args) { Pizza pizza = new ChickenTikkaPizza(new BasicPizza()); System.out.println(pizza.bakePizza()); } Benefits: Decorator design pattern provide more flexibility than the standard inheritance. Inheritance also extends the parent class responsibility but in a static manner. However decorator allows doing this in dynamic fashion. Drawback: Code debugging might be difficult since this pattern adds functionality at runtime. Interesting points: Adapter pattern plugs different interfaces together whereas decorator pattern enhances the functionality of the object. Unlike Decorator Pattern the Strategy pattern changes the original object without wrapping it. While Proxy pattern controls access to the object the decorator pattern enhances the functionality of the object. Both Composite and Decorator pattern uses the same tree structure but there are subtle differences between both of them. We can use composite pattern when we need to keep the group of objects having similar behavior inside another object. However decorator pattern is used when we need to modify the functionality of the object at runtime. There are various live examples of decorator pattern in Java API. java.io.BufferedReader; java.io.FileReader; java.io.Reader; If we see the constructor of the BufferedReader then we can see that the BufferedReader wraps the Reader class by adding more features e.g. readLine() which is not present in the reader class. We can use the same format like the above example on how the client uses the decorator pattern new BufferedReader(new FileReader(new File(“File1.txt”))); Similarly the BufferedInputStream is a decorator for the decorated object FileInputStream. BufferedInputStream bs = new BufferedInputStream(new FileInputStream(new File(“File1.txt”))); Anyways I hope my readers have liked this article. Please do not hesitate to provide your valuable feedback and comments.

When you deploy your Java web application to the Apache Tomcat server, via Eclipse, by default the web app will be deployed under {YOUR_ECLIPSE_WORKSPACE}\.metadata\.plugins\org.eclipse.wst.server.core\tmp{a-number}\wtpwebapps. Suppose if you want to deploy your web app to a location that is easily navigable, follow these steps. First make sure you have removed all the web apps that are currently added to your server instance (In servers view, right click on the server name and then Add and Remove). And then double click on the server instance in servers view which will open up that server’s configuration page. On that page, see under Server Locations and select either the option Use Tomcat Installation to deploy the web app under the directory where the Tomcat server is installed or Use custom location to manually specify. Save, Re-add the web application and then Publish. Now the deployed web app will be under the directory of your choice.

For the sake of completeness, here is a Java version of the Groovy post to test your Oracle Database connection. package atest; import java.sql.*; /** * Run arguments sample: * jdbc:oracle:thin:@localhost:1521:XE system mypassword123 oracle.jdbc.driver.OracleDriver */ public class DbConn { public static void main(String[] args) throws Exception { String url = args[0]; String username = args[1]; String password = args[2]; String driver = args[3]; Class.forName(driver); Connection conn = DriverManager.getConnection(url, username, password); try { Statement statement = conn.createStatement(); ResultSet rs = statement.executeQuery("SELECT SYSDATE FROM DUAL"); while(rs.next()) { System.out.println(rs.getObject(1)); } } finally { conn.close(); } } }

Here is a simple Groovy script to verify Oracle database connection using JDBC. @GrabConfig(systemClassLoader=true) @Grab('com.oracle:ojdbc6:11g') url= "jdbc:oracle:thin:@localhost:1521:XE" username = "system" password = "mypassword123" driver = "oracle.jdbc.driver.OracleDriver" // Groovy Sql connection test import groovy.sql.* sql = Sql.newInstance(url, username, password, driver) try { sql.eachRow('select sysdate from dual'){ row -> println row } } finally { sql.close() } This script should let you test connection and perform any quick ad hoc queries programmatically. However, when you first run it, it would likely failed without finding the Maven dependency for JDBC driver jar. In this case, you would need to first install the Oracle JDBC jar into maven local repository. This is due to Oracle has not publish their JDBC jar into any public Maven repository. So we are left with manually steps by installing it. Here are the onetime setup steps: 1. Download Oracle JDBC jar from their site: http://www.oracle.com/technetwork/database/features/jdbc/index-091264.html. 2. Unzip the file into C:/ojdbc directory. 3. Now you can install the jar file into Maven local repository using Cygwin. bash> cd /cygdrive/c/ojdbc bash> mvn install:install-file -DgroupId=com.oracle -DartifactId=ojdbc6 -Dversion=11g -Dpackaging=jar -Dfile=ojdbc6-11g.jar That should make your script run successfully. The Groovy way of using Sql has many sugarcoated methods that you let you quickly query and see data on screens. You can see more Groovy feature by studying their API doc. Note that you would need systemClassLoader=true to make Groovy load the JDBC jar into classpath and use it properly. Oh, BTW, if you are using Oracle DB production, you will likely using a RAC configuration. The JDBC url connection string for that should look something like this: jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=MY_DB))) Update: 12/07/2012 It appears that the groovy.sql.Sql class has a static withInstance method. This let you run onetime DB work without writing try/finally block. See this example: @GrabConfig(systemClassLoader=true) @Grab('com.oracle:ojdbc6:11g') url= "jdbc:oracle:thin:@localhost:1521:XE" username = "system" password = "mypassword123" driver = "oracle.jdbc.driver.OracleDriver" import groovy.sql.* Sql.withInstance(url, username, password, driver) { sql -> sql.eachRow('select sysdate from dual'){ row -> println row } } It's much more compact. But be aware of performance if you run it multiple times, because you will open and close the a java.sql.Connection per each call! I have also collected couple other popular databases connection test examples. These should have their driver jars already in Maven central, so Groovy Grab should able to grab them just fine. // MySQL database test @GrabConfig(systemClassLoader=true) @Grab('mysql:mysql-connector-java:5.1.22') import groovy.sql.* Sql.withInstance("jdbc:mysql://localhost:3306/mysql", "root", "mypassword123", "com.mysql.jdbc.Driver"){ sql -> sql.eachRow('SELECT * FROM USER'){ row -> println row } } // H2Database @GrabConfig(systemClassLoader=true) @Grab('com.h2database:h2:1.3.170') import groovy.sql.* Sql.withInstance("jdbc:h2:~/test", "sa", "", "org.h2.Driver"){ sql -> sql.eachRow('SELECT * FROM INFORMATION_SCHEMA.TABLES'){ row -> println row } }

As indicated by some recent mailing list emails and a lot of info returned from Google, ActiveMQ’s SystemUsage and particularly the MemoryUsage functionality has left some people confused. I’ll try to explain some details around MemoryUsage that might be helpful in understanding how it works. I won’t cover StoreUsage and TempUsage as my colleauges have covered thosein some depth. There is a section of the activemq.xml configuration you can use to specify SystemUsage limits, specifically around the memory, persistent store, and temporary store that a broker can use. Here is an example with the defaults that come with ActiveMQ 5.7: MemoryUsage MemoryUsage seems to cause the most confusion, so here goes my attempt to clarify its inner workings. When a message comes in to the broker, it has to go somewhere. It first gets unmarshalled off the wire into an ActiveMQ command object of type ActiveMQMessage. At this moment, the object is obviously in memory but the broker isn’t keeping track of it. Which brings us to our first point. The MemoryUsage is really just a counter of bytes that the broker needs and uses to keep track of how much of our JVM memory is being used by messages. This gives the broker some way of monitoring and ensuring we don’t hit our limits (more on that in a bit). Otherwise we could take on messages without knowing where our limits are until the JVM runs out of heap space. So we left off with the message coming in off the wire. Once we have that, the broker will take a look at which destination (or multiple destinations) the message needs to be routed. Once it finds the destination, it will “send” it there. The destination will increment a reference count of the message (to later know whether or not the message is considered “alive”) and proceed to do something with it. For the first reference count, the memory usage is incremented. For the last reference count, the memory usage is decremented. If the destination is a queue, it will store the message into a persistent location and try to dispatch it to a consumer subscription. If it’s a Topic, it will try to dispatch it to all subscriptions. Along the way (from the initial entry into the destination to the subscription that will send the message to the consumer), the message reference count may be incremented or decremented. As long as it has a reference count greater than or equal to 1, it will be accounted for in memory. Again, the MemoryUsage is just an object that counts bytes of messages to know how much JVM memory has been used to hold messages. So now that we have a basic understanding of what the MemoryUsage is, let’s take a closer look at a couple things: MemoryUsage hierarchies (what’s this destination memory limit that I can configure on policy entries)?? Producer Flow Control Splitting memory usage between destinations and subscriptions (producers and consumers)? Main Broker Memory, Destination Memory, Subscription Memory When the broker loads up, it will create its own SystemUsage object (or use the one specified in the configuration). As we know, the SystemUsage object has a MemoryUsage, StoreUsage, and TempUsage associated with it. The memory component will be known as the broker’s Main memory. It’s a usage object that keeps track of overall (destination, subscription, etc) memory. A destination, when it’s created, will create its own SystemUsage object (which creates its own separate Memory, Store, and Temp Usage objects) but it will set its parent to the be broker’s main SystemUsage object. A destination can have its memory limits tuned individually (but not Store and Temp, those will still delegate to the parent). To set a destination’s memory limit: So the destination usage objects can be used to more finely control MemoryUsage, but it will always coordinate with the Main memory for all usage counts. This functionality can be used to limit the number of messages that a destination keeps around so that a single destination cannot starve other destinations. For queues, it also affects the store cursor’s high water mark. A queue has different cursors for persistent and non-persistent messages. If we hit the high water mark (a threshold of the destination’s memory limit), no more messages be cached ready to be dispatched, and non-persistent messages can be purged to temp disk as necessary (if the StoreCursor will use FilePendingMessageCursor… otherwise it will just use a VMPendingMessageCursor and won’t purge to temporary store). If you don’t specify a memory limit for individual destinations, the destination’s SystemUsage will delegate to the parent (Main SystemUsage) for all usage counts. This means it will effectively use the broker’s Main SystemUsage for all memory-related counts. Consumer subscriptions, on the other hand, don’t have any notion of their own SystemUsage or MemoryUsage counters. They will always use the broker’s Main SystemUsage objects. The main thing to note about this is when using a FilePendingMessageCursor for subscriptions (for example, for a Topic subscription), the messages will not be swapped to disk until the cursor high-water mark (70% by default) is reached.. but that means 70% of Main memory will need to be reached. That could be a while, and a lot of messages could be kept in memory. And if your subscription is the one holding most of those messages, swapping to disk could take a while. As topics dispatch messages to one subscription at a time, if one subscription grinds to a halt because it’s swapping its messages to disk, the rest of the subscription ready to receive the message will also feel the slow down. You can set the cursor high water mark for subscriptions of a topic to be lower than the default: For those interested… When a message comes in the the destination, a MemoryUsage object is set on the message so that when Message.incrementReferenceCount() can increment the memory usage (on first referenced). So that means it’s accounted for by the destination’s Memory usage (and also the Main memory since the destination’s memory also informs its parent when its usage changes) and continues to do so. The only time this will change is if the message gets swapped to disk. When it gets swapped, its reference counts will be decremented, its memory usage will be decremented, and it will lose its MemoryUsage object once it gets to disk. So when it comes back to life, which MemoryUsage object will get associated with it, and where will it be counted? If it was swapped to a queue’s store, when it reconstitutes, it will be again associated with the destination memory usage. If it was swapped to a temp store in a subscription (like in a FilePendingMessageCursor), when it reconstitutes, it will NOT be associated with the destination’s memory usage anymore. It will be associated with the subscription’s memory usage (which is main memory). Producer Flow Control The big win for keeping track of memory used by messages is for Producer Flow Control (PFC). PFC is enabled by default and basically slows down the producers when usage limits are reached. This keeps the broker from exceeding its limits and running out of resources. For producers sending synchronously or for async sends with a producer window specified, if system usages are reached the broker will block that individual producer, but it will not block the connection. It will instead put the message away temporarily to wait for space to become available. It will only send back a ProducerAck once the message has been stored. Until then, the client is expected to block its send operation (which won’t block the connection itself). The ActiveMQ 5.x client libraries handle this for you. However, if an async send is sent without a producer window, or if a producer doesn’t behave properly and ignores ProducerAcks, PFC will actually block the entire connection when memory is reached. This could result in deadlock if you have consumers sharing the same connection. If producer flow control is turned off, then you have to be a little more careful about how you set up your system usages. When producer flow control is off, it basically means “broker, you have to accept every message that comes in, no matter if the consumers cannot keep up”. This can be used to handle spikes for incoming messages to a destination. If you’ve ever seen memory usages in your logs severely exceed the limits you’ve set, you probably had PFC turned off and that is expected behavior. Splitting Broker’s Main Memory So… I said earlier that a destination’s memory uses the broker’s main memory as a parent, and that subscriptions don’t have their own memory counters, they just use the broker’s main memory. Well this is true in the default case, but if you find a reason, you can further tune how memory is divided and limited. The idea here is you can partition the broker’s main memory into “Producer” and “Consumer” parts. The Producer part will be used for all things related to messages coming in to the broker, therefore it will be used in destinations. So this means when a destination creates its own MemoryUsage, it will use the Producer memory as its parent, and the Producer memory will use a portion of the broker’s main memory. On the other hand, the Consumer part will be used for all things related to dispatching messages to consumers. This means subscriptions. Instead of a subscription using the broker’s main memory directly, it will use the Consumer memory which will be a portion of the main memory. Ideally, the Consumer portion and the Producer portion will equal the entire broker’s main memory. To split the memory between producer and consumer, set the splitSystemUsageForProducersConsumers property on the main element: By default this will split the broker’s Main memory usage into 60% for the producers and 40% for the consumers. To tune this even further, set the producerSystemUsagePortion and consumerSystemUsagePortion on the main broker element: There you have it. Hopefully this sheds some light into the MemoryUsage of the broker. The topic is huge, and the tuning options are plenty, so if you have specific questions please ask in the activemq mailing list or leave a comment below.

YAML is well-known format within Ruby community, quite widely used for a long time now. But we as Java developers mostly deal with property files and XMLs in case we need some configuration for our apps. How many times we needed to express complicated configuration by inventing our own XML schema or imposing property names convention? Though JSON is becoming a popular format for web applications, using JSON files to describe the configuration is a bit cumbersome and, in my opinion, is not as expressive as YAML. Let's see what YAML can do for us to make our life easier. For sure, let's start with the problem. In order for our application to function properly, we need to feed it following data somehow: version and release date database connection parameters list of supported protocols list of users with their passwords This list of parameters sounds a bit weird, but the purpose is to demonstrate different data types in work: strings, numbers, dates, lists and maps. The Java model consists of two simple classes: Connection package com.example.yaml; public final class Connection { private String url; private int poolSize; public String getUrl() { return url; } public void setUrl(String url) { this.url = url; } public int getPoolSize() { return poolSize; } public void setPoolSize(int poolSize) { this.poolSize = poolSize; } @Override public String toString() { return String.format( "'%s' with pool of %d", getUrl(), getPoolSize() ); } } and Configuration, both are typical Java POJOs, verbose because of property setters and getters (we get used to it, right?). package com.example.yaml; import static java.lang.String.format; import java.util.Date; import java.util.List; import java.util.Map; public final class Configuration { private Date released; private String version; private Connection connection; private List< String > protocols; private Map< String, String > users; public Date getReleased() { return released; } public String getVersion() { return version; } public void setReleased(Date released) { this.released = released; } public void setVersion(String version) { this.version = version; } public Connection getConnection() { return connection; } public void setConnection(Connection connection) { this.connection = connection; } public List< String > getProtocols() { return protocols; } public void setProtocols(List< String > protocols) { this.protocols = protocols; } public Map< String, String > getUsers() { return users; } public void setUsers(Map< String, String > users) { this.users = users; } @Override public String toString() { return new StringBuilder() .append( format( "Version: %s\n", version ) ) .append( format( "Released: %s\n", released ) ) .append( format( "Connecting to database: %s\n", connection ) ) .append( format( "Supported protocols: %s\n", protocols ) ) .append( format( "Users: %s\n", users ) ) .toString(); } } ow, as model is quite clear, let us try to express it as the human being normally does it. Looking back to our list of required configuration, let's try to write it down one by one. 1. version and release date version: 1.0 released: 2012-11-30 2. database connection parameters connection: url: jdbc:mysql://localhost:3306/db poolSize: 5 3. list of supported protocols protocols: - http - https 4. list of users with their passwords users: tom: passwd bob: passwd And this is it, our configuration expressed in YAML syntax is completed! The whole file sample.yml looks like this: version: 1.0 released: 2012-11-30 # Connection parameters connection: url: jdbc:mysql://localhost:3306/db poolSize: 5 # Protocols protocols: - http - https # Users users: tom: passwd bob: passwd To make it work in Java, we just need to use the awesome library called snakeyml, respectively the Maven POM file is quite simple: 4.0.0 com.example yaml 0.0.1-SNAPSHOT jar UTF-8 org.yaml snakeyaml 1.11 org.apache.maven.plugins maven-compiler-plugin 2.3.1 1.7 1.7 Please notice the usage of Java 1.7, the language extensions and additional libraries simplify a lot of regular tasks as we could see looking into YamlConfigRunner: package com.example.yaml; import java.io.IOException; import java.io.InputStream; import java.nio.file.Files; import java.nio.file.Paths; import org.yaml.snakeyaml.Yaml; public class YamlConfigRunner { public static void main(String[] args) throws IOException { if( args.length != 1 ) { System.out.println( "Usage: " ); return; } Yaml yaml = new Yaml(); try( InputStream in = Files.newInputStream( Paths.get( args[ 0 ] ) ) ) { Configuration config = yaml.loadAs( in, Configuration.class ); System.out.println( config.toString() ); } } } The code snippet here loads the configuration from file (args[ 0 ]), tries to parse it and fill up the Configuration class with meaningful data using JavaBeans conventions, converting to the declared types where possible. Running this class with sample.yml as an argument generates the following output: Version: 1.0 Released: Thu Nov 29 19:00:00 EST 2012 Connecting to database: 'jdbc:mysql://localhost:3306/db' with pool of 5 Supported protocols: [http, https] Users: {tom=passwd, bob=passwd} Totally identical to the values we have configured!

When running a Java program, verbose options can be used to tell the JVM which kind of information to see. JVM suports three verbose options out of the box. As the name suggests, verbose is for displaying the work done by JVM. Mostly the information provided by these parameters is used for debugging purposes. Since it is used for debugging, its use is in development. One would never have to use verbose parameters in production enviornment. The three verbose options supported by JVM are: -verbose:class -verbose:gc -verbose:jni -verbose:class is used to display the information about classes being loaded by JVM. This is useful when using class loaders for loading classes dynamically or for analysing what all classes are getting loaded in a particular scenario. A very simple program which does nothing also loads so many classes as shown below: package com.example; public class Test { public static void main(String args[]) { } } Output: [Opened C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.lang.Object from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.io.Serializable from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.lang.Comparable from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.lang.CharSequence from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] .............................................................................. .............................................................................. .............................................................................. [Loaded java.lang.Void from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.lang.Shutdown from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] [Loaded java.lang.Shutdown$Lock from C:\Program Files\Java\jdk1.7.0_04\jre\lib\rt.jar] I am showing only selected output from Eclipse console. Classes from java.lang, java.io and java.util packages are loaded into memory by default. As stated earlier, one would use the verbose:class option when checking for what all classes are getting loaded which is usually a requirement when working with class loaders in Java. -verbose:gc is used to check garbage collection event information. When used as command line argument for running Java program, the details of garbage collection are printed on the console. The following program demonstrates the usage: package com.example; public class Test { public static void main(String args[]) throws InterruptedException { Test t1 = new Test(); t1=null; System.gc(); } } Output: [GC 318K->304K(61056K), 0.0081277 secs] [Full GC 304K->225K(61056K), 0.0054004 secs] -verbose:jni is used for printing the native methods as and when they are registered in the application. These methods include JDK as well as custom native methods. Note that jni stands for Java Native Interface. The sourcecode and output demonstrating the usage of this option is shown below: package com.example; public class Test { public static void main(String args[]) throws InterruptedException { } } Output: [Dynamic-linking native method java.lang.Object.registerNatives ... JNI] [Registering JNI native method java.lang.Object.hashCode] [Registering JNI native method java.lang.Object.wait] [Registering JNI native method java.lang.Object.notify] [Registering JNI native method java.lang.Object.notifyAll] [Registering JNI native method java.lang.Object.clone] [Dynamic-linking native method java.lang.System.registerNatives ... JNI] ............................................................... ............................................................... ............................................................... [Registering JNI native method sun.misc.Perf.highResCounter] [Registering JNI native method sun.misc.Perf.highResFrequency] [Dynamic-linking native method java.lang.ClassLoader.defineClass1 ... JNI] [Dynamic-linking native method java.lang.Runtime.gc ... JNI] [Dynamic-linking native method java.lang.ref.Finalizer.invokeFinalizeMethod ... JNI]

Groovy has an extension to its HTTPBuilder class called RESTClient which makes it fairly easy to test a RESTful web service.

So you have your executable web application in JAR with bundled Tomcat (make sure to read that one first). However there are these annoying Tomcat logs at the beginning, independent from our application logs and not customizable Nov 24, 2012 11:44:02 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-bio-8080"] Nov 24, 2012 11:44:02 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service Tomcat Nov 24, 2012 11:44:02 PM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/7.0.30 Nov 24, 2012 11:44:05 PM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["http-bio-8080"] I would really like to quite them down, or even better save them somewhere since they sometimes reveal important failures. But I definitely don't want to have a separate java.util.logging configuration. Did you wonder after reading the previous article how did I knew that runnable Tomcat JAR supports -httpPort parameter and few others? Well, I checked the sources, but it's simpler to just ask for help: $ java -jar target/standalone.jar -help usage: java -jar [path to your exec war jar] -ajpPort ajp port to use -clientAuth enable client authentication for https -D key=value -extractDirectory path to extract war content, default value: .extract -h,--help help -httpPort http port to use -httpProtocol http protocol to use: HTTP/1.1 or org.apache.coyote.http11.Http11Nio Protocol -httpsPort https port to use -keyAlias alias from keystore for ssl -loggerName logger to use: slf4j to use slf4j bridge on top of jul -obfuscate obfuscate the password and exit -resetExtract clean previous extract directory -serverXmlPath server.xml to use, optional -uriEncoding connector uriEncoding default ISO-8859-1 -X,--debug The -loggerName parameter looks quite promising. First try: $ java -jar target/standalone.jar -loggerName slf4j WARNING: issue configuring slf4j jul bridge, skip it No good. Quick look at the source code again and it turns out that SLF4J library is missing. Since this parameter is interpreted during Tomcat bootstrapping (way before web application is deployed), slf4j-api.jar inside my web application is not enough, it has to be available for root class loader (equivalent to /lib directory in packaged Tomcat). Luckily plugin exposes configuration parameter: /standalone false standalone.jar utf-8 org.slf4j slf4j-api 1.7.2 org.slf4j jul-to-slf4j 1.7.2 ch.qos.logback logback-classic 1.0.7 ch.qos.logback logback-core 1.0.7 Running Tomcat and... success! 00:01:27.110 [main] INFO o.a.coyote.http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"] 00:01:27.127 [main] INFO o.a.catalina.core.StandardService - Starting service Tomcat 00:01:27.128 [main] INFO o.a.catalina.core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.33 00:01:29.645 [main] INFO o.a.coyote.http11.Http11Protocol - Starting ProtocolHandler ["http-bio-8080"] Well, not quite. If you use Logback on a daily basis you are familiar with default console logging pattern. We are not picking up any logback.xml. From my experience it seems that placing logback.xml externally somewhere in your file system is superior to putting it inside your binary, especially with auto refreshing feature turned on: Put some fallback logback.xml file in the root of your CLASSPATH in case no other file was specified like below and voilà: $ java -jar standalone.jar -httpPort=8081 -loggerName=slf4j \ -Dlogback.configurationFile=/etc/foo/logback.xml Finally, clean and consistent logging, most likely to a single file.

If you're writing a desktop or Java Web Start program in Java using Swing, you might feel the need to run some stuff in the background by creating your own threads. There's nothing stopping you from using standard multi-threading techniques in Swing, and the usual considerations apply. If you have multiple threads accessing the same variables, you'll need to use synchronized methods or code blocks (or thread-safe classes like AtomicInteger or ArrayBlockingQueue). However, there is a pitfall for the unwary. As with most user interface APIs, you can't update the user interface from threads you've created yourself. Well, as every Java undergraduate knows, you often can, but you shouldn't. If you do this, sometimes your program will work and other times it won't. You can get around this problem by using the specialised SwingWorker class. In this article, I'll show you how you can get your programs working even if you're using the Thread class, and then we'll go on to look at the SwingWorker solution. For demonstration purposes, I've created a little Swing program. As you can see, it consists of two labels and a start button. At the moment, clicking the start button invokes a handler method which does nothing. Here's the Java code: import java.awt.Font; import java.awt.GridBagConstraints; import java.awt.GridBagLayout; import java.awt.event.ActionEvent; import java.awt.event.ActionListener; import java.util.List; import java.util.concurrent.ExecutionException; import javax.swing.JButton; import javax.swing.JFrame; import javax.swing.JLabel; import javax.swing.SwingUtilities; import javax.swing.SwingWorker; public class MainFrame extends JFrame { private JLabel countLabel1 = new JLabel("0"); private JLabel statusLabel = new JLabel("Task not completed."); private JButton startButton = new JButton("Start"); public MainFrame(String title) { super(title); setLayout(new GridBagLayout()); countLabel1.setFont(new Font("serif", Font.BOLD, 28)); GridBagConstraints gc = new GridBagConstraints(); gc.fill = GridBagConstraints.NONE; gc.gridx = 0; gc.gridy = 0; gc.weightx = 1; gc.weighty = 1; add(countLabel1, gc); gc.gridx = 0; gc.gridy = 1; gc.weightx = 1; gc.weighty = 1; add(statusLabel, gc); gc.gridx = 0; gc.gridy = 2; gc.weightx = 1; gc.weighty = 1; add(startButton, gc); startButton.addActionListener(new ActionListener() { public void actionPerformed(ActionEvent arg0) { start(); } }); setSize(200, 400); setDefaultCloseOperation(EXIT_ON_CLOSE); setVisible(true); } private void start() { } public static void main(String[] args) { SwingUtilities.invokeLater(new Runnable() { @Override public void run() { new MainFrame("SwingWorker Demo"); } }); } } We're going to add some code into the start() method which is called in response to the start button being clicked. First let's try a normal thread. private void start() { Thread worker = new Thread() { public void run() { // Simulate doing something useful. for(int i=0; i<=10; i++) { // Bad practice countLabel1.setText(Integer.toString(i)); try { Thread.sleep(1000); } catch (InterruptedException e) { } } // Bad practice statusLabel.setText("Completed."); } }; worker.start(); } As a matter of fact, this code seems to work (at least for me anyway). The program ends up looking like this: This isn't recommended practice, however. We're updating the GUI from our own thread, and under some circumstances that will certainly cause exceptions to be thrown. If we want to update the GUI from another thread, we should use SwingUtilities to schedule our update code to run on the event dispatch thread. The following code is fine, but ugly as the devil himself. private void start() { Thread worker = new Thread() { public void run() { // Simulate doing something useful. for(int i=0; i<=10; i++) { final int count = i; SwingUtilities.invokeLater(new Runnable() { public void run() { countLabel1.setText(Integer.toString(count)); } }); try { Thread.sleep(1000); } catch (InterruptedException e) { } } SwingUtilities.invokeLater(new Runnable() { public void run() { statusLabel.setText("Completed."); } }); } }; worker.start(); } Surely there must be something we can do to make our code more elegant? The SwingWorker Class SwingWorker is an alternative to using the Thread class, specifically designed for Swing. It's an abstract class and it takes two template parameters, which make it look highly ferocious and puts most people off using it. But in fact it's not as complex as it seems. Let's take a look at some code that just runs a background thread. For this first example, we won't be using either of the template parameters, so we'll set them both to Void, Java's class equivalent of the primitive void type (with a lower-case 'v'). Running a Background Task We can run a task in the background by implementing the doInBackground method and calling execute to run our code. SwingWorker worker = new SwingWorker() { @Override protected Void doInBackground() throws Exception { // Simulate doing something useful. for (int i = 0; i <= 10; i++) { Thread.sleep(1000); System.out.println("Running " + i); } return null; } }; worker.execute(); Note that SwingWorker is a one-shot affair, so if we want to run the code again, we'd need to create another SwingWorker; you can't restart the same one. Pretty simple, hey? But what if we want to update the GUI with some kind of status after running our code? You cannot update the GUI from doInBackground, because it's not running in the main event dispatch thread. But there is a solution. We need to make use of the first template parameter. Updating the GUI After the Thread Completes We can update the GUI by returning a value from doInBackground() and then over-riding done(), which can safely update the GUI. We use the get() method to retrieve the value returned from doInBackground() So the first template parameter determines the return type of both doInBackground() and get(). SwingWorker worker = new SwingWorker() { @Override protected Boolean doInBackground() throws Exception { // Simulate doing something useful. for (int i = 0; i <= 10; i++) { Thread.sleep(1000); System.out.println("Running " + i); } // Here we can return some object of whatever type // we specified for the first template parameter. // (in this case we're auto-boxing 'true'). return true; } // Can safely update the GUI from this method. protected void done() { boolean status; try { // Retrieve the return value of doInBackground. status = get(); statusLabel.setText("Completed with status: " + status); } catch (InterruptedException e) { // This is thrown if the thread's interrupted. } catch (ExecutionException e) { // This is thrown if we throw an exception // from doInBackground. } } }; worker.execute(); What if we want to update the GUI as we're going along? That's what the second template parameter is for. Updating the GUI from a Running Thread To update the GUI from a running thread, we use the second template parameter. We call the publish() method to 'publish' the values with which we want to update the user interface (which can be of whatever type the second template parameter specifies). Then we override the process() method, which receives the values that we publish. Actually process() receives lists of published values, because several values may get published before process() is actually called. In this example we just publish the latest value to the user interface. SwingWorker worker = new SwingWorker() { @Override protected Boolean doInBackground() throws Exception { // Simulate doing something useful. for (int i = 0; i <= 10; i++) { Thread.sleep(1000); // The type we pass to publish() is determined // by the second template parameter. publish(i); } // Here we can return some object of whatever type // we specified for the first template parameter. // (in this case we're auto-boxing 'true'). return true; } // Can safely update the GUI from this method. protected void done() { boolean status; try { // Retrieve the return value of doInBackground. status = get(); statusLabel.setText("Completed with status: " + status); } catch (InterruptedException e) { // This is thrown if the thread's interrupted. } catch (ExecutionException e) { // This is thrown if we throw an exception // from doInBackground. } } @Override // Can safely update the GUI from this method. protected void process(List chunks) { // Here we receive the values that we publish(). // They may come grouped in chunks. int mostRecentValue = chunks.get(chunks.size()-1); countLabel1.setText(Integer.toString(mostRecentValue)); } }; worker.execute(); More .... ? You Want More .... ? I hope you enjoyed this introduction to the highly-useful SwingWorker class. You can find more tutorials, including a complete free video course on multi-threading and courses on Swing, Android and Servlets, on my site Cave of Programming. Until next time .... happy coding. - John Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

When it comes to deploying your application, simplicity is the biggest advantage. You'll understand that especially when project evolves and needs some changes in the environment. Packaging up your whole application in one, standalone and self-sufficient JAR seems like a good idea, especially compared to installing and upgrading Tomcat in target environment. In the past I would typically include Tomcat JARs in my web application and write thin command-line runner using Tomcat API. Luckily there is a tomcat7:exec-war maven goal that does just that. It takes your WAR artifact and packages it together with all Tomcat dependencies. At the end it also includes Tomcat7RunnerCli Main-class to manifest. Curious to try it? Take your existing WAR project and add the following to your pom.xml: org.apache.tomcat.maven tomcat7-maven-plugin 2.0 tomcat-run exec-war-only package /standalone false standalone.jar utf-8 Run mvn package and few seconds later you'll find shiny standalone.jar in your target directory. Running your web application was never that simple: $ java -jar target/standalone.jar ...and you can browse localhost:8080/standalone. Although the documentation of path parameter says (emphasis mine): The webapp context path to use for the web application being run. The name to store webapp in exec jar. Do not use / just between the two of us, / seems to work after all. It turns out that built in main class is actually a little bit more flexible. For example you can say (I hope it's self-explanatory): $ java -jar standalone.jar -httpPort=7070 What this runnable JAR does is it first unpacks WAR file inside of it to some directory (.extract by default1) and deploys it to Tomcat - all required Tomcat JARs are also included. Empty standalone.jar (with few KiB WAR inside) weights around 8.5 MiB - not that much if you claim that pushing whole Tomcat with every release alongside your application is wasteful. Talking about Tomcat JARs, you should wonder how to choose Tomcat version included in this runnable? Unfortunately I couldn't find any simple option, so we must fall back to explicitly redefining plugin dependencies (version 2.0 has hardcoded 7.0.30 Tomcat). It's quite boring, but not that complicated and might be useful for future reference: UTF-8 7.0.33 org.apache.tomcat.maven tomcat7-maven-plugin 2.0 tomcat-run exec-war-only package /standalone false standalone.jar utf-8 org.apache.tomcat.embed tomcat-embed-core ${tomcat7Version} org.apache.tomcat tomcat-util ${tomcat7Version} org.apache.tomcat tomcat-coyote ${tomcat7Version} org.apache.tomcat tomcat-api ${tomcat7Version} org.apache.tomcat tomcat-jdbc ${tomcat7Version} org.apache.tomcat tomcat-dbcp ${tomcat7Version} org.apache.tomcat tomcat-servlet-api ${tomcat7Version} org.apache.tomcat tomcat-jsp-api ${tomcat7Version} org.apache.tomcat tomcat-jasper ${tomcat7Version} org.apache.tomcat tomcat-jasper-el ${tomcat7Version} org.apache.tomcat tomcat-el-api ${tomcat7Version} org.apache.tomcat tomcat-catalina ${tomcat7Version} org.apache.tomcat tomcat-tribes ${tomcat7Version} org.apache.tomcat tomcat-catalina-ha ${tomcat7Version} org.apache.tomcat tomcat-annotations-api ${tomcat7Version} In the next article we will learn how to tame these pesky Tomcat internal logs appearing in the terminal (java.util.logging...) In the meantime I discovered and reported MTOMCAT-186 Closing executable JAR does not call ServletContextListener.contextDestroyed() - have look if this is a deal breaker for you. 1 - it might be a good idea to specify different directory using -extractDirectory and clean it before every restart with -resetExtract.

Lately I encountered a configuration tweak I was not aware of, the problem: I had a single Java installation on a Linux machine from which I had to start two JVM instances - each using a different set of JCE providers. A reminder: the JVM loads its security configuration, including the JCE providers list, from a master security properties file within the JRE folder (JRE_HOME/lib/security/java.security), the location of that file is fixed in the JVM and cannot be modified. Going over the documentation (not too much helpful, I must admit) and the code (more helpful, look for Security.java, for example here) reveled the secret. security.overridePropertiesFile It all starts within the default java.security file provided with the JVM, looking at it we will find the following (somewhere around the middle of the file) # # Determines whether this properties file can be appended to # or overridden on the command line via -Djava.security.properties # security.overridePropertiesFile=true If the overridePropertiesFile doesn’t equal to true we can stop here - the rest of this article is irrelevant (unless we have the option to change it – but I didn’t have that). Lucky to me by default it does equal to true. java.security.properties Next step, the interesting one, is to override or append configuration to the default java.security file per JVM execution. This is done by setting the 'java.security.properties' system property to point to a properties file as part of the JVM invocation; it is important to notice that referencing to the file can be done in one of two flavors: Overriding the entire file provided by the JVM - if the first character in the java.security.properties' value is the equals sign the default configuration file will be entirely ignored, only the values in the file we are pointing to will be affective Appending and overriding values of the default file - any other first character in the property's value (that is the first character in the alternate configuration file path) means that the alternate file will be loaded and appended to the default one. If the alternate file contains properties which are already in the default configuration file the alternate file will override those properties. Here are two examples # # Completely override the default java.security file content # (notice the *two* equal signs) # java -Djava.security.properties==/etc/sysconfig/jvm1.java.security # # Append or override parts of the default java.security file # (notice the *single* equal sign) # java -Djava.security.properties=/etc/sysconfig/jvm.java.security Be Carefull As an important configuration option as it is we must not forget its security implications. We should always make sure that no one can tamper the value of the property and that no one can tamper the alternate file content if he shouldn't be allowed to.

This code reads the port number defined in Server.XML of Tomcat, the code source is http://www.asjava.com/tomcat/how-to-get-tomcat-port-number-in-java/ public static Integer getTomcatPortFromConfigXml(File serverXml) { Integer port; try { DocumentBuilderFactory domFactory = DocumentBuilderFactory.newInstance(); domFactory.setNamespaceAware(true); // never forget this! DocumentBuilder builder = domFactory.newDocumentBuilder(); Document doc = builder.parse(serverXml); XPathFactory factory = XPathFactory.newInstance(); XPath xpath = factory.newXPath(); XPathExpression expr = xpath.compile("/Server/Service[@name='Catalina']/Connector[count(@scheme)=0]/@port[1]"); String result = (String) expr.evaluate(doc, XPathConstants.STRING); port = result != null && result.length() > 0 ? Integer.valueOf(result) : null; } catch (Exception e) { port = null; } return port; }

To get JDK 7 up, I downloaded the JDK from Oracle. They have a nice dmg file, which makes it easy to install. After reading their installation instructions and running /usr/libexec/java_home (which I didn't even know about), it still wasn't defaulting to JDK 7. Surgery required. So, I headed over to: /System/Library/Frameworks/JavaVM.framework/Versions This is where the system jvm's are stored. You'll notice a symbolic link for CurrentJDK. It probably points to: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents You're going to want to point that to the new JDK, which java_home tells us is located in: bone@zen:/usr/libexec$ /usr/libexec/java_home /Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home So, the magic commands you need are: bone@zen:/System/Library/Frameworks/JavaVM.framework/Versions$ sudo rm CurrentJDK bone@zen:/System/Library/Frameworks/JavaVM.framework/Versions$ sudo ln -s /Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/ CurrentJDK Then, you should be good: bone@zen:/System/Library/Frameworks/JavaVM.framework/Versions$ java -version java version "1.7.0_09" Java(TM) SE Runtime Environment (build 1.7.0_09-b05) Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)

This article will provide you with a quick reference guide on how to calculate the Java process size memory footprint for Java VM processes running on IBM AIX 5.3+ OS. This is a complementary post to my original article on this subject: how to monitor the Java native memory on AIX. I highly recommend this article to any individual involved in production support or development of Java applications deployed on AIX. Why is this knowledge important? From my perspective, basic knowledge on how the OS is managing the memory allocation of your JVM processes is very important. We often overlook this monitoring aspect and only focus on the Java heap itself. From my experience, most Java memory related problems are observed from the Java heap itself such as garbage collection problems, leaks etc. However, I’m confident that you will face situations in the future involving native memory problems or OS memory challenges. Proper knowledge of your OS and virtual memory management is crucial for proper root causes analysis, recommendations and solutions. AIX memory vs. pages As you may have seen from my earlier post, the AIX Virtual Memory Manager (VMM) is responsible to manage memory requests from the system and its applications. The actual physical memory is converted and partitioned in units called pages; allocated either in physical RAM or stored on disk until it is needed. Each page can have a size of 4 KB (small page), 64 KB (medium page) or 16 MB (large page). Typically for a 64-bit Java process you will see a mix of all of the above. What about the topas command? The typical reflex when supporting applications on AIX is to run the topas command, similar to Solaris top. Find below an example of output from AIX 5.3: As you can see, the topas command is not very helpful to get a clear view on the memory utilization since it is not providing the breakdown view that we need for our analysis. It is still useful to get a rough idea of the paging space utilization which can give you a quick idea of your top "paging space" consumer processes. Same can be achieved via the ps aux command. AIX OS command to the rescue: svmon The AIX svmon command is by far my preferred command to deep dive into the Java process memory utilization. This is a very powerful command, similar to Solaris pmap. It allows you to monitor the current memory “pages” allocation along with each segment e.g. Java Heap vs. native heap segments. Analyzing the svmon output will allow you to calculate the memory footprint for each page type (4 KB, 64 KB, and 16 MB). Now find below a real example which will allow you to understand how the calculation is done: # 64-bit JVM with -Xms2048m & -Xmx2048m (2 GB Java Heap) # Command: svmon –P As you can see, the total footprint of our Java process size was found at 2.2 GB which is aligned with current Java heap settings. You should be able to easily perform the same memory footprint analysis from your AIX environment. I hope this article has helped you to understand how to calculate the Java process size on AIX OS. Please feel free to post any comment or question.