We can exclude transitive dependencies easily from specific configurations. To exclude them from all configurations we can use Groovy's spread-dot operator and invoke the exclude() method on each configuration. We can only define the group, module or both as arguments for the exclude() method. The following part of a build file shows how we can exclude a dependency from all configurations: ... configurations { all*.exclude group: 'xml-apis', module: 'xmlParserAPIs' } // Equivalent to: configurations { all.collect { configuration -> configuration.exclude group: 'xml-apis', module: 'xmlParserAPIs' } } ...

Following up on the previous article where we highlighted the top 20 features of Code Completion, I’d like to talk about the top Refactoring features that help make IntelliJ IDEA an extremely useful development tool. IntelliJ IDEA was the first Java IDE to implement the extensive set of refactorings worked out and recommended by Martin Fowler, driving other IDEs to offer this feature. Nowadays it’s hard to imagine an IDE that doesn’t provide at least a basic set of refactorings. However, it’s never about the number of refactorings you can use, but rather about how confident you feel using them. That’s why IntelliJ IDEA has always focused on refactoring productivity and refactoring safety. It’s great to improve your code quickly, but you’ve got to make sure your changes are safe to the project as a whole. In this article I give an overview of the most important refactoring features that not everyone knows and uses, which make IntelliJ IDEA really shine. Out-of-the-box support for languages and frameworks The first and perhaps the most impressive aspect of refactorings in IntelliJ IDEA is its all-encompassing support for languages and frameworks. It not only recognizes many languages, expressions and dialects (even nested inside each other), but also their relationships within the project. You can safely call refactorings for any statement at the caret, and IntelliJ IDEA will take care of applying the corresponding changes to every piece of code related to the change. This includes SQL expressions; database table definitions; Spring expressions and annotations and configurations; JSF expressions; hibernate mappings; and more. For instance, you call the Rename refactoring on a class within a JPA statement. IntelliJ IDEA recognizes that you need to rename a JPA entity class, and applies changes to the class and every JPA or other expression in the project — in mere seconds. Undo Another aspect that changes the user experince significantly is how safe and easy you can undo any change resulting from even a complicated refactoring, with just one click. Don’t be afraid to apply changes, because you can always roll them back! Find and replace code duplicates Another thing that makes some developers think IntelliJ IDEA understands their code as well as they do (or better), is detection of code duplicates. This feature is available as a separate refactoring, which you can call on any project scope, and as a part of any other refactoring, such as introduce constant, variable, method, etc. Just apply the refactoring and IntelliJ IDEA willmake appropriate changes to your code to remove duplicates. Try it just once—and you’ll wonder how you’ve lived without it all along. Rename and name patterns recognition What could be simpler than the Rename refactoring, you ask? Well, IntelliJ IDEA offers incredible additional support for this refactoring. When you use it, the IDE offers to apply the corresponding changes to getters and setters, variables, constants, test classes and methods, implementation classes, etc. This can be a huge time-saver and a lot of help in keeping your code clean Type migration Another useful feature you will rarely find in other IDEs is type migration. Have you ever used some type for a long time and then decided to change it? I’m sure you have. IntelliJ IDEA takes care of automatically applying changes to method return types, local variables, parameters and other data-flow-dependent type entries across the entire project. You can even switch between arrays and collections, and the IDE will make all the changes for you. Invert boolean If we can automate type migration, why not do the same with semantics? Exactly. For example, IntelliJ IDEA can correctly invert all usages of a boolean member or variable. Safe delete As I hinted earlier, the real benefits of refactoring are always in the details. IntelliJ IDEA tries to keep things simple for you, but there’s a lot intelligence lurking behind every feature. Even with simple deletion, it ensures that not a single line of code gets broken. String fragments Yet another time saver not found in other IDEs. IntelliJ IDEA can even extract a part of a string expression. Just select the fragment you need, and the IDE will take care of the rest. Other productivity-boosting features Many other refactorings in IntelliJ IDEA also include productivity-boosting features. For instance, you can easily change the type of extracted variable (or parameter) via ⇧⇥, just in-place, as well as replace all occurrences or declare it final. If you extract a field, the IDE will prompt you to choose where you want to initialize it. If you do it within a test, it will suggest that you initialize it in a setUp method. Inline to anonymous Everyone is used to inlining methods. However, not everyone knows that IntelliJ IDEA also provides inline refactoring for constructors. This is especially useful for such classes as Thread or Runnable. After you call it, all usages will be inlined into anonymous classes. Clone class The Clone class refactoring is yet another example of how something so simple can still save your time. As most other refactorings, it is available from a usage and helps you create a copy of any class you need. Encapsulate fields This feature is quite simple and is present in most IDEs. It helps you encapsulate fields with one click. IntelliJ IDEA goes a bit further: it can do it for a whole class at once. Consistent behavior Most Java refactorings in IntelliJ IDEA are also available from non-Java files where references to Java classes exist. Since it comes with out-of-the-box support for many custom frameworks, it offers the same shortcuts and consistent behavior for all refactorings. Framework specific refactorings In addition to Java refactorings, IntelliJ IDEA offers refactorings specific to custom frameworks, such as Spring, Java EE, Android, etc. For example, you can easily morph any component of an Android application into another type, right from the designer. Framework-specific refactorings are a wide-ranging topic that’s probably out of the scope of this article. I hope to cover it later, as well as refactorings specific to other languages, such as Scala, Groovy, JavaScript, CSS, and XML. Additional refactorings The total number of refactorings available in IntelliJ IDEA is quite high. There are about 35 Java only refactorings, plus a large number of refactorings specific to other frameworks and languages. Whichever definition of refactoring you use, it’s got more of them than any other Java IDE. Here’s a list of just the unique ones Make static Inline super class Replace inheritance with delegation Extract method object Remove middleman Wrap return value Move instance method Convert to instance method Replace temp with query Refactor this If you cannot recall the shortcut for a particular refactoring, or if you don’t feel like using the mouse, IntelliJ IDEA offers Refactor this action available via ⌘⇧⌥T. It shows you the list of refactorings applicable at the current context. Structural replace The last feature for today is Structural replace available via ⌘⇧M. This is a very powerful tool, but also the least obvious. Thank to its advanced code analysis, IntelliJ IDEA knows pretty much everything about your code. This makes possible Structural replace, which lets you use language-specific tokens in lookup and replace expressions. For example, we have a library with a new version where a static method was replaced with a singleton. To update our code, we can use the following structural replace expressions: com.ij.j2ee.MakeUtil.$MethodCall$($Params$) for lookup and com.ij.j2ee.MakeUtil.getInstance().$MethodCall$($Params$) for replace. IntelliJ IDEA will find, resolve and replaces all usages correctly, regardless of how the class was imported. Structural replace can be rather complicated at first, but once you learn how to use it, it can save you a lot of time. Summary I hope this article helps you to discover the powerful refactoring functionality hidden in IntelliJ IDEA. The more you know about your IDE, the more time it can save you every day, and the more productive you become. Go ahead and get the most out of your IntelliJ IDEA!

The Internet doesn't lack expositions on REST architecture, RESTful services, and their implementation in Java. But, here is another one. Why? Because I couldn't find something concise enough to point readers of the eValhalla blog series. What is REST? The acronym stands for Representational State Transfer. It refers to an architectural style (or pattern) thought up by one of the main authors of the HTTP protocol. Don't try to infer what the phrase "representational state transfer" could possibly mean. It sounds like there's some transfer of state that's going on between systems, but that's a bit of a stretch. Mostly, there's transfer of resources between clients and servers. The clients initiate requests and get back responses. The responses are resources in some standard media type such as XML or JSON or HTML. But, and that's a crucial aspect of the paradigm, the interaction itself is stateless. That's a major architectural departure from the classic client-server model of the 90s. Unlike classic client-server, there's no notion of a client session here. REST is offered not as a procotol, but as an architectural paradigm. However, in reality we are pretty much talking about HTTP of which REST is an abstraction. The core aspects of the architecture are (1) resource identifiers (i.e. URIs); (2) different possible representations of resources, or internet media types (e.g. application/json); (3) CRUD operations support for resources like the HTTP methods GET, PUT, POST and DEL. Resources are in principle decoupled from their identifiers. That means the environment can deliver a cached version or it can load balance somehow to fulfill the request. In practice, we all know URIs are actually addresses that resolve to particular domains so there's at least that level of coupling. In addition, resources are decoupled from their representation. A server may be asked to return HTML or XML or something else. There's content negotiation going on where the server may offer the desired representation or not. The CRUD operations have constraints on their semantics that may or may not appear obvious to you. The GET, PUT and DEL operations require that a resource be identified while POST is supposed to create a new resource. The GET operation must not have side-effects. So all other things being equal, one should be able to invoke GET many times and get back the same result. PUT updates a resource, DEL removes it and therefore they both have side-effects just like POST. On the other hand, just like GET, PUT may be repeated multiple times always to the same effect. In practice, those semantics are roughly followed. The main exception is the POST method which is frequently used to send data to the server for some processing, but without necessarily expecting it to create a new resource. Implementing RESTful services revolves around implementing those CRUD operations for various resources. This can be done in Java with the help of a Java standard API called JAX-RS. REST in Java = JAX-RS = JSR 311 In the Java world, when it comes to REST, we have the wonderful JAX-RS. And I'm not being sarcastic! This is one of those technologies that the Java Community Process actually got right, unlike so many other screw ups. The API is defined as JSR 311 and it is at version 1.1, with work on version 2.0 under way. The beauty of JAX-RS is that it is almost entirely driven by annotations. This means you can turn almost any class into a RESTful service. You can simply turn a POJO into a REST endpoint by annotating it with JSR 311 annotations. Such an annotated POJOs is called a resource class in JAX-RS terms. Some of the JAX-RS annotations are at the class level, some at the method level and others at the method parameter level. Some are available both at class and method levels. Ultimately the annotations combine to make a given Java method into a RESTful endpoint accessible at an HTTP-based URL. The annotations must specify the following elements: The relative path of the Java method - this is accomplished with @Path annotation. What the HTTP verb is, i.e. what CRUD operation is being performed - this is done by specifying one of @GET, @PUT, @POST or @DELETE annotations. The media type accepted (i.e. the representation format) - @Consumes annotation. The media type returned - @Produces annotation. The two last ones are optional. If omitted, then all media types are assumed possible. Let's look at a simple example and take it apart: import javax.ws.rs.*; @Path("/mail") @Produces("application/json") public class EmailService { @POST @Path("/new") public String sendEmail(@FormParam("subject") String subject, @FormParam("to") String to, @FormParam("body") String body) { return "new email sent"; } @GET @Path("/new") public String getUnread() { return "[]"; } @DELETE @Path("/{id}") public String deleteEmail(@PathParam("id") int emailid) { return "delete " + id; } @GET @Path("/export") @Produces("text/html") public String exportHtml(@QueryParam("searchString") @DefaultValue("") String search) { return "..."; } } The class define a RESTful interface for a hypothetical HTTP-based email service. The top-level path mail is relative to the root application path. The root application path is associated with the JAX-RS javax.ws.rs.core.Application that you extend to plugin into the runtime environment. Then we've declared with the @Produces annotation that all methods in that service produce JSON. This is just a class-default that one can override for individual methods like we've done in the exportHtml method. The sendMail method defines a typical HTTP post where the content is sent as an HTML form. The intent here would be to post to http://myserver.com/mail/new a form for a new email that should be sent out. As you can see, the API allows you to bind each separate form field to a method parameter. Note also that you have a different method for the exact same path. If you do an HTTP get at /mail/new, the Java method annotated with @GET will be called instead. Presumably the semantics of get /mail/new would be to obtain the list of unread emails. Next, note how the path of the deleteEmail method is parametarized by an integer id of the email to delete. The curly braces indicate that "id" is actually a parameter. The value of that parameter is bound to the whatever is annotated with @PathParam("id"). Thus if we do an HTTP delete at http://myserver.com/mail/453 we would be calling the deleteEmail method with argument emailid=453. Finally, the exportHtml method demonstrates how we can get a handle on query parameters. When you annotate a parameter with @QueryParam("x") the value is taken from the HTTP query parameter named x. The @DefaultValue annotation provides a default in case that query parameter is missing. So, calling http://myserver.org/mail/export?searchString=RESTful will call the exportHtml method with a parameter search="RESTful". To expose this service, first we need to write an implementation of javax.ws.rs.core.Application. That's just a few lines: public class MyRestApp extends javax.ws.rs.core.Application { public Set>Class> getClasses() { HashSet S = new HashSet(); S.add(EmailService.class); return S; } } How this gets plugged into your server depends on your JAX-RS implementation. Before we leave the API, I should mentioned that there's more to it. You do have access to a Request and Response objects. You have annotations to access other contextual information and metadata like HTTP headers, cookies etc. And you can provide custom serialization and deserialization between media types and Java objects. RESTful vs Web Services Web services (SOAP, WSDL) were heavily promoted in the past decade, but they didn't become as ubiquitous as their fans had hoped. Blame XML. Blame the rigidity of the XML Schema strong typing. Blame the tremendous overhead, the complexity of deploying and managing a web service. Or, blame the frequent compatibility nightmares between implementations. Reasons are not hard to find and the end result is that RESTful services are much easier to develop and use. But there is a flip side! The simplicity of RESTful services means that one has less guidance in how to map application logic to a REST API. One of the issues is that instead of the programmatic types we have in programming languages, we have the Java primitives and media types. Fortunately, JAX-RS allows to implement whatever conversions we want between actual Java method arguments and what gets sent on the wire. The other issue is the limited set of operations that a REST service can offer. While with web services, you define the operation and its semantics just as in a general purpose programming language, with RESTful you're stuck with get, put, post and delete. So, free from the type mismatch nightmare, but tied into only 4 possible operations. This is not as bad as it seems if you view those operations as abstract, meta operations. The key point when designing RESTful services, whether you are exposing existing application logic or creating a new one, is to think in terms of data resources. That's not so hard since most of what common business applications do is manipulate data. First, because every single thing is identified as a resource, one must come up with an appropriate naming schema. Because URIs are hierarchical, it is easy to devise a nested structure like /productcategory/productname/version/partno. Second, one must decide what kinds of representations are to be supported, both in output and input. For a modern AJAX webpp, we'd mostly use JSON. I would recommend JSON over XML even in a B2B setting where servers talk to each other. Finally, one must categorize business operation as one of GET, PUT, POST and DELETE. This is probably a bit less intuitive, but it's just a matter of getting used to. For example, instead of thinking about a "Checkout Shopping Cart" operation, think about POSTing a new order. Instead of thinking about a "Login User" operation think about GETing an authentication token. In general, every business operation manipulates some data in some way. Therefore, every business operation can fit into this crude CRUD model. Clearly, most read-only operations should be a GET. However, sometimes you have to send a large chunk of data to the server in which case you should use POST. For example you could post some very time consuming query that require a lot of text to specify. Then the resource you are creating is for example the query result. Another way to decide if you should POST or no is if you have a unique resource identifier. If not, then use POST. Obviously, operations that cause some data to be removed should be a DELETE. The operations that "store" data are PUT and again POST. Deciding between those two is easy: use PUT whenever you are modifying an existing resource for which you have an identifier. Otherwise, use POST. Implementations & Resources There are several implementations to choose from. Since, I haven't tried them all, I can't offer specific comments. Most of them used to require a servlet containers. The Restlet framework by Jerome Louvel never did, and that's why I liked it. Its documentation leaves to be desired and if you look at its code, it's over-architected to a comical degree, but then what ambitious Java framework isn't. Another newcomer that is strictly about REST and seems lightweight is Wink, an Apache incubated project. I haven't tried it, but it looks promising. And of course, one should not forget the reference implementation Jersey. Jersey has the advantage of being the most up-to-date with the spec at any given time. Originally it was dependent on Tomcat. Nowadays, it seems it can run standalone so it's on par with Restlet which I mentioned first because that's what I have mostly used. Here are some further reading resources, may their representational state be transferred to your brain and properly encoded from HTML/PDF to a compact and efficient neural net: The Wikipedia article on REST is not in very good shape, but still a starting point if you want to dig deeper into the conceptual framework. Refcard from Dzone.com: http://refcardz.dzone.com/refcardz/rest-foundations-restful#refcard-download-social-buttons-display Wink's User Guide seems well written. Since it's an implementation of JAX-RS, it's a good documentation of that technology. https://dzone.com/articles/putting-java-rest: A fairly good show-and-tell introduction to the JAX-RS API, with a link in there to a more in-depth description of REST concepts by the same author. Worth the read. http://jcp.org/en/jsr/detail?id=311: The official JSR 311 page. Download the specification and API Javadocs from there. http://jsr311.java.net/nonav/javadoc/index.html: Online access of JSR 311 Javadocs. If you know of something better, something nice, please post it in a comment and I'll include in this list. PS: I'm curious if people start new projects with Servlets, JSP/JSF these days? I would be curious as to what the rationale would be to pick those over AJAX + RESTful services communication via JSON. As I said above, this entry is intended to help readers of the eValhalla blogs series which chronicles the development of the eValhalla project following precisely the AJAX+REST model.

We've talked previously about the Windows Azure command-line tools, and have used them in a few posts such as Brian's Migrating Drupal to a Windows Azure VM. While the tools are generally useful for tons of stuff, one of the things that's been painful to do with the command-line is export the settings for a VM, and then recreate the VM from those settings. You might be wondering why you'd want to export a VM and then recreate it. For me, cost is the first thing that comes to mind. It costs more to keep a VM running than it does to just keep the disk in storage. So if I had something in a VM that I'm only using a few hours a day, I'd delete the VM when I'm not using it and recreate it when I need it again. Another potential reason is that you want to create a copy of the disk so that you can create a duplicate virtual machine. The export process used to be pretty arcane stuff; using the azure vm show command with a --json parameter and piping the output to file. Then hacking the .json file to fix it up so it could be used with the azure vm create-from command. It was bad. It was so bad, the developers added a new export command to create the .json file for you. Here's the basic process: Create a VM VM creation has been covered multiple ways already; you're either going to use the portal or command line tools, and you're either going to select an image from the library or upload a VHD. In my case, I used the following command: azure vm create larryubuntu CANONICAL__Canonical-Ubuntu-12-04-amd64-server-20120528.1.3-en-us-30GB.vhd larry NotaRe This command creates a new VM in the East US data center, enables SSH on port 22 and then stores a disk image for this VM in a blob. You can see the new disk image in blob storage by running: azure vm disk list The results should return something like: info: Executing command vm disk list + Fetching disk images data: Name OS data: ---------------------------------------- ------- data: larryubuntu-larryubuntu-0-20121019170709 Linux info: vm disk list command OK That's the actual disk image that is mounted by the VM. Export and Delete the VM Alright, I've done my work and it's the weekend. I need to export the VM settings so I can recreate it on Monday, then delete the VM so I won't get charged for the next 48 hours of not working. To export the settings for the VM, I use the following command: azure vm export larryubuntu c:\stuff\vminfo.json This tells Windows Azure to find the VM named larryubuntu and export its settings to c:\stuff\vminfo.json. The .json file will contain something like this: { "RoleName":"larryubuntu", "RoleType":"PersistentVMRole", "ConfigurationSets": [ { "ConfigurationSetType":"NetworkConfiguration", "InputEndpoints": [ { "LocalPort":"22", "Name":"ssh", "Port":"22", "Protocol":"tcp", "Vip":"168.62.177.227" } ], "SubnetNames":[] } ], "DataVirtualHardDisks":[], "OSVirtualHardDisk": { "HostCaching":"ReadWrite", "DiskName":"larryubuntu-larryubuntu-0-20121024155441", "OS":"Linux" }, "RoleSize":"Small" } If you're like me, you'll immediately start thinking "Hrmmm, I wonder if I can mess around with things like RoleSize." And yes, you can. If you wanted to bump this up to medium, you'd just change that parameter to medium. If you want to play around more with the various settings, it looks like the schema is maintained at https://github.com/WindowsAzure/azure-sdk-for-node/blob/master/lib/services/serviceManagement/models/roleschema.json. Once I've got the file, I can safely delete the VM by using the following command. azure vm delete larryubuntu It spins a bit and then no more VM. Recreate the VM Ugh, Monday. Time to go back to work, and I need my VM back up and running. So I run the following command: azure vm create-from larryubuntu c:\stuff\vminfo.json --location "East US" It takes only a minute or two to spin up the VM and it's ready for work. That's it - fast, simple, and far easier than the old process of generating the .json settings file. Note that I haven't played around much with the various settings described in the schema for the json file that I linked above. If you find anything useful or interesting that can be accomplished by hacking around with the .json, leave a comment about it.

A short blog about a topic I was discussing last week with a customer: testing SOAP Web Services. If you follow my blog you would know by now that I’m not a fan of unit testing in MOCK environments. Not because I don’t like it or I have religious believes that don’t allow me to use JUnit and Mockito. It’s just because with the work I do (mostly Java EE using application servers) my code runs in a managed environment (i.e. containers) and when I start mocking all the container’s services, it becomes cumbersome and useless. Few months ago I wrote a post about integration testing with Arquillian. But you don’t always need Arquillian to test inside a container because today, most of the containers are light and run in-memory. Think of an in-memory database. An in-memory web container. An in-memory EJB container. So first, let’s write a SOAP Web Service. I’m using the one I use on my book : a SOAP Web Service that validates a credit card. If you look at the code, there is nothing special about it (the credit card validation algorithm is a dummy one: even numbers are valid, odd are invalid). Let’s start with the interface: import javax.jws.WebService; @WebService public interface Validator { public boolean validate(CreditCard creditCard); } Then the SOAP Web Service implementation: @WebService(endpointInterface = "org.agoncal.book.javaee7.chapter21.Validator") public class CardValidator implements Validator { public boolean validate(CreditCard creditCard) { Character lastDigit = creditCard.getNumber().charAt(creditCard.getNumber().length() - 1); return Integer.parseInt(lastDigit.toString()) % 2 != 0; } } In this unit test I instantiate the CardValidator class and invoke the validate method. This is acceptable, but what if your SOAP Web Serivce uses Handlers ? What if it overrides mapping with the webservice.xml deployment descriptor ? Uses the WebServiceContext ? In short, what if your SOAP Web Service uses containers’ services ? Unit testing becomes useless. So let’s test your SOAP Web Service inside the container and write an the integration test. For that we can use an in-memory web container. And I’m not just talking about a GlassFish, JBoss or Tomcat, but something as simple as the web container that come with the SUN’s JDK. Sun’s implementation of Java SE 6 includes a light-weight HTTP server API and implementation : com.sun.net.httpserver. Note that this default HTTP server is in a com.sun package. So this might not be portable depending on the version of your JDK. Instead of using the default HTTP server it is also possible to plug other implementations as long as they provide a Service Provider Implementation (SPI) for example Jetty’s J2se6HttpServerSPI. So this is how an integration test using an in memory web container can look like: public class CardValidatorIT { @Test public void shouldCheckCreditCardValidity() throws MalformedURLException { // Publishes the SOAP Web Service Endpoint endpoint = Endpoint.publish("http://localhost:8080/cardValidator", new CardValidator()); assertTrue(endpoint.isPublished()); assertEquals("http://schemas.xmlsoap.org/wsdl/soap/http", endpoint.getBinding().getBindingID()); // Data to access the web service URL wsdlDocumentLocation = new URL("http://localhost:8080/cardValidator?wsdl"); String namespaceURI = "http://chapter21.javaee7.book.agoncal.org/"; String servicePart = "CardValidatorService"; String portName = "CardValidatorPort"; QName serviceQN = new QName(namespaceURI, servicePart); QName portQN = new QName(namespaceURI, portName); // Creates a service instance Service service = Service.create(wsdlDocumentLocation, serviceQN); Validator cardValidator = service.getPort(portQN, Validator.class); // Invokes the web service CreditCard creditCard = new CreditCard("12341234", "10/10", 1234, "VISA"); assertFalse("Credit card should be valid", cardValidator.validate(creditCard)); creditCard.setNumber("12341233"); assertTrue("Credit card should not be valid", cardValidator.validate(creditCard)); // Unpublishes the SOAP Web Service endpoint.stop(); assertFalse(endpoint.isPublished()); } } The Endpoint.publish() method uses by default the light-weight HTTP server implementation that is included in Sun’s Java SE 6. It publishes the SOAP Web Service and starts listening on URL http://localhost:8080/cardValidator. You can even go to http://localhost:8080/cardValidator?wsdl to see the generated WSDL. The integration test looks for the WSDL document, creates a service using the WSDL information, gets the port to the SOAP Web Service and then invokes the validate method. The method Endpoint.stop() stops the publishin of the service and shutsdown the in-memory web server. Again, you should be careful as this integration test uses the default HTTP server which is in a com.sun package and therefore not portable.

This is the second article in the series of "Become a Java GC Expert". In the first issue Understanding Java Garbage Collection we have learned about the processes for different GC algorithms, about how GC works, what Young and Old Generation is, what you should know about the 5 types of GC in the new JDK 7, and what the performance implications are for each of these GC types. In this article, I will explain how JVM is actually running Garbage Collection in the real time. What is GC Monitoring? Garbage Collection Monitoring refers to the process of figuring out how JVM is running GC. For example, we can find out: when an object in young has moved to old and by how much, or when stop-the-world has occurred and for how long. GC monitoring is carried out to see if JVM is running GC efficiently, and to check if additional GC tuning is necessary. Based on this information, the application can be edited or GC method can be changed (GC tuning). How to Monitor GC? There are different ways to monitor GC, but the only difference is how the GC operation information is shown. GC is done by JVM, and since the GC monitoring tools disclose the GC information provided by JVM, you will get the same results no matter how you monitor GC. Therefore, you do not need to learn all methods to monitor GC, but since it only requires a little amount of time to learn each GC monitoring method, knowing a few of them can help you use the right one for different situations and environments. The tools or JVM options listed below cannot be used universally regardless of the HVM vendor. This is because there is no need for a "standard" for disclosing GC information. In this example we will use HotSpot JVM (Oracle JVM). Since NHN is using Oracle (Sun) JVM, there should be no difficulties in applying the tools or JVM options that we are explaining here. First, the GC monitoring methods can be separated into CUI and GUI depending on the access interface. The typical CUI GC monitoring method involves using a separate CUI application called "jstat", or selecting a JVM option called "verbosegc" when running JVM. GUI GC monitoring is done by using a separate GUI application, and three most commonly used applications would be "jconsole", "jvisualvm" and "Visual GC". Let's learn more about each method. jstat jstat is a monitoring tool in HotSpot JVM. Other monitoring tools for HotSpot JVM are jps and jstatd. Sometimes, you need all three tools to monitor a Java application. jstat does not provide only the GC operation information display. It also provides class loader operation information or Just-in-Time compiler operation information. Among all the information jstat can provide, in this article we will only cover its functionality to monitor GC operating information. jstat is located in $JDK_HOME/bin, so if java or javac can run without setting a separate directory from the command line, so can jstat. You can try running the following in the command line. $> jstat –gc $ 1000 S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT 3008.0 3072.0 0.0 1511.1 343360.0 46383.0 699072.0 283690.2 75392.0 41064.3 2540 18.454 4 1.133 19.588 3008.0 3072.0 0.0 1511.1 343360.0 47530.9 699072.0 283690.2 75392.0 41064.3 2540 18.454 4 1.133 19.588 3008.0 3072.0 0.0 1511.1 343360.0 47793.0 699072.0 283690.2 75392.0 41064.3 2540 18.454 4 1.133 19.588 $> Just like in the example, the real type data will be output along with the following columns: S0C S1C S0U S1U EC EU OC OU PC. vmid (Virtual Machine ID), as its name implies, is the ID for the VM. Java applications running either on a local machine or on a remote machine can be specified using vmid. The vmid for Java application running on a local machine is called lvmid (Local vmid), and usually is PID. To find out the lvmid, you can write the PID value using a ps command or Windows task manager, but we suggest jps because PID and lvmid does not always match. jps stands for Java PS. jps shows vmids and main method information. Just like ps shows PIDs and process names. Find out the vmid of the Java application that you want to monitor by using jps, then use it as a parameter in jstat. If you use jps alone, only bootstrap information will show when several WAS instances are running in one equipment. We suggest that you use ps -ef | grep java command along with jps. GC performance data needs constant observation, therefore when running jstat, try to output the GC monitoring information on a regular basis. For example, running "jstat –gc 1000" (or 1s) will display the GC monitoring data on the console every 1 second. "jstat –gc 1000 10" will display the GC monitoring information once every 1 second for 10 times in total. There are many options other than -gc, among which GC related ones are listed below. Option Name Description gc It shows the current size for each heap area and its current usage (Ede, survivor, old, etc.), total number of GC performed, and the accumulated time for GC operations. gccapactiy It shows the minimum size (ms) and maximum size (mx) of each heap area, current size, and the number of GC performed for each area. (Does not show current usage and accumulated time for GC operations.) gccause It shows the "information provided by -gcutil" + reason for the last GC and the reason for the current GC. gcnew Shows the GC performance data for the new area. gcnewcapacity Shows statistics for the size of new area. gcold Shows the GC performance data for the old area. gcoldcapacity Shows statistics for the size of old area. gcpermcapacity Shows statistics for the permanent area. gcutil Shows the usage for each heap area in percentage. Also shows the total number of GC performed and the accumulated time for GC operations. Only looking at frequency, you will probably use -gcutil (or -gccause), -gc and -gccapacity the most in that order. -gcutil is used to check the usage of heap areas, the number of GC performed, and the total accumulated time for GC operations, while -gccapacity option and others can be used to check the actual size allocated. You can see the following output by using the -gc option: S0C S1C … GCT 1248.0 896.0 … 1.246 1248.0 896.0 … 1.246 … … … … Different jstat options show different types of columns, which are listed below. Each column information will be displayed when you use the "jstat option" listed on the right. Column Description Jstat Option S0C Displays the current size of Survivor0 area in KB -gc -gccapacity -gcnew -gcnewcapacity S1C Displays the current size of Survivor1 area in KB -gc -gccapacity -gcnew -gcnewcapacity S0U Displays the current usage of Survivor0 area in KB -gc -gcnew S1U Displays the current usage of Survivor1 area in KB -gc -gcnew EC Displays the current size of Eden area in KB -gc -gccapacity -gcnew -gcnewcapacity EU Displays the current usage of Eden area in KB -gc -gcnew OC Displays the current size of old area in KB -gc -gccapacity -gcold -gcoldcapacity OU Displays the current usage of old area in KB -gc -gcold PC Displays the current size of permanent area in KB -gc -gccapacity -gcold -gcoldcapacity -gcpermcapacity PU Displays the current usage of permanent area in KB -gc -gcold YGC The number of GC event occurred in young area -gc -gccapacity -gcnew -gcnewcapacity -gcold -gcoldcapacity -gcpermcapacity -gcutil -gccause YGCT The accumulated time for GC operations for Yong area -gc -gcnew -gcutil -gccause FGC The number of full GC event occurred -gc -gccapacity -gcnew -gcnewcapacity -gcold -gcoldcapacity -gcpermcapacity -gcutil -gccause FGCT The accumulated time for full GC operations -gc -gcold -gcoldcapacity -gcpermcapacity -gcutil -gccause GCT The total accumulated time for GC operations -gc -gcold -gcoldcapacity -gcpermcapacity -gcutil -gccause NGCMN The minimum size of new area in KB -gccapacity -gcnewcapacity NGCMX The maximum size of max area in KB -gccapacity -gcnewcapacity NGC The current size of new area in KB -gccapacity -gcnewcapacity OGCMN The minimum size of old area in KB -gccapacity -gcoldcapacity OGCMX The maximum size of old area in KB -gccapacity -gcoldcapacity OGC The current size of old area in KB -gccapacity -gcoldcapacity PGCMN The minimum size of permanent area in KB -gccapacity -gcpermcapacity PGCMX The maximum size of permanent area in KB -gccapacity -gcpermcapacity PGC The current size of permanent generation area in KB -gccapacity -gcpermcapacity PC The current size of permanent area in KB -gccapacity -gcpermcapacity PU The current usage of permanent area in KB -gc -gcold LGCC The cause for the last GC occurrence -gccause GCC The cause for the current GC occurrence -gccause TT Tenuring threshold. If copied this amount of times in young area (S0 ->S1, S1->S0), they are then moved to old area. -gcnew MTT Maximum Tenuring threshold. If copied this amount of times inside young arae, then they are moved to old area. -gcnew DSS Adequate size of survivor in KB -gcnew The advantage of jstat is that it can always monitor the GC operation data of Java applications running on local/remote machine, as long as a console can be used. From these items, the following result is output when –gcutil is used. At the time of GC tuning, pay careful attention to YGC, YGCT, FGC, FGCT and GCT. S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995 0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995 0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995 These items are important because they show how much time was spent in running GC. In this example, YGC is 217 and YGCT is 0.928. So, after calculating the arithmetical average, you can see that it required about 4 ms (0.004 seconds) for each young GC. Likewise, the average full GC time us 33ms. But the arithmetical average often does not help analyzing the actual GC problem. This is due to the severe deviations in GC operation time. (In other words, if the average time is 0.067 seconds for a full GC, one GC may have lasted 1 ms while the other one lasted 57 ms.) In order to check the individual GC time instead of the arithmetical average time, it is better to use -verbosegc. -verbosegc -verbosegc is one of the JVM options specified when running a Java application. While jstat can monitor any JVM application that has not specified any options, -verbosegc needs to be specified in the beginning, so it could be seen as an unnecessary option (since jstat can be used instead). However, as -verbosegc displays easy to understand output results whenever a GC occurs, it is very helpful for monitoring rough GC information. jstat -verbosegc Monitoring Target Java application running on a machine that can log in to a terminal, or a remote Java application that can connect to the network by using jstatd Only when -verbogc was specified as a JVM starting option Output information Heap status (usage, maximum size, number of times for GC/time, etc.) Size of ew and old area before/after GC, and GC operation time Output Time Every designated time Whenever GC occurs Whenever useful When trying to observe the changes of the size of heap area When trying to see the effect of a single GC The followings are other options that can be used with -verbosegc. -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps (from JDK 6 update 4) If only -verbosegc is used, then -XX:+PrintGCDetails is applied by default. Additional options for –verbosgc are not exclusive and can be mixed and used together. When using -verbosegc, you can see the results in the following format whenever a minor GC occurs. [GC [: -> , secs] -> , secs] ] Collector Name of Collector Used for minor gc starting occupancy1 The size of young area before GC ending occupancy1 The size of young area after GC pause time1 The time when the Java application stopped running for minor GC starting occupancy3 The total size of heap area before GC ending occupancy3 The total size of heap area after GC pause time3 The time when the Java application stopped running for overall heap GC, including major GC This is an example of -verbosegc output for minor GC: S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995 0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995 0.00 66.44 54.12 10.58 86.63 217 0.928 2 0.067 0.995 This is the example of output results after an Full GC occurred. [Full GC [Tenured: 3485K->4095K(4096K), 0.1745373 secs] 61244K->7418K(63104K), [Perm : 10756K->10756K(12288K)], 0.1762129 secs] [Times: user=0.19 sys=0.00, real=0.19 secs] If a CMS collector is used, then the following CMS information can be provided as well. As -verbosegc option outputs a log every time a GC event occurs, it is easy to see the changes of the heap usage rates caused by GC operation. (Java) VisualVM + Visual GC Java Visual VM is a GUI profiling/monitoring tool provided by Oracle JDK. Figure 1: VisualVM Screenshot. Instead of the version that is included with JDK, you can download Visual VM directly from its website. For the sake of convenience, the version included with JDK will be referred to as Java VisualVM (jvisualvm), and the version available from the website will be referred to as Visual VM (visualvm). The features of the two are not exactly identical, as there are slight differences, such as when installing plug-ins. Personally, I prefer the Visual VM version, which can be downloaded from the website. After running Visual VM, if you select the application that you wish to monitor from the window on the left side, you can find the "Monitoring" tab there. You can get the basic information about GC and Heap from this Monitoring tab. Though the basic GC status is also available through the basic features of VisualVM, you cannot access detailed information that is available from either jstat or -verbosegc option. If you want the detailed information provided by jstat, then it is recommended to install the Visual GC plug-in. Visual GC can be accessed in real time from the Tools menu. Figure 2: Viusal GC Installation Screenshot. By using Visual GC, you can see the information provided by running jstatd in a more intuitive way. Figure 3: Visual GC execution screenshot. HPJMeter HPJMeter is convenient for analyzing -verbosegc output results. If Visual GC can be considered as the GUI equivalent of jstat, then HPJMeter would be the GUI equivalent of -verbosgc. Of course, GC analysis is just one of the many features provided by HPJMeter. HPJMeter is a performance monitoring tool developed by HP. It can be used in HP-UX, as well as Linux and MS Windows. Originally, a tool called HPTune used to provide the GUI analysis feature for -verbosegc. However, since the HPTune feature has been integrated into HPJMeter since version 3.0, there is no need to download HPTune separately. When executing an application, the -verbosegc output results will be redirected to a separate file. You can open the redirected file with HPJMeter, which allows faster and easier GC performance data analysis through the intuitive GUI. Figure 4: HPJMeter.

If you've read some of my other articles on this blog you probably know I'm a fan of HTML5. With HTML5 we get all this interesting functionality, directly in the browser, in a way that, eventually, is standard across browsers. One of the new HTML5 APIs that is slowly moving through the standardization process is the Web Audio API. With this API, currently only supported in Chrome, we get access to all kinds of interesting audio components you can use to create, modify and visualize sounds (such as the following spectrogram). So why do I start with visualizations? It looks nice, that's one reason, but not the important one. This API provides a number of more complex components, whose behavior is much easier to explain when you can see what happens. With a filter you can instantly see whether some frequencies are filtered, instead of trying to listen to the resulting audio for thse changes. There are many interesting examples that use this API. The problem is, though, that getting started with this API and with digital signal processing (DSP) usually isn't explained. In this article I'll walk you through a couple of steps that shows how to do the following: Create a signal volume meter Visualize the frequencies using a spectrum analyzer And show a time based spectrogram We start with the basic setup that we can use as the basis for the components we'll create. Setting up the basic If we want to experiment with sound, we need some sound source. We could use the microphone (as we'll do later in this series), but to keep it simple, for now we'll just use an mp3 as our input. To get this working using web audio we have to take the following steps: Load the data Read it in a buffer node and play the sound Load the data With the web audio we can use different types of audio sources. We've got a MediaElementAudioSourceNode that can be used to use the audio provided by a media element. There's also a MediaStreamAudioSourceNode. With this audio source node we can use the microphone as input (see my previous article on sound recognition). Finally there is the AudioBufferSourceNode. With this node we can load the data from an existing audio file (e.g mp3) and use that as input. For this example we'll use this last approach. // create the audio context (chrome only for now) var context = new webkitAudioContext(); var audioBuffer; var sourceNode; // load the sound setupAudioNodes(); loadSound("wagner-short.ogg"); function setupAudioNodes() { // create a buffer source node sourceNode = context.createBufferSource(); // and connect to destination sourceNode.connect(context.destination); } // load the specified sound function loadSound(url) { var request = new XMLHttpRequest(); request.open('GET', url, true); request.responseType = 'arraybuffer'; // When loaded decode the data request.onload = function() { // decode the data context.decodeAudioData(request.response, function(buffer) { // when the audio is decoded play the sound playSound(buffer); }, onError); } request.send(); } function playSound(buffer) { sourceNode.buffer = buffer; sourceNode.noteOn(0); } // log if an error occurs function onError(e) { console.log(e); } In this example you can see a couple of functions. The setupAudioNodes function creates a BufferSource audio node and connects it to the destination. The loadSound function shows how you can load an audio file. The buffer which is passed into the playSound function contains decoded audio that can be used by the web audio API. In this example I use an .ogg file, for a complete overview of the formats supported look at: https://sites.google.com/a/chromium.org/dev/audio-video Play the sound To play this audio file, all we have to do is turn the source node on, this is done in the playSound function: function playSound(buffer) { sourceNode.buffer = buffer; sourceNode.noteOn(0); } You can test this out at the following page: Example 1: Loading and playing a sound with Web Audio API. When you open that page, you'll hear some music. Nothing to spectacular for now, but nevertheless an easy way to load audio that'll use for the rest of this article. The first item on our list was the volume meter. Create a volume meter One of the basic scenario's, and often one of the first steps someone new to this API tries to create, is a simple signal volume meter (or an UV meter). I expected this to be a standard component in this API, where I could just read off the signal strength as a property. But, no such node exists. But not to worry, with the components that are available, it's pretty easy (not straightforward, but easy nevertheless) to get an indication of the signal strength of your audio file. Int this section we'll create the following simple volume meter: As you can see this is a simple volume meter where we measure the signal strength for the left and the right audio channel. This is drawn on the canvas, but you could have also used divs or svg to visualize this. Lets start with a single volume meter, instead of one for each channel. For this we need to do the following: Create an analyzer node: With this node we get realtime information about the data that is processed. This data we use to determine the signal strength Create a javascript node: We use this node as a timer to update the volume meters with new information Connect everything together Analyser node With the analyser node we can perform real-time frequency and time domain analysis. From the specification: a node which is able to provide real-time frequency and time-domain analysis information. The audio stream will be passed un-processed from input to output. I won't go into the mathematical details behind this node, since there are many articles out there that explain how this works (a good one is the chapter on fourier transformation from here). What you should now about this node is that it splits up the signal in frequency buckets and we get the amplitude (the signal strenght) for each set of frequencies (the bucket). The best way to understand this, is to skip a bit ahead in this article and look at the frequency distribution we'll create later on. This image plots the result from the analyser node. The frequencies increase from left to right, and the height of the bar shows the strength of that specific frequency bucket. More on this later on in the article. For now we don't want to see the strength of the separate frequency buckets, but the strength of the total signal. For this we'll just add all the strenghts from each bucket and divide it by the number of buckets. First we need to create an analyzer node // setup a analyzer analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.3; analyser.fftSize = 1024; This creates an analyzer node whose result will be used to create the volume meter. We use a smoothingTimeConstant to make the meter less jittery. With this variable we use input from a longer time period to calculate the amplitudes, this results in a more smooth meter. The fftSize determine how many buckets we get containing frequency information. If we have a fftSize of 1024 we get 512 buckets (more info on this in the book on DPS and fourier transformations). When this node receives a stream of data, it analyzes this stream and provides us with information about the frequencies in that signal and their strengths. We now need a timer to update the meter at regular intervals. We could use the standard javascript setInterval function, but since we're looking at the Web Audio API lets use one of its nodes. The JavaScriptNode. The javascript node With the javascriptnode we can process the raw audio data directly from javascript. We can use this to write our own analyzers or complex components. We're not going to do that, though. When creating the javascript node, you can specify the interval at which it is called. We'll use that feature to update the meter at regulat intervals. Creating a javascript node is very easy. // setup a javascript node javascriptNode = context.createJavaScriptNode(2048, 1, 1); This will create a javascriptnode that is called whenever the 2048 frames have been sampled. Since our data is sampled at 44.1k, this function will be called approximately 21 times a second. Now what happens when this function is called: // when the javascript node is called // we use information from the analyzer node // to draw the volume javascriptNode.onaudioprocess = function() { // get the average, bincount is fftsize / 2 var array = new Uint8Array(analyser.frequencyBinCount); analyser.getByteFrequencyData(array); var average = getAverageVolume(array) // clear the current state ctx.clearRect(0, 0, 60, 130); // set the fill style ctx.fillStyle=gradient; // create the meters ctx.fillRect(0,130-average,25,130); } function getAverageVolume(array) { var values = 0; var average; var length = array.length; // get all the frequency amplitudes for (var i = 0; i < length; i++) { values += array[i]; } average = values / length; return average; } In these two functions we calculate the average and draw the meter directly on the canvas (using a gradient so we have nice colors). Now all we have to do is connect the output from the audiosource to the analyser, the analyser to the javasource node (and if we want audio to hear, we also need to connect something to the destionation). Connect everything together Connecting everything together is easy: function setupAudioNodes() { // setup a javascript node javascriptNode = context.createJavaScriptNode(2048, 1, 1); // connect to destination, else it isn't called javascriptNode.connect(context.destination); // setup a analyzer analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.3; analyser.fftSize = 1024; // create a buffer source node sourceNode = context.createBufferSource(); // connect the source to the analyser sourceNode.connect(analyser); // we use the javascript node to draw at a specific interval. analyser.connect(javascriptNode); // and connect to destination, if you want audio sourceNode.connect(context.destination); } And that's it. This will draw a single volume meter, for the complete signal. Now what do we do when we want to have a volume meter for each channel. For this we use a ChannelSplitter. Let's dive right into the code to connect everything: function setupAudioNodes() { // setup a javascript node javascriptNode = context.createJavaScriptNode(2048, 1, 1); // connect to destination, else it isn't called javascriptNode.connect(context.destination); // setup a analyzer analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.3; analyser.fftSize = 1024; analyser2 = context.createAnalyser(); analyser2.smoothingTimeConstant = 0.0; analyser2.fftSize = 1024; // create a buffer source node sourceNode = context.createBufferSource(); splitter = context.createChannelSplitter(); // connect the source to the analyser and the splitter sourceNode.connect(splitter); // connect one of the outputs from the splitter to // the analyser splitter.connect(analyser,0,0); splitter.connect(analyser2,1,0); // we use the javascript node to draw at a // specific interval. analyser.connect(javascriptNode); // and connect to destination sourceNode.connect(context.destination); } As you can see we don't really change much. We introduce a new node, the splitter node. This node splits the sound into a left and a right channel. These channels can be processed separately. With this layout the following happens: The audiosource creates a signal based on the buffered audio. This signal is sent to the splitter, who splits the signal into a left and right stream. Each of these two streams is processed by their own realtime analyser. From the javascript node, we now get the information from both analysers and plot both meters I've shown step 1 through 3, let's quickly move on the step 4. For this we simply add the following to the onaudioprocess node: javascriptNode.onaudioprocess = function() { // get the average for the first channel var array = new Uint8Array(analyser.frequencyBinCount); analyser.getByteFrequencyData(array); var average = getAverageVolume(array); // get the average for the second channel var array2 = new Uint8Array(analyser2.frequencyBinCount); analyser2.getByteFrequencyData(array2); var average2 = getAverageVolume(array2); // clear the current state ctx.clearRect(0, 0, 60, 130); // set the fill style ctx.fillStyle=gradient; // create the meters ctx.fillRect(0,130-average,25,130); ctx.fillRect(30,130-average2,25,130); } And now we've got two signal meters, one for each channel. Example 2: Visualize the signal strength with a volume meter. Or view the result on youtube: Now lets see how we can get the view of the frequencies I showed earlier. Create a frequency spectrum With all the work we already did in the previous section, creating a frequency spectrum overview is now very easy. We're going to aim for this: We set up the nodes just like we did in the first example: function setupAudioNodes() { // setup a javascript node javascriptNode = context.createJavaScriptNode(2048, 1, 1); // connect to destination, else it isn't called javascriptNode.connect(context.destination); // setup a analyzer analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.3; analyser.fftSize = 512; // create a buffer source node sourceNode = context.createBufferSource(); sourceNode.connect(analyser); analyser.connect(javascriptNode); // sourceNode.connect(context.destination); } So this time we don't split the channels and we set the fftSize to 512. This means we get 256 bars that represent our frequency. We now just need to alter the onaudioprocess method and the gradient we use: var gradient = ctx.createLinearGradient(0,0,0,300); gradient.addColorStop(1,'#000000'); gradient.addColorStop(0.75,'#ff0000'); gradient.addColorStop(0.25,'#ffff00'); gradient.addColorStop(0,'#ffffff'); // when the javascript node is called // we use information from the analyzer node // to draw the volume javascriptNode.onaudioprocess = function() { // get the average for the first channel var array = new Uint8Array(analyser.frequencyBinCount); analyser.getByteFrequencyData(array); // clear the current state ctx.clearRect(0, 0, 1000, 325); // set the fill style ctx.fillStyle=gradient; drawSpectrum(array); } function drawSpectrum(array) { for ( var i = 0; i < (array.length); i++ ){ var value = array[i]; ctx.fillRect(i*5,325-value,3,325); } }; In the drawSpectrum function we iterate over the array, and draw a vertical bar based on the value. That's it. For a live example, click on the following link: Example 3: Visualize the frequency spectrum. Or view it on youtube: And then the final one. The spectrogram. Time based spectrogram When you run the previous demo you see the strength of the various frequency buckets in real time. While this is a nice visualization, it doesn't allow you to analyze information over a period of time. If you want to do that you can create a spectrogram. With a spectrogram we plot a single line for each measurement. The y-axis represents the frequency, the x-asis the time and the color of a pixel the strength of that frequency. It can be used to analyze the received audio, and also creates nice looking images. The good thing, is that to output this data we don't have to change much from what we've already got in place. The only function that'll change is the onaudioprocess node and we'll create a slightly different analyser. analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0; analyser.fftSize = 1024; The enalyser we create here has an fftSize of 1024, this means we get 512 frequency buckets with strengths. So we can draw a spectrogram that has a height of 512 pixels. Also note that the smoothingTimeConstant is set to 0. This means we don't use any of the previous results in the analysis. We want to show the real information, not provide a smooth volume meter or frequency spectrum analysis. The easiest way to draw a spectrogram is by just start drawing the line at the left, and for each new set of frequencies increase the x-coordinate by one. The problem is that this will quickly fill up our canvas, and we'll only be able to see the first half a minute of the audio. To fix this, we need some creative canvas copying. The complete code for drawing the spectrogram is shown here: // create a temp canvas we use for copying and scrolling var tempCanvas = document.createElement("canvas"), tempCtx = tempCanvas.getContext("2d"); tempCanvas.width=800; tempCanvas.height=512; // used for color distribution var hot = new chroma.ColorScale({ colors:['#000000', '#ff0000', '#ffff00', '#ffffff'], positions:[0, .25, .75, 1], mode:'rgb', limits:[0, 300] }); ... // when the javascript node is called // we use information from the analyzer node // to draw the volume javascriptNode.onaudioprocess = function () { // get the average for the first channel var array = new Uint8Array(analyser.frequencyBinCount); analyser.getByteFrequencyData(array); // draw the spectrogram if (sourceNode.playbackState == sourceNode.PLAYING_STATE) { drawSpectrogram(array); } } function drawSpectrogram(array) { // copy the current canvas onto the temp canvas var canvas = document.getElementById("canvas"); tempCtx.drawImage(canvas, 0, 0, 800, 512); // iterate over the elements from the array for (var i = 0; i < array.length; i++) { // draw each pixel with the specific color var value = array[i]; ctx.fillStyle = hot.getColor(value).hex(); // draw the line at the right side of the canvas ctx.fillRect(800 - 1, 512 - i, 1, 1); } // set translate on the canvas ctx.translate(-1, 0); // draw the copied image ctx.drawImage(tempCanvas, 0, 0, 800, 512, 0, 0, 800, 512); // reset the transformation matrix ctx.setTransform(1, 0, 0, 1, 0, 0); } To draw the spectrogram we do the following: We copy what is currently drawn to a hidden canvas Next we draw a line of the current values at the far right of the canvas We set the translate on the canvas to -1 We copy the copied information back to the original canvas (that is now drawn 1 pixel to the left) And reset the transformation matrix See a running example here: Example 4: Create a spectrogram Or view it here: Last thing I'd like to mention regarding the code is the chroma.js library I used for the colors. If you ever need to draw something color or gradient related (e.g maps, strengths, levels) you can easily create color scales with this library. Two final pointers, I know I'll get questions about: Volume could be represented as a magnitude, just didn't want to complicate matters for this. The spectogram doesn't use logarithmic scales. Once again, didn't want to complicate things

Learn about the structure of JVM, how it works, executes Java bytecode, the order of execution, examples of common mistakes and their solutions, new Java SE 7 features.

For the past few months, I’ve been coaching a “Microsoft Student Partner” (who has a great blog on Kinect for Windows by the way!) on Windows Azure. One of the questions he recently had was around PartitionKey and RowKey in Windows Azure Table Storage. What are these for? Do I have to specify them manually? Let’s explain… Windows Azure storage partitions All Windows Azure storage abstractions (Blob, Table, Queue) are built upon the same stack (whitepaper here). While there’s much more to tell about it, the reason why it scales is because of its partitioning logic. Whenever you store something on Windows Azure storage, it is located on some partition in the system. Partitions are used for scale out in the system. Imagine that there’s only 3 physical machines that are used for storing data in Windows Azure storage: Based on the size and load of a partition, partitions are fanned out across these machines. Whenever a partition gets a high load or grows in size, the Windows Azure storage management can kick in and move a partition to another machine: By doing this, Windows Azure can ensure a high throughput as well as its storage guarantees. If a partition gets busy, it’s moved to a server which can support the higher load. If it gets large, it’s moved to a location where there’s enough disk space available. Partitions are different for every storage mechanism: In blob storage, each blob is in a separate partition. This means that every blob can get the maximal throughput guaranteed by the system. In queues, every queue is a separate partition. In tables, it’s different: you decide how data is co-located in the system. PartitionKey in Table Storage In Table Storage, you have to decide on the PartitionKey yourself. In essence, you are responsible for the throughput you’ll get on your system. If you put every entity in the same partition (by using the same partition key), you’ll be limited to the size of the storage machines for the amount of storage you can use. Plus, you’ll be constraining the maximal throughput as there’s lots of entities in the same partition. Should you set the PartitionKey to the same value for every entity stored? No. You’ll end up with scaling issues at some point. Should you set the PartitionKey to a unique value for every entity stored? No. You can do this and every entity stored will end up in its own partition, but you’ll find that querying your data becomes more difficult. And that’s where our next concept kicks in… RowKey in Table Storage A RowKey in Table Storage is a very simple thing: it’s your “primary key” within a partition. PartitionKey + RowKey form the composite unique identifier for an entity. Within one PartitionKey, you can only have unique RowKeys. If you use multiple partitions, the same RowKey can be reused in every partition. So in essence, a RowKey is just the identifier of an entity within a partition. PartitionKey and RowKey and performance Before building your code, it’s a good idea to think about both properties. Don’t just assign them a guid or a random string as it does matter for performance. The fastest way of querying? Specifying both PartitionKey and RowKey. By doing this, table storage will immediately know which partition to query and can simply do an ID lookup on RowKey within that partition. Less fast but still fast enough will be querying by specifying PartitionKey: table storage will know which partition to query. Less fast: querying on only RowKey. Doing this will give table storage no pointer on which partition to search in, resulting in a query that possibly spans multiple partitions, possibly multiple storage nodes as well. Wihtin a partition, searching on RowKey is still pretty fast as it’s a unique index. Slow: searching on other properties (again, spans multiple partitions and properties). Note that Windows Azure storage may decide to group partitions in so-called "Range partitions" - see http://msdn.microsoft.com/en-us/library/windowsazure/hh508997.aspx. In order to improve query performance, think about your PartitionKey and RowKey upfront, as they are the fast way into your datasets. Deciding on PartitionKey and RowKey Here’s an exercise: say you want to store customers, orders and orderlines. What will you choose as the PartitionKey (PK) / RowKey (RK)? Let’s use three tables: Customer, Order and Orderline. An ideal setup may be this one, depending on how you want to query everything: Customer (PK: sales region, RK: customer id) – it enables fast searches on region and on customer id Order (PK: customer id, RK; order id) – it allows me to quickly fetch all orders for a specific customer (as they are colocated in one partition), it still allows fast querying on a specific order id as well) Orderline (PK: order id, RK: order line id) – allows fast querying on both order id as well as order line id. Of course, depending on the system you are building, the following may be a better setup: Customer (PK: customer id, RK: display name) – it enables fast searches on customer id and display name Order (PK: customer id, RK; order id) – it allows me to quickly fetch all orders for a specific customer (as they are colocated in one partition), it still allows fast querying on a specific order id as well) Orderline (PK: order id, RK: item id) – allows fast querying on both order id as well as the item bought, of course given that one order can only contain one order line for a specific item (PK + RK should be unique) You see? Choose them wisely, depending on your queries. And maybe an important sidenote: don’t be afraid of denormalizing your data and storing data twice in a different format, supporting more query variations. There’s one additional “index” That’s right! People have been asking Microsoft for a secondary index. And it’s already there… The table name itself! Take our customer – order – orderline sample again… Having a Customer table containing all customers may be interesting to search within that data. But having an Orders table containing every order for every customer may not be the ideal solution. Maybe you want to create an order table per customer? Doing that, you can easily query the order id (it’s the table name) and within the order table, you can have more detail in PK and RK. And there's one more: your account name. Split data over multiple storage accounts and you have yet another "partition". Conclusion In conclusion? Choose PartitionKey and RowKey wisely. The more meaningful to your application or business domain, the faster querying will be and the more efficient table storage will work in the long run.

ASP.NET Web API is a great tool to build an API with. Or as my buddy Kristof Rennen (and the French) always say: “it makes you ‘api”. One of the things I like a lot is the fact that you can do very powerful things that you know and love from the ASP.NET MVC stack, like, for example, using filter attributes. Action filters, result filters and… authorization filters. Say you wanted to protect your API and make use of the controller’s User property to return user-specific information. You probably will add an [Authorize] attribute (to ensure the user is authenticated) to either the entire API controller or to one of its action methods, like this: [Authorize] public class SuperSecretController : ApiController { public string Get() { return string.Format("Hello, {0}", User.Identity.Name); } } Great! But how will your application know who’s calling? Forms authentication doesn’t really make sense for a lot of API’s. Configuring IIS and switching to Windows authentication or basic authentication may be an option. But not every ASP.NET Web API will live in IIS, right? And maybe you want to use some other form of authentication for your API, for example one that uses a custom HTTP header containing an API key? Let’s see how you can do that… Our API authentication? An API key API keys may make sense for your API. They provide an easy means of authenticating your API consumers based on a simple token that is passed around in a custom header. OAuth2 may make sense as well, but even that one boils down to a custom Authorization header at the HTTP level. (hint: the approach outlined in this post can be used for OAuth2 tokens as well) Let’s build our API and require every API consumer to pass in a custom header, named “X-ApiKey”. Calls to our API will look like this: GET http://localhost:60573/api/v1/SuperSecret HTTP/1.1 Host: localhost:60573 X-ApiKey: 12345 In our SuperSecretController above, we want to make sure that we’re working with a traditional IPrincipal which we can query for username, roles and possibly even claims if needed. How do we get that identity there? Translating the API key using a DelegatingHandler The title already gives you a pointer. We want to add a plugin into ASP.NET Web API’s pipeline which replaces the current thread’s IPrincipal with one that is mapped from the incoming API key. That plugin will come in the form of a DelegatingHandler, a class that’s plugged in really early in the ASP.NET Web API pipeline. I’m not going to elaborate on what DelegatingHandler does and where it fits, there’s a perfect post on that to be found here. Our handler, which I’ll call AuthorizationHeaderHandler will be inheriting ASP.NET Web API’s DelegatingHandler. The method we’re interested in is SendAsync, which will be called on every request into our API. public class AuthorizationHeaderHandler : DelegatingHandler { protected override Task SendAsync( HttpRequestMessage request, CancellationToken cancellationToken) { // ... } } This method offers access to the HttpRequestMessage, which contains everything you’ll probably be needing such as… HTTP headers! Let’s read out our X-ApiKey header, convert it to a ClaimsIdentity (so we can add additional claims if needed) and assign it to the current thread: public class AuthorizationHeaderHandler : DelegatingHandler { protected override Task SendAsync( HttpRequestMessage request, CancellationToken cancellationToken) { IEnumerable apiKeyHeaderValues = null; if (request.Headers.TryGetValues("X-ApiKey", out apiKeyHeaderValues)) { var apiKeyHeaderValue = apiKeyHeaderValues.First(); // ... your authentication logic here ... var username = (apiKeyHeaderValue == "12345" ? "Maarten" : "OtherUser"); var usernameClaim = new Claim(ClaimTypes.Name, username); var identity = new ClaimsIdentity(new[] {usernameClaim}, "ApiKey"); var principal = new ClaimsPrincipal(identity); Thread.CurrentPrincipal = principal; } return base.SendAsync(request, cancellationToken); } } Easy, no? The only thing left to do is registering this handler in the pipeline during your application’s start: GlobalConfiguration.Configuration.MessageHandlers.Add(new AuthorizationHeaderHandler()); From now on, any request coming in with the X-ApiKey header will be translated into an IPrincipal which you can easily use throughout your web API. Enjoy! PS: if you’re looking into OAuth2, I’ve used a similar approach in “ASP.NET Web API OAuth2 delegation with Windows Azure Access Control Service” to handle OAuth2 tokens.

The content of this article was originally written by Tae Jin Gu on the Cubrid blog. When there is an obstacle, or when a Java based Web application is running much slower than expected, we need to use thread dumps. If thread dumps feel like very complicated to you, this article may help you very much. Here I will explain what threads are in Java, their types, how they are created, how to manage them, how you can dump threads from a running application, and finally how you can analyze them and determine the bottleneck or blocking threads. This article is a result of long experience in Java application debugging. Java and Thread A web server uses tens to hundreds of threads to process a large number of concurrent users. If two or more threads utilize the same resources, a contention between the threads is inevitable, and sometimes deadlock occurs. Thread contention is a status in which one thread is waiting for a lock, held by another thread, to be lifted. Different threads frequently access shared resources on a web application. For example, to record a log, the thread trying to record the log must obtain a lock and access the shared resources. Deadlock is a special type of thread contention, in which two or more threads are waiting for the other threads to complete their tasks in order to complete their own tasks. Different issues can arise from thread contention. To analyze such issues, you need to use the thread dump. A thread dump will give you the information on the exact status of each thread. Background Information for Java Threads Thread Synchronization A thread can be processed with other threads at the same time. In order to ensure compatibility when multiple threads are trying to use shared resources, one thread at a time should be allowed to access the shared resources by using thread synchronization. Thread synchronization on Java can be done using monitor. Every Java object has a single monitor. The monitor can be owned by only one thread. For a thread to own a monitor that is owned by a different thread, it needs to wait in the wait queue until the other thread releases its monitor. Thread Status In order to analyze a thread dump, you need to know the status of threads. The statuses of threads are stated on java.lang.Thread.State. Figure 1: Thread Status. NEW: The thread is created but has not been processed yet. RUNNABLE: The thread is occupying the CPU and processing a task. (It may be in WAITING status due to the OS's resource distribution.) BLOCKED: The thread is waiting for a different thread to release its lock in order to get the monitor lock. WAITING: The thread is waiting by using a wait, join or park method. TIMED_WAITING: The thread is waiting by using a sleep, wait, join or park method. (The difference from WAITING is that the maximum waiting time is specified by the method parameter, and WAITING can be relieved by time as well as external changes.) Thread Types Java threads can be divided into two: daemon threads; and non-daemon threads. Daemon threads stop working when there are no other non-daemon threads. Even if you do not create any threads, the Java application will create several threads by default. Most of them are daemon threads, mainly for processing tasks such as garbage collection or JMX. A thread running the 'static void main(String[] args)’ method is created as a non-daemon thread, and when this thread stops working, all other daemon threads will stop as well. (The thread running this main method is called the VM thread in HotSpot VM.) Getting a Thread Dump We will introduce the three most commonly used methods. Note that there are many other ways to get a thread dump. A thread dump can only show the thread status at the time of measurement, so in order to see the change in thread status, it is recommended to extract them from 5 to 10 times with 5-second intervals. Getting a Thread Dump Using jstack In JDK 1.6 and higher, it is possible to get a thread dump on MS Windows using jstack. Use PID via jps to check the PID of the currently running Java application process. [user@linux ~]$ jps -v 25780 RemoteTestRunner -Dfile.encoding=UTF-8 25590 sub.rmi.registry.RegistryImpl 2999 -Dapplication.home=/home1/user/java/jdk.1.6.0_24 -Xms8m 26300 sun.tools.jps.Jps -mlvV -Dapplication.home=/home1/user/java/jdk.1.6.0_24 -Xms8m Use the extracted PID as the parameter of jstack to obtain a thread dump. [user@linux ~]$ jstack -f 5824 A Thread Dump Using jVisualVM Generate a thread dump by using a program such as jVisualVM. Figure 2: A Thread Dump Using visualvm. The task on the left indicates the list of currently running processes. Click on the process for which you want the information, and select the thread tab to check the thread information in real time. Click the Thread Dump button on the top right corner to get the thread dump file. Generating in a Linux Terminal Obtain the process pid by using ps -ef command to check the pid of the currently running Java process. [user@linux ~]$ ps - ef | grep java user 2477 1 0 Dec23 ? 00:10:45 ... user 25780 25361 0 15:02 pts/3 00:00:02 ./jstatd -J -Djava.security.policy=jstatd.all.policy -p 2999 user 26335 25361 0 15:49 pts/3 00:00:00 grep java Use the extracted pid as the parameter of kill –SIGQUIT(3) to obtain a thread dump. Thread Information from the Thread Dump File "pool-1-thread-13" prio=6 tid=0x000000000729a000 nid=0x2fb4 runnable [0x0000000007f0f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158) - locked <0x0000000780b7e688> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(InputStreamReader.java:167) at java.io.BufferedReader.fill(BufferedReader.java:136) at java.io.BufferedReader.readLine(BufferedReader.java:299) - locked <0x0000000780b7e688> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(BufferedReader.java:362) ) Thread name: When using Java.lang.Thread class to generate a thread, the thread will be named Thread-(Number), whereas when using java.util.concurrent.ThreadFactory class, it will be named pool-(number)-thread-(number). Priority: Represents the priority of the threads. Thread ID: Represents the unique ID for the threads. (Some useful information, including the CPU usage or memory usage of the thread, can be obtained by using thread ID.) Thread status: Represents the status of the threads. Thread callstack: Represents the call stack information of the threads. Thread Dump Patterns by Type When Unable to Obtain a Lock (BLOCKED) This is when the overall performance of the application slows down because a thread is occupying the lock and prevents other threads from obtaining it. In the following example, BLOCKED_TEST pool-1-thread-1 thread is running with <0x0000000780a000b0> lock, while BLOCKED_TEST pool-1-thread-2 and BLOCKED_TEST pool-1-thread-3 threads are waiting to obtain <0x0000000780a000b0> lock. Figure 3: A thread blocking other threads. "BLOCKED_TEST pool-1-thread-1" prio=6 tid=0x0000000006904800 nid=0x28f4 runnable [0x000000000785f000] java.lang.Thread.State: RUNNABLE at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:282) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) - locked <0x0000000780a31778> (a java.io.BufferedOutputStream) at java.io.PrintStream.write(PrintStream.java:432) - locked <0x0000000780a04118> (a java.io.PrintStream) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272) at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85) - locked <0x0000000780a040c0> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168) at java.io.PrintStream.newLine(PrintStream.java:496) - locked <0x0000000780a04118> (a java.io.PrintStream) at java.io.PrintStream.println(PrintStream.java:687) - locked <0x0000000780a04118> (a java.io.PrintStream) at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:44) - locked <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState) at com.nbp.theplatform.threaddump.ThreadBlockedState$1.run(ThreadBlockedState.java:7) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000780a31758> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "BLOCKED_TEST pool-1-thread-2" prio=6 tid=0x0000000007673800 nid=0x260c waiting for monitor entry [0x0000000008abf000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:43) - waiting to lock <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState) at com.nbp.theplatform.threaddump.ThreadBlockedState$2.run(ThreadBlockedState.java:26) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000780b0c6a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "BLOCKED_TEST pool-1-thread-3" prio=6 tid=0x00000000074f5800 nid=0x1994 waiting for monitor entry [0x0000000008bbf000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:42) - waiting to lock <0x0000000780a000b0> (a com.nbp.theplatform.threaddump.ThreadBlockedState) at com.nbp.theplatform.threaddump.ThreadBlockedState$3.run(ThreadBlockedState.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000780b0e1b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) When in Deadlock Status This is when thread A needs to obtain thread B's lock to continue its task, while thread B needs to obtain thread A's lock to continue its task. In the thread dump, you can see that DEADLOCK_TEST-1 thread has 0x00000007d58f5e48 lock, and is trying to obtain 0x00000007d58f5e60 lock. You can also see that DEADLOCK_TEST-2 thread has 0x00000007d58f5e60 lock, and is trying to obtain 0x00000007d58f5e78 lock. Also, DEADLOCK_TEST-3 thread has 0x00000007d58f5e78 lock, and is trying to obtain 0x00000007d58f5e48 lock. As you can see, each thread is waiting to obtain another thread's lock, and this status will not change until one thread discards its lock. Figure 4: Threads in a Deadlock status. "DEADLOCK_TEST-1" daemon prio=6 tid=0x000000000690f800 nid=0x1820 waiting for monitor entry [0x000000000805f000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197) - waiting to lock <0x00000007d58f5e60> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182) - locked <0x00000007d58f5e48> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers: - None "DEADLOCK_TEST-2" daemon prio=6 tid=0x0000000006858800 nid=0x17b8 waiting for monitor entry [0x000000000815f000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197) - waiting to lock <0x00000007d58f5e78> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182) - locked <0x00000007d58f5e60> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers: - None "DEADLOCK_TEST-3" daemon prio=6 tid=0x0000000006859000 nid=0x25dc waiting for monitor entry [0x000000000825f000] java.lang.Thread.State: BLOCKED (on object monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197) - waiting to lock <0x00000007d58f5e48> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182) - locked <0x00000007d58f5e78> (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor) at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers: - None When Continuously Waiting to Receive Messages from a Remote Server The thread appears to be normal, since its state keeps showing as RUNNABLE. However, when you align the thread dumps chronologically, you can see that socketReadThread thread is waiting infinitely to read the socket. Figure 5: Continuous Waiting Status. "socketReadThread" prio=6 tid=0x0000000006a0d800 nid=0x1b40 runnable [0x00000000089ef000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158) - locked <0x00000007d78a2230> (a java.io.InputStreamReader) at sun.nio.cs.StreamDecoder.read0(StreamDecoder.java:107) - locked <0x00000007d78a2230> (a java.io.InputStreamReader) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:93) at java.io.InputStreamReader.read(InputStreamReader.java:151) at com.nbp.theplatform.threaddump.ThreadSocketReadState$1.run(ThreadSocketReadState.java:27) at java.lang.Thread.run(Thread.java:662) When Waiting The thread is maintaining WAIT status. In the thread dump, IoWaitThread thread keeps waiting to receive a message from LinkedBlockingQueue. If there continues to be no message for LinkedBlockingQueue, then the thread status will not change. Figure 6: Waiting status. "IoWaitThread" prio=6 tid=0x0000000007334800 nid=0x2b3c waiting on condition [0x000000000893f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007d5c45850> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:440) at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:629) at com.nbp.theplatform.threaddump.ThreadIoWaitState$IoWaitHandler2.run(ThreadIoWaitState.java:89) at java.lang.Thread.run(Thread.java:662) When Thread Resources Cannot be Organized Normally Unnecessary threads will pile up when thread resources cannot be organized normally. If this occurs, it is recommended to monitor the thread organization process or check the conditions for thread termination. Figure 7: Unorganized Threads. How to Solve Problems by Using Thread Dump Example 1: When the CPU Usage is Abnormally High 1. Extract the thread that has the highest CPU usage. [user@linux ~]$ ps -mo pid.lwp.stime.time.cpu -C java PID LWP STIME TIME %CPU 10029 - Dec07 00:02:02 99.5 - 10039 Dec07 00:00:00 0.1 - 10040 Dec07 00:00:00 95.5 From the application, find out which thread is using the CPU the most. Acquire the Light Weight Process (LWP) that uses the CPU the most and convert its unique number (10039) into a hexadecimal number (0x2737). 2. After acquiring the thread dump, check the thread's action. Extract the thread dump of an application with a PID of 10029, then find the thread with an nid of 0x2737. "NioProcessor-2" prio=10 tid=0x0a8d2800 nid=0x2737 runnable [0x49aa5000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x74c52678> (a sun.nio.ch.Util$1) - locked <0x74c52668> (a java.util.Collections$UnmodifiableSet) - locked <0x74c501b0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at external.org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65) at external.org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:708) at external.org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Extract thread dumps several times every hour, and check the status change of the threads to determine the problem. Example 2: When the Processing Performance is Abnormally Slow After acquiring thread dumps several times, find the list of threads with BLOCKED status. " DB-Processor-13" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f000] java.lang.Thread.State: BLOCKED (on object monitor) at beans.ConnectionPool.getConnection(ConnectionPool.java:102) - waiting to lock <0xe0375410> (a beans.ConnectionPool) at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111) at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43) "DB-Processor-14" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f020] java.lang.Thread.State: BLOCKED (on object monitor) at beans.ConnectionPool.getConnection(ConnectionPool.java:102) - waiting to lock <0xe0375410> (a beans.ConnectionPool) at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111) at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43) " DB-Processor-3" daemon prio=5 tid=0x00928248 nid=0x8b waiting for monitor entry [0x000000000825d080] java.lang.Thread.State: RUNNABLE at oracle.jdbc.driver.OracleConnection.isClosed(OracleConnection.java:570) - waiting to lock <0xe03ba2e0> (a oracle.jdbc.driver.OracleConnection) at beans.ConnectionPool.getConnection(ConnectionPool.java:112) - locked <0xe0386580> (a java.util.Vector) - locked <0xe0375410> (a beans.ConnectionPool) at beans.cus.Cue_1700c.GetNationList(Cue_1700c.java:66) at org.apache.jsp.cue_1700c_jsp._jspService(cue_1700c_jsp.java:120) Acquire the list of threads with BLOCKED status after getting the thread dumps several times. If the threads are BLOCKED, extract the threads related to the lock that the threads are trying to obtain. Through the thread dump, you can confirm that the thread status stays BLOCKED because <0xe0375410> lock could not be obtained. This problem can be solved by analyzing stack trace from the thread currently holding the lock. There are two reasons why the above pattern frequently appears in applications using DBMS. The first reason is inadequate configurations. Despite the fact that the threads are still working, they cannot show their best performance because the configurations for DBCP and the like are not adequate. If you extract thread dumps multiple times and compare them, you will often see that some of the threads that were BLOCKED previously are in a different state. The second reason is the abnormal connection. When the connection with DBMS stays abnormal, the threads wait until the time is out. In this case, even after extracting the thread dumps several times and comparing them, you will see that the threads related to DBMS are still in a BLOCKED state. By adequately changing the values, such as the timeout value, you can shorten the time in which the problem occurs. Coding for Easy Thread Dump Naming Threads When a thread is created using java.lang.Thread object, the thread will be named Thread-(Number). When a thread is created using java.util.concurrent.DefaultThreadFactory object, the thread will be named pool-(Number)-thread-(Number). When analyzing tens to thousands of threads for an application, if all the threads still have their default names, analyzing them becomes very difficult, because it is difficult to distinguish the threads to be analyzed. Therefore, you are recommended to develop the habit of naming the threads whenever a new thread is created. When you create a thread using java.lang.Thread, you can give the thread a custom name by using the creator parameter. public Thread(Runnable target, String name); public Thread(ThreadGroup group, String name); public Thread(ThreadGroup group, Runnable target, String name); public Thread(ThreadGroup group, Runnable target, String name, long stackSize); When you create a thread using java.util.concurrent.ThreadFactory, you can name it by generating your own ThreadFactory. If you do not need special functionalities, then you can use MyThreadFactory as described below: import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ThreadFactory; import java.util.concurrent.atomic.AtomicInteger; public class MyThreadFactory implements ThreadFactory { private static final ConcurrentHashMap POOL_NUMBER = new ConcurrentHashMap(); private final ThreadGroup group; private final AtomicInteger threadNumber = new AtomicInteger(1); private final String namePrefix; public MyThreadFactory(String threadPoolName) { if (threadPoolName == null) { throw new NullPointerException("threadPoolName"); } POOL_NUMBER.putIfAbsent(threadPoolName, new AtomicInteger()); SecurityManager securityManager = System.getSecurityManager(); group = (securityManager != null) ? securityManager.getThreadGroup() : Thread.currentThread().getThreadGroup(); AtomicInteger poolCount = POOL_NUMBER.get(threadPoolName); if (poolCount == null) { namePrefix = threadPoolName + " pool-00-thread-"; } else { namePrefix = threadPoolName + " pool-" + poolCount.getAndIncrement() + "-thread-"; } } public Thread newThread(Runnable runnable) { Thread thread = new Thread(group, runnable, namePrefix + threadNumber.getAndIncrement(), 0); if (thread.isDaemon()) { thread.setDaemon(false); } if (thread.getPriority() != Thread.NORM_PRIORITY) { thread.setPriority(Thread.NORM_PRIORITY); } return thread; } } Obtaining More Detailed Information by Using MBean You can obtain ThreadInfo objects using MBean. You can also obtain more information that would be difficult to acquire via thread dumps, by using ThreadInfo. ThreadMXBean mxBean = ManagementFactory.getThreadMXBean(); long[] threadIds = mxBean.getAllThreadIds(); ThreadInfo[] threadInfos = mxBean.getThreadInfo(threadIds); for (ThreadInfo threadInfo : threadInfos) { System.out.println( threadInfo.getThreadName()); System.out.println( threadInfo.getBlockedCount()); System.out.println( threadInfo.getBlockedTime()); System.out.println( threadInfo.getWaitedCount()); System.out.println( threadInfo.getWaitedTime()); } You can acquire the amount of time that the threads WAITed or were BLOCKED by using the method in ThreadInfo, and by using this you can also obtain the list of threads that have been inactive for an abnormally long period of time. In Conclusion In this article I was concerned that for developers with a lot of experience in multi-thread programming, this material may be common knowledge, whereas for less experienced developers, I felt that I was skipping straight to thread dumps, without providing enough background information about the thread activities. This was because of my lack of knowledge, as I was not able to explain the thread activities in a clear yet concise manner. I sincerely hope that this article will prove helpful for many developers.

It's awesome that JUnit is recognizing the usefulness of Hamcrest, because I use these two a lot. However, I find JUnit packaging of their dependencies odd, and can cause class loading problem if you are not careful. Let's take a closer look. If you look at junit:junit:4.10 from Maven Central, you will see that it has this dependencies graph: +- junit:junit:jar:4.10:test | - org.hamcrest:hamcrest-core:jar:1.1:test This is great, except that inside the junit-4.10.jar, you will also find the hamcrest-core-1.1.jar content are embedded! But why??? I suppose it's a convenient for folks who use Ant, so that they save one jar to package in their lib folder, but it's not very Maven friendly. And you also expect classloading trouble if you want to upgrade Hamcrest or use extra Hamcrest modules. Now if you use Hamcrest long enough, you know that most of their goodies are in the second module named hamcrest-library, but this JUnit didn't package in. JUnit however chose to include some JUnit+Hamcrest extension of their own. Now including duplicated classes in jar are very trouble maker, so JUnit has a separated module junit-dep that doesn't include Hamcrest core package and help you avoid this issue. So if you are using Maven project, you should use this instead. junit junit-dep 4.10 test org.hamcrest hamcrest-core org.hamcrest hamcrest-library 1.2.1 test See how I have to exclude hamcrest from junit. This is needed if you want hamcrest-library that has higher version than the one JUnit comes with, which is 1.1. Interesting enough, Maven's dependencies in pom is order sensitive when it comes to auto resolving conflicting versions dependencies. Actually it would just pick the first one found and ignore the rest. So you can shorten above without exclusion if, only if, you place the Hamcrest bofore JUnit like this: org.hamcrest hamcrest-library 1.2.1 test junit junit-dep 4.10 test This should make Maven use the following dependencies: +- org.hamcrest:hamcrest-library:jar:1.2.1:test | \- org.hamcrest:hamcrest-core:jar:1.2.1:test +- junit:junit-dep:jar:4.10:test However I think using the exclusion tag would probably give you more stable build and not rely on Maven implicit ordering rule. And it avoid easy mistake for Maven beginer users. However I wish JUnit would do a better job at packaging and remove duplicated classes in jar. I personally think it's more productive for JUnit to also include hamcrest-libray instead of just the hamcrest-core jar. What do you think?

EasyNetQ, my super simple .NET API for RabbitMQ, now (from version 0.7.2.34) supports RabbitMQ clusters without any need to deploy a load balancer. Simply list the nodes of the cluster in the connection string ... var bus = RabbitHutch.CreateBus("host=ubuntu:5672,ubuntu:5673"); In this example I have set up a cluster on a single machine, 'ubuntu', with node 1 on port 5672 and node 2 on port 5673. When the CreateBus statement executes, EasyNetQ will attempt to connect to the first host listed (ubuntu:5672). If it fails to connect it will attempt to connect to the second host listed (ubuntu:5673). If neither node is available it will sit in a re-try loop attempting to connect to both servers every five seconds. It logs all this activity to the registered IEasyNetQLogger. You might see something like this if the first node was unavailable: DEBUG: Trying to connect ERROR: Failed to connect to Broker: 'ubuntu', Port: 5672 VHost: '/'. ExceptionMessage: 'None of the specified endpoints were reachable' DEBUG: OnConnected event fired INFO: Connected to RabbitMQ. Broker: 'ubuntu', Port: 5674, VHost: '/' If the node that EasyNetQ is connected to fails, EasyNetQ will attempt to connect to the next listed node. Once connected, it will re-declare all the exchanges and queues and re-start all the consumers. Here's an example log record showing one node failing then EasyNetQ connecting to the other node and recreating the subscribers: INFO: Disconnected from RabbitMQ Broker DEBUG: Trying to connect DEBUG: OnConnected event fired DEBUG: Re-creating subscribers INFO: Connected to RabbitMQ. Broker: 'ubuntu', Port: 5674, VHost: '/' You get automatic fail-over out of the box. That’s pretty cool. If you have multiple services using EasyNetQ to connect to a RabbitMQ cluster, they will all initially connect to the first listed node in their respective connection strings. For this reason the EasyNetQ cluster support is not really suitable for load balancing high throughput systems. I would recommend that you use a dedicated hardware or software load balancer instead, if that’s what you want.

Continuing to discover the powerful set of Redis features, the one worth mentioning about is out of the box support of pub/sub messaging. Pub/Sub messaging is essential part of many software architectures. Some software systems demand from messaging solution to provide high-performance, scalability, queues persistence and durability, fail-over support, transactions, and many more nice-to-have features, which in Java world mostly always leads to using one of JMS implementation providers. In my previous projects I have actively used Apache ActiveMQ (now moving towards Apache ActiveMQ Apollo). Though it's a great implementation, sometimes I just needed simple queuing support and Apache ActiveMQ just looked overcomplicated for that. Alternatives? Please welcome Redis pub/sub! If you are already using Redis as key/value store, few additional lines of configuration will bring pub/sub messaging to your application in no time. Spring Data Redis project abstracts very well Redis pub/sub API and provides the model so familiar to everyone who uses Spring capabilities to integrate with JMS. As always, let's start with the POM configuration file. It's pretty small and simple, includes necessary Spring dependencies, Spring Data Redis and Jedis, great Java client for Redis. 4.0.0 com.example.spring redis 0.0.1-SNAPSHOT jar UTF-8 3.1.1.RELEASE org.springframework.data spring-data-redis 1.0.1.RELEASE cglib cglib-nodep 2.2 log4j log4j 1.2.16 redis.clients jedis 2.0.0 jar org.springframework spring-core ${spring.version} org.springframework spring-context ${spring.version} org.apache.maven.plugins maven-compiler-plugin 2.3.2 1.6 1.6 Moving on to configuring Spring context, let's understand what we need to have in order for a publisher to publish some messages and for a consumer to consume them. Knowing the respective Spring abstractions for JMS will help a lot with that. we need connection factory -> JedisConnectionFactory we need a template for publisher to publish messages -> RedisTemplate we need a message listener for consumer to consume messages -> RedisMessageListenerContainer Using Spring Java configuration, let's describe our context: package com.example.redis.config; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.data.redis.connection.jedis.JedisConnectionFactory; import org.springframework.data.redis.core.RedisTemplate; import org.springframework.data.redis.listener.ChannelTopic; import org.springframework.data.redis.listener.RedisMessageListenerContainer; import org.springframework.data.redis.listener.adapter.MessageListenerAdapter; import org.springframework.data.redis.serializer.GenericToStringSerializer; import org.springframework.data.redis.serializer.StringRedisSerializer; import org.springframework.scheduling.annotation.EnableScheduling; import com.example.redis.IRedisPublisher; import com.example.redis.impl.RedisMessageListener; import com.example.redis.impl.RedisPublisherImpl; @Configuration @EnableScheduling public class AppConfig { @Bean JedisConnectionFactory jedisConnectionFactory() { return new JedisConnectionFactory(); } @Bean RedisTemplate< String, Object > redisTemplate() { final RedisTemplate< String, Object > template = new RedisTemplate< String, Object >(); template.setConnectionFactory( jedisConnectionFactory() ); template.setKeySerializer( new StringRedisSerializer() ); template.setHashValueSerializer( new GenericToStringSerializer< Object >( Object.class ) ); template.setValueSerializer( new GenericToStringSerializer< Object >( Object.class ) ); return template; } @Bean MessageListenerAdapter messageListener() { return new MessageListenerAdapter( new RedisMessageListener() ); } @Bean RedisMessageListenerContainer redisContainer() { final RedisMessageListenerContainer container = new RedisMessageListenerContainer(); container.setConnectionFactory( jedisConnectionFactory() ); container.addMessageListener( messageListener(), topic() ); return container; } @Bean IRedisPublisher redisPublisher() { return new RedisPublisherImpl( redisTemplate(), topic() ); } @Bean ChannelTopic topic() { return new ChannelTopic( "pubsub:queue" ); } } Very easy and straightforward. The presence of @EnableScheduling annotation is not necessary and is required only for our publisher implementation: the publisher will publish a string message every 100 ms. package com.example.redis.impl; import java.util.concurrent.atomic.AtomicLong; import org.springframework.data.redis.core.RedisTemplate; import org.springframework.data.redis.listener.ChannelTopic; import org.springframework.scheduling.annotation.Scheduled; import com.example.redis.IRedisPublisher; public class RedisPublisherImpl implements IRedisPublisher { private final RedisTemplate< String, Object > template; private final ChannelTopic topic; private final AtomicLong counter = new AtomicLong( 0 ); public RedisPublisherImpl( final RedisTemplate< String, Object > template, final ChannelTopic topic ) { this.template = template; this.topic = topic; } @Scheduled( fixedDelay = 100 ) public void publish() { template.convertAndSend( topic.getTopic(), "Message " + counter.incrementAndGet() + ", " + Thread.currentThread().getName() ); } } And finally our message listener implementation (which just prints message on a console). package com.example.redis.impl; import org.springframework.data.redis.connection.Message; import org.springframework.data.redis.connection.MessageListener; public class RedisMessageListener implements MessageListener { @Override public void onMessage( final Message message, final byte[] pattern ) { System.out.println( "Message received: " + message.toString() ); } } Awesome, just two small classes, one configuration to wire things together and we have full pub/sub messaging support in our application! Let's run the application as standalone ... package com.example.redis; import org.springframework.context.ApplicationContext; import org.springframework.context.annotation.AnnotationConfigApplicationContext; import com.example.redis.config.AppConfig; public class RedisPubSubStarter { public static void main(String[] args) { new AnnotationConfigApplicationContext( AppConfig.class ); } } ... and see following output in a console: ... Message received: Message 1, pool-1-thread-1 Message received: Message 2, pool-1-thread-1 Message received: Message 3, pool-1-thread-1 Message received: Message 4, pool-1-thread-1 Message received: Message 5, pool-1-thread-1 Message received: Message 6, pool-1-thread-1 Message received: Message 7, pool-1-thread-1 Message received: Message 8, pool-1-thread-1 Message received: Message 9, pool-1-thread-1 Message received: Message 10, pool-1-thread-1 Message received: Message 11, pool-1-thread-1 Message received: Message 12, pool-1-thread-1 Message received: Message 13, pool-1-thread-1 Message received: Message 14, pool-1-thread-1 Message received: Message 15, pool-1-thread-1 Message received: Message 16, pool-1-thread-1 ... Great! There is much more which you could do with Redis pub/sub, excellent documentation is available for you on Redis official web site.

Curator's note: This article originally appeared at WindowsAzure.com. To use this feature and other new Windows Azure capabilities, sign up for the free preview. Just as you can quickly create and deploy a web application created from the gallery, you can also deploy a website created on a workstation with traditional developer tools from Microsoft or other companies. Table of Contents Deployment Options How to: Create a Website Using the Management Portal How to: Create a Website from the Gallery How to: Delete a Website Next Steps Deployment Options Windows Azure supports deploying websites from remote computers using WebDeploy, FTP, GIT or TFS. Many development tools provide integrated support for publication using one or more of these methods and may only require that you provide the necessary credentials, site URL and hostname or URL for your chosen deployment method. Credentials and deployment URLs for all enabled deployment methods are stored in the website's publish profile, a file which can be downloaded in the Windows Azure (Preview) Management Portal from the Quick Start page or the quick glance section of the Dashboard page. If you prefer to deploy your website with a separate client application, high quality open source GIT and FTP clients are available for download on the Internet for this purpose. How to: Create a Website Using the Management Portal Follow these steps to create a website in Windows Azure. Login to the Windows Azure (Preview) Management Portal. Click the Create New icon on the bottom left of the Management Portal. Click the Web Site icon, click the Quick Create icon, enter a value for URL and then click the check mark next to create web site on the bottom right corner of the page. When the website has been created you will see the text Creation of Web Site '[SITENAME]' Completed. Click the name of the website displayed in the list of websites to open the website's Quick Start management page. On the Quick Start page you are provided with options to set up TFS or GIT publishing if you would like to deploy your finished website to Windows Azure using these methods. FTP publishing is set up by default for websites and the FTP Host name is displayed under FTP Hostname on the Quick Start and Dashboard pages. Before publishing with FTP or GIT choose the option to Reset deployment credentials on the Dashboard page. Then specify the new credentials (username and password) to authenticate against the FTP Host or the Git Repository when deploying content to the website. The Configure management page exposes several configurable application settings in the following sections: Framework: Set the version of .NET framework or PHP required by your web application. Diagnostics: Set logging options for gathering diagnostic information for your website in this section. App Settings: Specify name/value pairs that will be loaded by your web application on start up. For .NET sites, these settings will be injected into your .NET configuration AppSettings at runtime, overriding existing settings. For PHP and Node sites these settings will be available as environment variables at runtime. Connection Strings: View connection strings for linked resources. For .NET sites, these connection strings will be injected into your .NET configuration connectionStrings settings at runtime, overriding existing entries where the key equals the linked database name. For PHP and Node sites these settings will be available as environment variables at runtime. Default Documents: Add your web application's default document to this list if it is not already in the list. If your web application contains more than one of the files in the list then make sure your website's default document appears at the top of the list. How to: Create a Website from the Gallery The gallery makes available a wide range of popular web applications developed by Microsoft, third party companies, and open source software initiatives. Web applications created from the gallery do not require installation of any software other than the browser used to connect to the Windows Azure Management Portal. In this tutorial, you'll learn: How to create a new site through the gallery. How to deploy the site through the Windows Azure Portal. You'll build a Word press blog that uses a default template. The following illustration shows the completed application: Note To complete this tutorial, you need a Windows Azure account that has the Windows Azure Web Sites feature enabled. You can create a free trial account and enable preview features in just a couple of minutes. For details, see Create a Windows Azure account and enable preview features. Create a web site in the portal Login to the Windows Azure Management Portal. Click the New icon on the bottom left of the dashboard. Click the Web Site icon, and click From Gallery. Locate and click the WordPress icon in list, and then click Next. On the Configure Your App page, enter or select values for all fields: Enter a URL name of your choice Leave Create a new MySQL database selected in the Database field Select the region closest to you Then click Next. On the Create New Database page, you can specify a name for your new MySQL database or use the default name. Select the region closest to you as the hosting location. Select the box at the bottom of the screen to agree to ClearDB's usage terms for your hosted MySQL database. Then click the check to complete the site creation. After you click Complete Windows Azure will initiate build and deploy operations. While the web site is being built and deployed the status of these operations is displayed at the bottom of the Web Sites page. After all operations are performed, A final status message when the site has been successfully deployed. Launch and manage your WordPress site Click on your new site from the Web Sites page to open the dashboard for the site. On the Dashboard management page, scroll down and click the link on the left under Site Url to open the site’s welcome page. Enter appropriate configuration information required by WordPress and click Install WordPress to finalize configuration and open the web site’s login page. Login to the new WordPress web site by entering the username and password that you specified on the Welcome page. You'll have a new WordPress site that looks similar to the site below. How to: Delete a Website Websites are deleted using the Delete icon in the Windows Azure Management Portal. The Delete icon is available in the Windows Azure Portal when you click Web Sites to list all of your websites and at the bottom of each of the website management pages. Next Steps For more information about Websites, see the following: Walkthrough: Troubleshooting a Website on Windows Azure

SOAP headers can be added to a Web service request in different ways, if you use Apache CXF. The way I prefer is the one I’ve mentioned here – as it doesn’t require changes to wsdl or method signatures and it’s much faster as it doesn’t break streaming and the memory overhead is less. The headers in the list are streamed at the appropriate time to the wire according to the databinding object found in the Header object. About SOAP headers, Like any good messaging protocol, SOAP defines the concept of a message header. It is an optional part of any SOAP message. If it exists, the header contains application-specific information (like authentication, payment, etc) about the SOAP message i.e. information about the message, or about the context in which the message is sent, or basically whatever the creator of the message thought was a good idea to put there instead of the actual body of the message. If the Header element is present, it must be the first child element of the Envelope element. /** * @author Singaram Subramanian * */ /* Create a ClientProxyFactoryBean reference and assign it an instance of JaxWsProxyFactoryBean, a factory for creating JAX-WS proxies. This class provides access to the internal properties used to set-up proxies. Using it provides more control than the standard JAX-WS APIs. */ ClientProxyFactoryBean factory = new JaxWsProxyFactoryBean(); factory.setServiceClass(singz.ws.cxf.sample.SampleServiceInterface.class); // Set the web service endpoint URL here factory.setAddress("http://xxx.xxx.com/services/SampleService/v1"); SampleServiceInterface serviceClient = (SampleServiceInterface) factory.create(); // Get the underlying Client object from the proxy object of service interface Client proxy = ClientProxy.getClient(serviceClient); // Creating SOAP headers to the web service request // Create a list for holding all SOAP headers List headersList = new ArrayList(); Header testSoapHeader1 = new Header(new QName("uri:singz.ws.sample", "soapheader1"), "SOAP Header Message 1", new JAXBDataBinding(String.class)); Header testSoapHeader2 = new Header(new QName("uri:singz.ws.sample", "soapheader2"), "SOAP Header Message 2", new JAXBDataBinding(String.class)); headersList.add(testSoapHeader1); headersList.add(testSoapHeader2); // Add SOAP headers to the web service request proxy.getRequestContext().put(Header.HEADER_LIST, headersList); More on this @ http://cxf.apache.org/faq.html#FAQ-HowcanIaddsoapheaderstotherequest%2Fresponse%3F

For my own interest I started exploring Spring REST Services with GWT. It took some time to figure it out and then I came accross RestyGWT. With the help of RestyGWT, I managed to integrate GWT with Spring REST Services. My idea was to keep the GWT way of creating the service and serviceAsync and yet make a REST call. Below are the steps which will help achieve above. My preffered development environment is Eclipse, so as a prerequisite you must have Eclipse with Maven support installed. Lets Begin, Create a Maven project, goto File-->New-->Other. In the wizard type "Maven". Select Maven Project and click on Next. In the "Select project name and location" page of the wizard, make sure that "Create a simple project (skip archetype selection)" option is checked, hit "Next" to continue with default values. In the "Enter group id for the artifact" page of the wizard, enter values for group id and artifactid. Select the packaging as "war", hit "Finish" to exit the wizard and to create your project. Modify the POM file to add required dependencies as below 4.0.0 com.sagar.restgwt RestGWT 0.0.1-SNAPSHOT war com.google.gwt gwt-servlet ${gwt.version} runtime com.google.gwt gwt-user ${gwt.version} provided org.fusesource.restygwt restygwt 1.2 javax.ws.rs jsr311-api 1.1 provided org.codehaus.jackson jackson-mapper-asl 1.4.1 org.springframework spring-core ${org.springframework.version} org.springframework spring-web ${org.springframework.version} org.springframework spring-webmvc ${org.springframework.version} . 2.5.0-rc1 1.6 3.1.1.RELEASE UTF-8 ${project.build.directory}\${project.build.finalName} restgwt ${webappDirectory}/WEB-INF/classes org.codehaus.mojo gwt-maven-plugin 2.5.0-rc1 compile test generateAsync ${webappDirectory} org.apache.maven.plugins maven-compiler-plugin ${java-version} ${java-version} true "Update Project Configuration" by Right clicking on your project-->Maven. The below steps assumes that your are aware of the GWT project structure. Create your GWT module. This can be done by installing GWT plugin for Eclipse. Once the GWT module is ready update your .gwt.xml with the below given content. In the client package create the service to make REST Call. With this approach we dont have to create the ServiceAsync interface. We will be creating our service interface by extending the "RestService", provided by RestyGWT. Code Snippet: InfoService.Java [ A service interface to make REST call. ] @Path("/service") public interface InfoService extends RestService { public static class Util { private static InfoService instance; public static InfoService getService() { if (instance == null) { instance = GWT.create(InfoService.class); } Resource resource = new Resource(GWT.getModuleBaseURL() + "service"); ((RestServiceProxy) instance).setResource(resource); return instance; } } @GET @Path("/loadInfo") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.APPLICATION_JSON) public void getInfo(MethodCallback callback); } OrderConfirmation.java [ Model which will be returned as a response. ] public class OrderConfirmation { public String message; public Long ready_time; /** * Example of how to create an instance of a JsonEncoderDecoder for a data * transfer object. */ public interface OrderConfirmationJED extends JsonEncoderDecoder { } @Override public String toString() { if (GWT.isClient()) { OrderConfirmationJED jed = GWT.create(OrderConfirmationJED.class); return jed.encode(this).toString(); } return super.toString(); } } RestGWT.java [ GWT module entrypoint to see things running. ] public class RestGWT implements EntryPoint { public void onModuleLoad() { Button button = new Button("Click Me"); button.addClickHandler(new ClickHandler() { @Override public void onClick(ClickEvent event) { InfoService.Util.getService().getInfo(new MethodCallback() { @Override public void onSuccess(Method method, OrderConfirmation response) { RootPanel.get().add(new Label(response.toString())); } @Override public void onFailure(Method method, Throwable exception) { GWT.log("Error"); } }); } }); RootPanel.get().add(button); } } Create the Spring managed controller as below, this should be presnt in the "server" package as per GWT project structure. Also you can notice that we don't have to implement our service interface. RestGWTController.java [ Spring managed controller. ] @Controller public class RestGWTController { @RequestMapping(value = "/loadInfo", method = RequestMethod.GET, headers = "Accept=application/json") public @ResponseBody OrderConfirmation handleRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { GreetingServiceEndpoint endpoint = greetingService.getGreetingServiceEndpointPort(); OrderConfirmation confirmation = new OrderConfirmation(); confirmation.message = endpoint.sayHello(); confirmation.ready_time = System.currentTimeMillis() + 1000 * 60 * 30; return confirmation; } } Settings required for Spring src/main/webapp/WEB-INF/web.xml Rest GWT This is web-project for RestGWT contextConfigLocation /WEB-INF/applicationContext.xml org.springframework.web.context.ContextLoaderListener Spring MVC Dispatcher Servlet org.springframework.web.servlet.DispatcherServlet contextConfigLocation /WEB-INF/classes/action-servlet.xml 1 Spring MVC Dispatcher Servlet /restgwt/service/* RestGWT.html src/main/webapp/WEB-INF/applicationCotext.xml src/main/webapp/WEB-INF/classes/action-servlet.xml Now all configuration is completed. To build the application right click on your project → Run As → Maven Install. This will create the war file in your project's target/restgwt folder. To test this approach we are going to deploy our web application to an Apache Tomcat 7 server. To launch the application point your browser to the following address http://localhost:8080/restgwt/ Enjoy Coding Nilabh

Explore SQL query optimization and normalization.

this simple cheat sheet contains all the key goals, principals and design patterns that you should be aware of when designing soa services and contains helpful links to places where you can find more in-depth information on each topic. when i was studying for my soa certified architect exams, i kept notes on all the best bits from the course material. after 9 months and several hundred hours of study, i found that there were certain key pieces of information that i kept referring back to time and time again, such as… how do you define service-orientation? what are the goals and strategic benefits of having a service-oriented business? what are the design principals you should apply to soa service design & soa governance? what are the characteristics of soa based businesses – how can you recognise one? what are the most useful soa design patterns and how are they grouped? i thought it might be useful to bring all this information together into one place, so in collaboration with soagrowers we have published a free pdf cheat sheet on soa service design which you can print out and keep close to hand so it’s there whenever you need it. it’s not meant to be an exhaustive guide – it’s just a set of place-holders to remind you of the topics that may be of relevance to you when designing services. however, it should prove useful to any service architect or developer who’s interested in service design or anyone who is going through the same certification programme as i did – even if you just use it as a check-list or aide-mémoire . none of it is particularly technology specific. the same set of goals, principals and patterns can be applied equally to soap based web services , restful services or any other kind of distributed components – that’s the beauty of service-orientation, it’s vendor and technology neutral. in the sheet i’ve also highlighted something that often get’s overlooked when technologists have the lead on soa implementations:- soa has some very attractive and unique business benefits that can only be fully realised when you apply the design paradigm correctly. for my money, it’s this outcome oriented viewpoint (the business case if you like) that really differentiates soa from other tactics like eai/esb, but all too often this message gets lost in the melee . we hope you find it useful. to get your copy of the soa service design cheat sheet, just click on the image below. if you like it please share it (there are handy share buttons on the page below). click on the image to download the pdf get involved. did you find this useful? is there something you think could be added or removed? did you notice how esb is just a small fraction of the bigger picture? let me know your thoughts in the comments below. ————————————————————————— updated: 18/09/2012. i’ve now added a small section on contract first service design, just because it so fundamentally underpins many of the most important goals, principals and patterns used to deliver successful soa. for more information on contract first, see spring-ws’s excellent whitepaper . contract-first isn’t just a soap thing by the way. ‘contract’ in a soa design context means operations, data types, policies and anything else to do with the service’s public facia. so although rest has an implicit contract with predetermined operations (get, put, post, etc.) it still has data type’s and flexible url’s that convey some meaning. therefore, if you want to make a rest architecture more interoperable and less brittle for clients, it helps to plan these datatypes and url’s in advance if you can so they become better standardised and therefore more reusable.

For a given WSDL, there are several different ways to generate Java web service code (CXF, Axis2, etc..). And depending on certain settings within the WSDL file and settings used by the relevant build tool, there are different ways of exposing those services described in the WSDL. This post will briefly document the generating of Java code for a WSDL using Maven and the jaxws wsimport plugin. It will also show the difference in the services exposed when using wrapped and non-wrapped bindings. Below is an extract from a pom.xml to generate the Java code: org.codehaus.mojo jaxws-maven-plugin 1.10 wsimport City81SOAPService.wsdl ${basedir}/src/wsdl generate-sources generate-sources ......... ${project.build.directory}/generated-sources/jaxws-wsimport true true true ${basedir}/src/jax-ws-catalog.xml For the below WSDL file, the wsimport plugin will generate the following classes: com\city81\soap\Balance.java com\city81\soap\City81SOAP.java com\city81\soap\City81SOAPImplService.java com\city81\soap\CreateCustomer.java com\city81\soap\CreateCustomerResponse.java com\city81\soap\CreateCustomerResponseType.java com\city81\soap\CreateStatus.java com\city81\soap\ObjectFactory.java com\city81\soap\package-info.java For the above settings, the generated City81SOAP class will be as below: @WebService(name = "City81SOAP", targetNamespace = "http://soap.city81.com/") @SOAPBinding(parameterStyle = SOAPBinding.ParameterStyle.BARE) @XmlSeeAlso({ ObjectFactory.class }) public interface City81SOAP { @WebMethod(action = "http://soap.city81.com/createCustomer") @WebResult(name = "createCustomerResponse", targetNamespace = "http://soap.city81.com/", partName = "params") public CreateCustomerResponse createCustomer(@WebParam(name = "createCustomer", targetNamespace = "http://soap.city81.com/", partName = "params") CreateCustomer params); } The binding style as can be seen from the @SOAPBinding annotation at the head of the class is BARE ie non-wrapped. The method's args and return parameters are in each case represented as a single Java object. CreateCustomer and CreateCustomerResponse. This has happened because in the pom.xml file, there is a bindingDirectory tag which points to a folder containing a binding.xml file. This file, shown below, has an enableWrapperStyle tag and the boolean value of false. false If the boolean was true, or if there was no bindingDirectory tag in the pom.xml file, then the default SOAP binding style would be used ie WRAPPED. This would then result in the below generated City81SOAP class: @WebService(name = "City81SOAP", targetNamespace = "http://soap.city81.com/") @XmlSeeAlso({ ObjectFactory.class }) public interface City81SOAP { @WebMethod(action = "http://soap.city81.com/createCustomer") @RequestWrapper(localName = "createCustomer", targetNamespace = "http://soap.city81.com/", className = "com.city81.soap.CreateCustomer") @ResponseWrapper(localName = "createCustomerResponse", targetNamespace = "http://soap.city81.com/", className = "com.city81.soap.CreateCustomerResponse") public void createCustomer( @WebParam(name = "surname", targetNamespace = "") String surname, @WebParam(name = "firstName", targetNamespace = "") String firstName, @WebParam(name = "balance", targetNamespace = "") Balance balance, @WebParam(name = "customerId", targetNamespace = "", mode = WebParam.Mode.OUT) Holder customerId, @WebParam(name = "status", targetNamespace = "", mode = WebParam.Mode.OUT) Holder status); } The method's args are now individual Java objects and the return parameters are each represented as Holder objects with a WebParam.Mode.OUT value denoting they are return objects. This means that return objects are set as opposed to actually being returned in the method's signature. Another way to specify bindings other than using the binding.xml file is to embed the enableWrapperStyle as a child of the portType but if a WSDL is from a third party, then having to change it every time a new version of the WSDL is released is open to errors. false ... Back to the generated interfaces, and these of course need to be implemented. For an interface with a binding type of BARE, the implemented class would look like below: @WebService(targetNamespace = "http://soap.city81.com/", name = "City81SOAP", portName = "City81SOAPImplPort", serviceName = "City81SOAPImplService") @SOAPBinding(style = SOAPBinding.Style.DOCUMENT, use = SOAPBinding.Use.LITERAL, parameterStyle = SOAPBinding.ParameterStyle.BARE) public class City81SOAPImpl implements City81SOAP { @Override public CreateCustomerResponse createCustomer(CreateCustomer createCustomer) { CreateCustomerResponse createCustomerResponse = new CreateCustomerResponse(); ..... return createCustomerResponse; } } In the case of WRAPPED binding style, the SOAPBinding annotation would include parameterStyle = SOAPBinding.ParameterStyle.WRAPPED and the createCustomer method would be as below: public void createCustomer( String surname, String firstName, Balance balance, Holder customerId, Holder status) { customerId= new Holder("1"); status = new Holder(CreateStatus.CREATE_PENDING); } This post shows that there are different ways to ultimately achieve the same result.