A couple years ago I started a series of articles on XML parsing. I covered lxml’s etree and Python builtin minidom XML parsing library. For whatever reason I didn’t notice lxml’s objectify sub-package, but I saw it recently and decided to check it out. In my mind, the objectify module seems to be even more “Pythonic” than etree is. Let’s take some time and go over my old XML examples using objectify and see how it’s different! Let’s Get This Party Started! If you haven’t already, go out and download lxml or you won’t be able to follow along very well. Once you have it, we can continue. We’ll be using the following piece of XML for our parsing pleasure: 1181251680 040000008200E000 1181572063 1800 Bring pizza home 1234360800 1800 Check MS Office website for updates 604f4792-eb89-478b-a14f-dd34d3cc6c21-1234360800 dismissed Now we need to write some code that can parse and modify the XML. Let’s take a look at this little demo that shows a bunch of the neat abilities that objectify provides. from lxml import etree, objectify #---------------------------------------------------------------------- def parseXML(xmlFile): """""" with open(xmlFile) as f: xml = f.read() root = objectify.fromstring(xml) # returns attributes in element node as dict attrib = root.attrib # how to extract element data begin = root.appointment.begin uid = root.appointment.uid # loop over elements and print their tags and text for e in root.appointment.iterchildren(): print "%s => %s" % (e.tag, e.text) # how to change an element's text root.appointment.begin = "something else" print root.appointment.begin # how to add a new element root.appointment.new_element = "new data" # print the xml obj_xml = etree.tostring(root, pretty_print=True) print obj_xml # remove the py:pytype stuff #objectify.deannotate(root) etree.cleanup_namespaces(root) obj_xml = etree.tostring(root, pretty_print=True) print obj_xml # save your xml with open("new.xml", "w") as f: f.write(obj_xml) #---------------------------------------------------------------------- if __name__ == "__main__": f = r'path\to\sample.xml' parseXML(f) The code is pretty well commented, but we’ll spend a little time going over it anyway. First we pass it our sample XML file and objectify it. If you want to get access to a tag’s attributes, use the attrib property. It will return a dictionary of the attribute’s of the tag. To get to sub-tag elements, you just use dot notation. As you can see, to get to the begin tag’s value, we can just do something like this: begin = root.appointment.begin If you need to iterate over the children elements, you can use iterchildren. You may have to use a nested for loop structure to get everything. Changing an element’s value is as simple as just assigning it a new value. And if you need to create a new element, just add a period and the name of the new element (see below): root.appointment.new_element = "new data" When we add or change items using objectify, it will add some annotations to the XML, such as xmlns:py="http://codespeak.net/lxml/objectify/pytype" py:pytype="str". You may not want that included, so you'll have to call the following method to remove that stuff: [python] etree.cleanup_namespaces(root) You can also use “objectify.deannotate(root)” to do some deannotation chores, but I wasn’t able to get it to work for this example. To save the new XML, you actually seem to need lxml’s etree module to convert it to a string so you can save it. At this point, you should be able to parse most XML documents and edit them effectively with lxml’s objectify. I thought it was very intuitive and easy to pick up. Hopefully you will find it useful in your endeavors as well.

After my article Connection Pooling in a Java Web Application with Glassfish and NetBeans IDE, here are the instructions for Tomcat. Requirements NetBeans IDE (this tutorial uses NetBeans 7) Tomcat (this tutorial uses Tomcat 7 that is bundled within NetBeans) MySQL database MySQL Java Driver Steps Assuming your MySQL database is ready, connect to it and create a database. Lets call it connpool: mysql> create database connpool; Now we create and populate the table from which we will fetch the data: mysql> use connpool; mysql> create table data(id int(5) not null unique auto_increment, name varchar(255) not null); mysql> insert into data(name) values("Fred Flintstone"), ("Pink Panther"), ("Wayne Cramp"), ("Johnny Bravo"), ("Spongebob Squarepants"); That is it for the database part. We now create our web application. In NetBeans IDE, click File → New Project... Select Java Web → Web Application: Click Next and give the project the name TomPool. Click Next Choose the server as Tomcat and, since we are not going to use any frameworks, click Finish. The project will be created and the start page, index.jsp, opened for us in the IDE. Now we create the connection pooling parameters. In the Projects window, expand configuration files and open "context.xml". You will see that the IDE has added this code for us: Delete the last line: and then add the following to the context.xml file. I have explained the sections along the way. Make sure you edit your MySQL username and password appropriately: Next, expand the Web Pages node, right-click WEB-INF → New → Other → XML → XML Document. Click Next and type web for the File Name. Click next and choose Well-Formed Document then Finish. You will now have the file "web.xml": Delete everything in the file and paste this code: MySQL Test App DB Connection connpool javax.sql.DataSource Container That is it for the connection pool. We now edit our code to make use of it. Edit index.jsp by adding this code just after the initial coments but before Edit the section of the page: Data in my Connection Pooled Database Now, we test the connection pool by running the application: If you want to have the one connection pool used in multiple applications, you need to edit the following two files: 1. /conf/web.xml Just before the closing tag, add the code DB Connection connpool javax.sql.DataSource Container 2. /conf/context.xml Just before the closing tag, add the code Now you can use the pool without editing XML files in each of your applications. Just use the sample code as given in index.jsp That's it folks!

Some time ago I came across What's the purpose of AsyncContext.start(...) in Servlet 3.0? question. Quoting the Javadoc of aforementioned method: Causes the container to dispatch a thread, possibly from a managed thread pool, to run the specified Runnable. To remind all of you, AsyncContext is a standard way defined in Servlet 3.0 specification to handle HTTP requests asynchronously. Basically HTTP request is no longer tied to an HTTP thread, allowing us to handle it later, possibly using fewer threads. It turned out that the specification provides an API to handle asynchronous threads in a different thread pool out of the box. First we will see how this feature is completely broken and useless in Tomcat and Jetty - and then we will discuss why the usefulness of it is questionable in general. Our test servlet will simply sleep for given amount of time. This is a scalability killer in normal circumstances because even though sleeping servlet is not consuming CPU, but sleeping HTTP thread tied to that particular request consumes memory - and no other incoming request can use that thread. In our test setup I limited the number of HTTP worker threads to 10 which means only 10 concurrent requests are completely blocking the application (it is unresponsive from the outside) even though the application itself is almost completely idle. So clearly sleeping is an enemy of scalability. @WebServlet(urlPatterns = Array("/*")) class SlowServlet extends HttpServlet with Logging { protected override def doGet(req: HttpServletRequest, resp: HttpServletResponse) { logger.info("Request received") val sleepParam = Option(req.getParameter("sleep")) map {_.toLong} TimeUnit.MILLISECONDS.sleep(sleepParam getOrElse 10) logger.info("Request done") } } Benchmarking this code reveals that the average response times are close to sleep parameter as long as the number of concurrent connections is below the number of HTTP threads. Unsurprisingly the response times begin to grow the moment we exceed the HTTP threads count. Eleventh connection has to wait for any other request to finish and release worker thread. When the concurrency level exceeds 100, Tomcat begins to drop connections - too many clients are already queued. So what about the the fancy AsyncContext.start() method (do not confuse with ServletRequest.startAsync())? According to the JavaDoc I can submit any Runnable and the container will use some managed thread pool to handle it. This will help partially as I no longer block HTTP worker threads (but still another thread somewhere in the servlet container is used). Quickly switching to asynchronous servlet: @WebServlet(urlPatterns = Array("/*"), asyncSupported = true) class SlowServlet extends HttpServlet with Logging { protected override def doGet(req: HttpServletRequest, resp: HttpServletResponse) { logger.info("Request received") val asyncContext = req.startAsync() asyncContext.setTimeout(TimeUnit.MINUTES.toMillis(10)) asyncContext.start(new Runnable() { def run() { logger.info("Handling request") val sleepParam = Option(req.getParameter("sleep")) map {_.toLong} TimeUnit.MILLISECONDS.sleep(sleepParam getOrElse 10) logger.info("Request done") asyncContext.complete() } }) } } We are first enabling the asynchronous processing and then simply moving sleep() into a Runnable and hopefully a different thread pool, releasing the HTTP thread pool. Quick stress test reveals slightly unexpected results (here: response times vs. number of concurrent connections): Guess what, the response times are exactly the same as with no asynchronous support at all (!) After closer examination I discovered that when AsyncContext.start() is called Tomcat submits given task back to... HTTP worker thread pool, the same one that is used for all HTTP requests! This basically means that we have released one HTTP thread just to utilize another one milliseconds later (maybe even the same one). There is absolutely no benefit of calling AsyncContext.start() in Tomcat. I have no idea whether this is a bug or a feature. On one hand this is clearly not what the API designers intended. The servlet container was suppose to manage separate, independent thread pool so that HTTP worker thread pool is still usable. I mean, the whole point of asynchronous processing is to escape the HTTP pool. Tomcat pretends to delegate our work to another thread, while it still uses the original worker thread pool. So why I consider this to be a feature? Because Jetty is "broken" in exactly same way... No matter whether this works as designed or is only a poor API implementation, using AsyncContext.start() in Tomcat and Jetty is pointless and only unnecessarily complicates the code. It won't give you anything, the application works exactly the same under high load as if there was no asynchronous logic at all. But what about using this API feature on correct implementations like IBM WAS? It is better, but still the API as is doesn't give us much in terms of scalability. To explain again: the whole point of asynchronous processing is the ability to decouple HTTP request from an underlying thread, preferably by handling several connections using the same thread. AsyncContext.start() will run the provided Runnable in a separate thread pool. Your application is still responsive and can handle ordinary requests while long-running request that you decided to handle asynchronously are processed in a separate thread pool. It is better, unfortunately the thread pool and thread per connection idiom is still a bottle-neck. For the JVM it doesn't matter what type of threads are started - they still occupy memory. So we are no longer blocking HTTP worker threads, but our application is not more scalable in terms of concurrent long-running tasks we can support. In this simple and unrealistic example with sleeping servlet we can actually support thousand of concurrent (waiting) connections using Servlet 3.0 asynchronous support with only one extra thread - and without AsyncContext.start(). Do you know how? Hint: ScheduledExecutorService. Postscriptum: Scala goodness I almost forgot. Even though examples were written in Scala, I haven't used any cool language features yet. Here is one: implicit conversions. Make this available in your scope: implicit def blockToRunnable[T](block: => T) = new Runnable { def run() { block } } And suddenly you can use code block instead of instantiating Runnable manually and explicitly: asyncContext start { logger.info("Handling request") val sleepParam = Option(req.getParameter("sleep")) map { _.toLong} TimeUnit.MILLISECONDS.sleep(sleepParam getOrElse 10) logger.info("Request done") asyncContext.complete() } Sweet!

In my last two blog posts, I showed examples of using Selenium WebDriver to capture screenshots, and running in a headless (no X-server) mode. This example combines the two solutions to capture screenshots inside a virtual display. To achieve this, I use a combination of Selenium WebDriver and pyvirtualdisplay (which uses xvfb) to run a browser in a virtual display and capture screenshots. the setup you need is: Selenium 2 Python bindings: PyPI pyvirtualdisplay Python package (depends on xvfb): PyPI On Debian/Ubuntu Linux systems, you can install everything with: $ sudo apt-get install python-pip xvfb xserver-xephyr $ sudo pip install selenium once you have it setup, the following code example should work: #!/usr/bin/env python from pyvirtualdisplay import Display from selenium import webdriver display = Display(visible=0, size=(800, 600)) display.start() browser = webdriver.Firefox() browser.get('http://www.google.com') browser.save_screenshot('screenie.png') browser.quit() display.stop() this will: launch a virtual display launch Firefox browser inside the virtual display navigate to google.com capture and save a screenshot close the browser stop the virtual display

Introduction In recent times, many programming languages that run on JVM have emerged. Many of these languages support the concept of writing code in a functional style. Programmers have started realizing the benefits of functional programming and are beginning to rediscover the powerful style of this programming paradigm. The emergence of multiple languages on JVM have only helped to reignite the strong interest in this paradigm. Java at its core is an imperative programming language. However in recent past many new languages like Scala, Clojure, Groovy etc. have become popular which supports functional programming style and yet run on JVM. However none of these languages can be considered as pure functional language since all of them allow Java code to be called from within them and Java on its own is not a functional language. Still they have different degree of support for writing code in functional style and have their own benefits. Functional programming requires different kind of thinking and has its own advantages as compared to imperative programming. It seems that Java has also realized functional programming advantages and is slowly inching towards it. First sign of this can be seen in the form of Lambda Expressions that will be supported in Java 8. Although it's too early to comment on this as the draft for Java 8 is still under review and is expected to be released next year, but it does show that Java has plans of supporting functional programming style going forward. In this article we will first discuss what functional programming is and how it is different from imperative programming. Later we will see where does each of the above mentioned Java based programming languages i.e. Scala, Clojure and Groovy fits in the world of functional programming and what each of them has to offer. And at the last we will sneak-peak into Java 8's lambda expressions. Why Functional Programming? Computers of current era are shipped with multicore processors. Going forward the number of processors in a machine is only going to increase. The code we write today and tomorrow will probably never run on a single processor system. In order to get best out of this, software must be designed to make more and more use of concurrency and hence keep all available processors busy. Java does provide concurrency concepts like threads, synchronization, locks etc. to execute code in parallel. But shared memory multi-threading approach in Java causes more trouble than solving the problem. Java based functional programming languages like Scala, Clojure, Groovy etc. looks into these problems with a different angle and provides less complex and less error-prone solutions as compared to imperative programming. They provide immutability concepts out of the box and hence eliminate need of synchronization and associated risk of deadlocks or livelocks. Concepts like Actors, Agents and DataFlow variables provide high level concurrency abstraction and makes very easy to write concurrent programs. What is Functional Programming? Functional Programming is a concept which treats functions as first class citizens. At the core of functional programming is immutability. It emphasizes on application of functions in contrast to imperative programming style which emphasizes on change in state. Functional programming has no side effects whereas programming in imperative style can result in side-effects. Let's elaborate more on each of these characteristics to understand the concept behind functional programming. Immutable state - The state of an object doesn't change and hence need not be protected or synchronized. That might sound a bit awkward at first, since if nothing changes, one might think that we are not writing a useful program. However that's not what immutable state means. In functional programming, change in state occurs via series of transformations which keeps the object immutable and yet achieves change in state. Functions as first class citizens - There was a major shift in the way programs were written when Object oriented concepts came into picture. Everything was conceptualized as object and any action to be performed was treated as method call on objects. Hence there is a series of method calls executed on objects to get the desired work done. In functional programming world, it's more about thinking in terms of communication chain between functions than method calls on objects. This makes functions as first class citizens of functional programming since everything is modelled around functions. Higher-order functions - Functions in functional programming are higher order functions since following actions can be performed with them. 1. Functions can be passed within functions as arguments. 2. Functions can be created within functions just as objects can be created in functions 3. Functions can be returned from functions Functions with no side-effects - In functional programming, function execution has no side-effects. In other words a function code will always return same result for same argument when called multiple times. It doesn't change anything outside its boundaries and is also not affected by any external change outside it's boundary. It doesn't change input value and can only produce new output. However once the output has been produced and returned by function, it also becomes immutable and cannot be modified by any other function. In other words, they support referential transparency i.e. if a function takes an input and returns some output, multiple invocation of that function at different point of time will always return same output as long as input remains same. This is one of the main motivations behind using functional language as it makes easy to understand and predict behaviour of program. Characteristics like immutability and no side-effects are extremely helpful while writing multi-threaded code and developers need not to worry for synchronizing the state. Hence functional code is very easy to distribute across multiple cores as they don't have any side effects. JVM based Functional Programming Languages There are many JVM based languages which supports functional programming paradigm. However I intend to limit discussion around following. Scala Clojure Groovy Lambda Expressions in Java 8 Lambda Expressions is not a programming language but a feature that will be supported in Java8. The reason for including it in this article is to emphasize on the fact that going forward Java will also support writing code in functional style. Scala Scala is a statically typed multi-paradigm programming language designed to integrate features of object oriented programming and functional programming. Since it is static, one cannot change class definition at run time i.e. one cannot add new methods or variables at run-time. However Scala does provide functional programming concepts i.e. immutability, higher-order functions, nested functions etc. Apart from supporting Java's concurrency model, it also provides concept of Actor model out of the box for event based asynchronous message passing between objects. The code written in Scala gets compiled into very efficient bytecode which can then be executed on JVM. Creating immutable list in Scala is very simple and doesn't require any extra effort. "val" keyword does the trick. val numbers = List(1,2,3,4) Functions can be passed as arguments. Let's see this with an example. Suppose we have a list of 10 numbers and we want to calculate sum of all the numbers in list. val numbers = List(1,2,3,4,5,6,7,8,9,10) val total = numbers.foldLeft(0){(a,b) => a+b } As can be seen in above example, we are passing a function to add two variables "a" and "b" to another function "foldLeft" which is provided by Scala library on collections. We have also not used any iteration logic and temporary variable to calculate the sum. "foldLeft" method eliminates the need to maintain state in temporary variable which would have otherwise be required if we were to write this code in pure Java way (as mentioned below). int total = 0; for(int number in numbers){ total+=number; } Scala function can easily be executed in parallel without any need for synchronization since it does not mutate state. This was just a small example to showcase the power of Scala as functional programming language. There are whole lot of features available in Scala to write code in functional style. Clojure Clojure is a dynamic language with an excellent support for writing code in functional style. It is a dialect of "lisp" programming language with an efficient and robust infrastructure for multithreaded programming. Clojure is predominantly a functional programming language, and features a rich set of immutable, persistent data structures. When mutable state is needed, Clojure offers a software transactional memory system and reactive Agent system that ensure clean, correct multithreaded designs. Apart from this since Clojure is a dynamic language, it allows to modify class definition at run time by adding new methods or modifying existing one at run time. This makes it different from Scala which is a statically typed language. Immutability is in the root of Clojure. To create immutable list just following needs to be done. By default list in Clojure is immutable, so does not require any extra effort. (def numbers (list 1 2 3 4 5 6 7 8 9 10)) To add numbers without maintaining state, reduce function can be used as mentioned below (reduce + 0 '(1 2 3 4 5 6 7 8 9 10)) As can be seen, adding list of numbers just requires one line of code without mutating any state. This is the beauty about functional programming languages and plays an important role for parallel execution. Groovy Groovy is again a dynamic language with some support for functional programming. Amongst the 3 languages, Groovy can be considered weakest in terms of functional programming features. However because of it's dynamic nature and close resemblance to Java, it has been widely accepted and considered good alternative to Java. Groovy does not provide immutable objects out of the box but has excellent support for higher order functions. Immutable objects can be created with annotation @Immutation, but it's far less flexible than immutablity support in Scala and Clojure. In Groovy functions can be passed around just as any other variable in the form of Closures. The same example in Groovy can be written as follows def numbers = [1,2,3,4,5,6,7,8,9,10] def total = numbers.inject(0){a,b -> a+b } However the point to be noted is that variables "numbers" and "total" are not immutable and can be modified at any point of time. Hence writing multithreaded code can be a bit challenging. But Groovy does provide the concept of Actors, Agents and DataFlow variables via library called GPars(Groovy Parallel System) which reduces the challenges associated with multithreaded code to a greater extent. Java8 Lambda Expression Java has finally realized the power of writing code in functional style and is going to support the concept of closures starting from Java8. JSR 335 - Lambda Expressions for the JavaTM Programming Language aims to support programming in a multicore environment by adding closures and related features to the Java language. So it will finally be possible to pass around functions similar to variables in pure Java code. Currently if someone wants to try out and play around lambda expressions, Project Lambda of OpenJDK provides prototype implementation of JSR-335. Following code snippet should run fine with OpenJDK Project Lambda compiler. ExecutorService executor = Executors.newCachedThreadPool(); executor.submit(() -> {System.out.println("I am running")}) As can be seen above, a closure(function) has been passed to executor's submit method. It does not take any argument and hence empty brackets () have been placed. This function just prints "I am running" when executed. Just as we can pass functions to function, it will also be possible to create closure within functions and return closure from function. I would recommend to try out OpenJDK to get a feel of lambda expressions which is going to be part of Java8 Conclusion So this was all about functional programming, it's concepts, benefits and options available on JVM to write function code. Functional programming requires a different mind-set and can be very useful if used correctly. Functional Programming along with Object Oriented Programming can be a jewel in crown. As discussed there are various options available to write code in functional style that can be executed on JVM. Choice depends on various factors and there is no one language that can be considered best in all aspects. However one thing is for sure, going forward we are going to see more and more usage of functional programming.

At my employer we have a local NuGet server to host all of our internal packages. Occasionally, I’ll be working on a project and realize that I need to tweak something in one of my NuGet packages. Initially, I got into the habit of opening up a second instance of Visual Studio, making the necessary changes and using the NuGet web interface to re-upload the package. I quickly realized that manually uploading the package was too time consuming. Therefore, I started looking for a way to automate the process instead. Eventually that led me to the PowerShell script you see below. $nugetServer = "https://" $apiKey = "" $packageName = "" $latestRelease = nuget list $packageName $version = $latestRelease.split(" ")[1]; $versionTokens = $version.split(".") $buildNumber = [System.Double]::Parse($versionTokens[$versionTokens.Count -1]) $versionTokens[$versionTokens.Count -1] = $buildNumber +1 $newVersion = [string]::join('.', $versionTokens) echo $newVersion get-childitem | where {$_.extension -eq ".nupkg"} | foreach ($_) {remove-item $_.fullname} nuget pack -Version $newVersion $package = get-childitem | where {$_.extension -eq ".nupkg"} nuget push -Source $nugetServer $package $apiKey The script needs a few variables defined in order for it to run. The first variable ($nugetServer) is the URL of the NuGet Server. The second variable ($apiKey) is your personal API key. You can get your API key by logging into your NuGet Server with a browser. After you log in, click on your username in the upper right hand corner. This will take you to your account page. On the bottom of the “My Account” page there is a box which you can click on to make your API key visible. Finally the last variable ($packageName) is the name of the package you are uploading. This can be easily acquired by looking at your project properties and copying the Assembly name from the Application tab. Depending on how your machine is configured you may have the option to Run with PowerShell on your context menu. If not, you can take a look at this blog post in order to configure it manually. Alternatively you can use the following command instead. powershell.exe "\publish.ps1" If you have any problems running the script then please refer to the following TechNet article or send me a question and I’ll be glad to help.

Learn how Apache Camel implements the EIPs and offers a standardized, internal domain-specific language (DSL) to integrate applications.

This article will describe the complete root cause analysis of a recent Java deadlock problem observed from a Weblogic 11g production system running on the IBM JVM 1.6.This case study will also demonstrate the importance of mastering Thread Dump analysis skills; including for the IBM JVM Thread Dump format. Environment specification Java EE server: Oracle Weblogic Server 11g & Spring 2.0 OS: AIX 5.3 Java VM: IBM JRE 1.6.0 Platform type: Portal & ordering application Monitoring and troubleshooting tools JVM Thread Dump (IBM JVM format) Compuware Server Vantage (Weblogic JMX monitoring & alerting) Problem overview A major stuck Threads problem was observed & reported from Compuware Server Vantage and affecting 2 of our Weblogic 11g production managed servers causing application impact and timeout conditions from our end users. Gathering and validation of facts As usual, a Java EE problem investigation requires gathering of technical and non-technical facts so we can either derived other facts and/or conclude on the root cause. Before applying a corrective measure, the facts below were verified in order to conclude on the root cause: · What is the client impact? MEDIUM (only 2 managed servers / JVM affected out of 16) · Recent change of the affected platform? Yes (new JMS related asynchronous component) · Any recent traffic increase to the affected platform? No · How does this problem manifest itself? A sudden increase of Threads was observed leading to rapid Thread depletion · Did a Weblogic managed server restart resolve the problem? Yes, but problem is returning after few hours (unpredictable & intermittent pattern) - Conclusion #1: The problem is related to an intermittent stuck Threads behaviour affecting only a few Weblogic managed servers at the time - Conclusion #2: Since problem is intermittent, a global root cause such as a non-responsive downstream system is not likely Thread Dump analysis – first pass The first thing to do when dealing with stuck Thread problems is to generate a JVM Thread Dump. This is a golden rule regardless of your environment specifications & problem context. A JVM Thread Dump snapshot provides you with crucial information about the active Threads and what type of processing / tasks they are performing at that time. Now back to our case study, an IBM JVM Thread Dump (javacore.xyz format) was generated which did reveal the following Java Thread deadlock condition below: 1LKDEADLOCK Deadlock detected !!! NULL --------------------- NULL 2LKDEADLOCKTHR Thread "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012CC08B00) 3LKDEADLOCKWTR is waiting for: 4LKDEADLOCKMON sys_mon_t:0x0000000126171DF8 infl_mon_t: 0x0000000126171E38: 4LKDEADLOCKOBJ weblogic/jms/frontend/FESession@0x07000000198048C0/0x07000000198048D8: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012E560500) 3LKDEADLOCKWTR which is waiting for: 4LKDEADLOCKMON sys_mon_t:0x000000012884CD60 infl_mon_t: 0x000000012884CDA0: 4LKDEADLOCKOBJ weblogic/jms/frontend/FEConnection@0x0700000019822F08/0x0700000019822F20: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" (0x000000012CC08B00) This deadlock situation can be translated as per below: - Weblogic Thread #8 is waiting to acquire an Object monitor lock owned by Weblogic Thread #10 - Weblogic Thread #10 is waiting to acquire an Object monitor lock owned by Weblogic Thread #8 Conclusion: both Weblogic Threads #8 & #10 are waiting on each other; forever! Now before going any deeper in this root cause analysis, let me provide you a high level overview on Java Thread deadlocks. Java Thread deadlock overview Most of you are probably familiar with Java Thread deadlock principles but did you really experience a true deadlock problem? From my experience, true Java deadlocks are rare and I have only seen ~5 occurrences over the last 10 years. The reason is that most stuck Threads related problems are due to Thread hanging conditions (waiting on remote IO call etc.) but not involved in a true deadlock condition with other Thread(s). A Java Thread deadlock is a situation for example where Thread A is waiting to acquire an Object monitor lock held by Thread B which is itself waiting to acquire an Object monitor lock held by Thread A. Both these Threads will wait for each other forever. This situation can be visualized as per below diagram: Thread deadlock is confirmed…now what can you do? Once the deadlock is confirmed (most JVM Thread Dump implementations will highlight it for you), the next step is to perform a deeper dive analysis by reviewing each Thread involved in the deadlock situation along with their current task & wait condition.Find below the partial Thread Stack Trace from our problem case for each Thread involved in the deadlock condition: ** Please note that the real application Java package name was renamed for confidentiality purposes ** Weblogic Thread #8 "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" J9VMThread:0x000000012CC08B00, j9thread_t:0x00000001299E5100, java/lang/Thread:0x070000001D72EE00, state:B, prio=1 (native thread ID:0x111200F, native priority:0x1, native policy:UNKNOWN) Java callstack: at weblogic/jms/frontend/FEConnection.stop(FEConnection.java:671(Compiled Code)) at weblogic/jms/frontend/FEConnection.invoke(FEConnection.java:1685(Compiled Code)) at weblogic/messaging/dispatcher/Request.wrappedFiniteStateMachine(Request.java:961(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.syncRequest(DispatcherImpl.java:184(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.dispatchSync(DispatcherImpl.java:212(Compiled Code)) at weblogic/jms/dispatcher/DispatcherAdapter.dispatchSync(DispatcherAdapter.java:43(Compiled Code)) at weblogic/jms/client/JMSConnection.stop(JMSConnection.java:863(Compiled Code)) at weblogic/jms/client/WLConnectionImpl.stop(WLConnectionImpl.java:843) at org/springframework/jms/connection/SingleConnectionFactory.closeConnection(SingleConnectionFactory.java:342) at org/springframework/jms/connection/SingleConnectionFactory.resetConnection(SingleConnectionFactory.java:296) at org/app/JMSReceiver.receive() …………………………………………………………………… Weblogic Thread #10 "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" J9VMThread:0x000000012E560500, j9thread_t:0x000000012E35BCE0, java/lang/Thread:0x070000001ECA9200, state:B, prio=1 (native thread ID:0x4FA027, native priority:0x1, native policy:UNKNOWN) Java callstack: at weblogic/jms/frontend/FEConnection.getPeerVersion(FEConnection.java:1381(Compiled Code)) at weblogic/jms/frontend/FESession.setUpBackEndSession(FESession.java:755(Compiled Code)) at weblogic/jms/frontend/FESession.consumerCreate(FESession.java:1025(Compiled Code)) at weblogic/jms/frontend/FESession.invoke(FESession.java:2995(Compiled Code)) at weblogic/messaging/dispatcher/Request.wrappedFiniteStateMachine(Request.java:961(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.syncRequest(DispatcherImpl.java:184(Compiled Code)) at weblogic/messaging/dispatcher/DispatcherImpl.dispatchSync(DispatcherImpl.java:212(Compiled Code)) at weblogic/jms/dispatcher/DispatcherAdapter.dispatchSync(DispatcherAdapter.java:43(Compiled Code)) at weblogic/jms/client/JMSSession.consumerCreate(JMSSession.java:2982(Compiled Code)) at weblogic/jms/client/JMSSession.setupConsumer(JMSSession.java:2749(Compiled Code)) at weblogic/jms/client/JMSSession.createConsumer(JMSSession.java:2691(Compiled Code)) at weblogic/jms/client/JMSSession.createReceiver(JMSSession.java:2596(Compiled Code)) at weblogic/jms/client/WLSessionImpl.createReceiver(WLSessionImpl.java:991(Compiled Code)) at org/springframework/jms/core/JmsTemplate102.createConsumer(JmsTemplate102.java:204(Compiled Code)) at org/springframework/jms/core/JmsTemplate.doReceive(JmsTemplate.java:676(Compiled Code)) at org/springframework/jms/core/JmsTemplate$10.doInJms(JmsTemplate.java:652(Compiled Code)) at org/springframework/jms/core/JmsTemplate.execute(JmsTemplate.java:412(Compiled Code)) at org/springframework/jms/core/JmsTemplate.receiveSelected(JmsTemplate.java:650(Compiled Code)) at org/springframework/jms/core/JmsTemplate.receiveSelected(JmsTemplate.java:641(Compiled Code)) at org/app/JMSReceiver.receive() …………………………………………………………… As you can see in the above Thread Strack Traces, such deadlock did originate from our application code which is using the Spring framework API for the JMS consumer implementation (very useful when not using MDB’s). The Stack Traces are quite interesting and revealing that both Threads are in a race condition against the same Weblogic JMS consumer session / connection and leading to a deadlock situation: - Weblogic Thread #8 is attempting to reset and close the current JMS connection - Weblogic Thread #10 is attempting to use the same JMS Connection / Session in order to create a new JMS consumer - Thread deadlock is triggered! Root cause: non Thread safe Spring JMS SingleConnectionFactory implementation A code review and a quick research from Spring JIRA bug database did reveal the following Thread safe defect below with a perfect correlation with the above analysis: # SingleConnectionFactory's resetConnection is causing deadlocks with underlying OracleAQ's JMS connection https://jira.springsource.org/browse/SPR-5987 A patch for Spring SingleConnectionFactory was released back in 2009 which did involve adding proper synchronized{} block in order to prevent Thread deadlock in the event of a JMS Connection reset operation: synchronized (connectionMonitor) { //if condition added to avoid possible deadlocks when trying to reset the target connection if (!started) { this.target.start(); started = true; } } Solution Our team is currently planning to integrate this Spring patch in to our production environment shortly. The initial tests performed in our test environment are positive. Conclusion I hope this case study has helped understand a real-life Java Thread deadlock problem and how proper Thread Dump analysis skills can allow you to quickly pinpoint the root cause of stuck Thread related problems at the code level. Please don’t hesitate to post any comment or question.

Cross-site request forgery attacks (CSRF) are very common in web applications and can cause significant harm if allowed. If you have never heard of CSRF I recommend you check out OWASPs page about it. Luckily preventing CSRF attacks is quite simple, I’ll try to show you how they work and how we can defend from them in the least obtrusive way possible in Java based web apps. Imagine you are about to perform a money transfer in your bank’s secure web page, when you click on the transfer option a form page is loaded that allows you to choose the debit and credit accounts, and enter the amount of money to move. When you are satisfied with your options you press “submit” and send the form information to your bank’s web server, which in turns performs the transaction. Now add the following to the picture, a malicious website (which you think harmless of course) is open on another window/tab of your browser while you are innocently moving all your millions in your bank’s site. This evil site knows the bank’s web forms structure, and as you browse through it, it tries to post transactions withdrawing money from your accounts and depositing it on the evil overlord’s accounts, it can do it because you have an open and valid session with the banks site in the same browser! This is the basis for a CSRF attack. One simple and effective way to prevent it is to generate a random (i.e. unpredictable) string when the initial transfer form is loaded and send it to the browser. The browser then sends this piece of data along with the transfer options, and the server validates it before approving the transaction for processing. This way, malicious websites cannot post transactions even if they have access to a valid session in a browser. To implement this mechanism in Java I choose to use two filters, one to create the salt for each request, and another to validate it. Since the users request and subsequent POST or GETs that should be validated do not necessarily get executed in order, I decided to use a time based cache to store a list of valid salt strings. The first filter, used to generate a new salt for a request and store it in the cache can be coded as follows: package com.ricardozuasti.csrf; import com.google.common.cache.Cache; import com.google.common.cache.CacheBuilder; import com.google.common.cache.CacheLoader; import com.google.common.cache.LoadingCache; import java.io.IOException; import java.security.SecureRandom; import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; import javax.servlet.*; import javax.servlet.http.HttpServletRequest; import org.apache.commons.lang.RandomStringUtils; public class LoadSalt implements Filter { @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { // Assume its HTTP HttpServletRequest httpReq = (HttpServletRequest) request; // Check the user session for the salt cache, if none is present we create one Cache csrfPreventionSaltCache = (Cache) httpReq.getSession().getAttribute("csrfPreventionSaltCache"); if (csrfPreventionSaltCache == null){ csrfPreventionSaltCache = CacheBuilder.newBuilder() .maximumSize(5000) .expireAfterWrite(20, TimeUnit.MINUTES) .build(); httpReq.getSession().setAttribute("csrfPreventionSaltCache", csrfPreventionSaltCache); } // Generate the salt and store it in the users cache String salt = RandomStringUtils.random(20, 0, 0, true, true, null, new SecureRandom()); csrfPreventionSaltCache.put(salt, Boolean.TRUE); // Add the salt to the current request so it can be used // by the page rendered in this request httpReq.setAttribute("csrfPreventionSalt", salt); chain.doFilter(request, response); } @Override public void init(FilterConfig filterConfig) throws ServletException { } @Override public void destroy() { } } I used Guava CacheBuilder to create the salt cache since it has both a size limit and an expiration timeout per entry. To generate the actual salt I used Apache Commons RandomStringUtils, powered by Java 6 SecureRandom to ensure a strong generation seed. This filter should be used in all requests ending in a page that will link, post or call via AJAX a secured transaction, so in most cases it’s a good idea to map it to every request (maybe with the exception of static content such as images, CSS, etc.). It’s mapping in your web.xml should look similar to: ... loadSalt com.ricardozuasti.csrf.LoadSalt ... loadSalt * ... As I said, to validate the salt before executing secure transactions we can write another filter: package com.ricardozuasti.csrf; import com.google.common.cache.Cache; import java.io.IOException; import javax.servlet.*; import javax.servlet.http.HttpServletRequest; public class ValidateSalt implements Filter { @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { // Assume its HTTP HttpServletRequest httpReq = (HttpServletRequest) request; // Get the salt sent with the request String salt = (String) httpReq.getParameter("csrfPreventionSalt"); // Validate that the salt is in the cache Cache csrfPreventionSaltCache = (Cache) httpReq.getSession().getAttribute("csrfPreventionSaltCache"); if (csrfPreventionSaltCache != null && salt != null && csrfPreventionSaltCache.getIfPresent(salt) != null){ // If the salt is in the cache, we move on chain.doFilter(request, response); } else { // Otherwise we throw an exception aborting the request flow throw new ServletException("Potential CSRF detected!! Inform a scary sysadmin ASAP."); } } @Override public void init(FilterConfig filterConfig) throws ServletException { } @Override public void destroy() { } } You should configure this filter for every request that needs to be secure (i.e. retrieves or modifies sensitive information, move money, etc.), for example: ... validateSalt com.ricardozuasti.csrf.ValidateSalt ... validateSalt /transferMoneyServlet ... After configuring both servlets all your secured requests should fail :). To fix it you have to add, to each link and form post that ends in a secure URL, the csrfPreventionSalt parameter containing the value of the request parameter with the same name. For example, in an HTML form within a JSP page: ... ... ... Of course you can write a custom tag, a nice Javascript code or whatever you prefer to inject the new parameter in every needed link/form.

This article will provide you with a tutorial on how you can quickly pinpoint the Java Thread contributors to a high CPU problem on the Windows OS. Windows, like other OS such as Linux, Solaris & AIX allow you to monitor the CPU utilization at the process level but also for individual Thread executing a task within a process. For this tutorial, we created a simple Java program that will allow you to learn this technique in a step by step manner. Troubleshooting tools The following tools will be used below for this tutorial: - Windows Process Explorer (to pinpoint high CPU Thread contributors) - JVM Thread Dump (for Thread correlation and root cause analysis at code level) High CPU simulator Java program The simple program below is simply looping and creating new String objects. It will allow us to perform this CPU per Thread analysis. I recommend that you import it in an IDE of your choice e.g. Eclipse and run it from there. You should observe an increase of CPU on your Windows machine as soon as you execute it. package org.ph.javaee.tool.cpu; /** * HighCPUSimulator * @author Pierre-Hugues Charbonneau * http://javaeesupportpatterns.blogspot.com * */ public class HighCPUSimulator { private final static int NB_ITERATIONS = 500000000; // ~1 KB data footprint private final static String DATA_PREFIX = "datadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadata"; /** * @param args */ public static void main(String[] args) { System.out.println("HIGH CPU Simulator 1.0"); System.out.println("Author: Pierre-Hugues Charbonneau"); System.out.println("http://javaeesupportpatterns.blogspot.com/"); try { for (int i = 0; i < NB_ITERATIONS; i++) { // Perform some String manipulations to slowdown and expose looping process... String data = DATA_PREFIX + i; } } catch (Throwable any) { System.out.println("Unexpected Exception! " + any.getMessage() + " [" + any + "]"); } System.out.println("HighCPUSimulator done!"); } } Step #1 – Launch Process Explorer The Process Explorer tool visually shows the CPU usage dynamically. It is good for live analysis. If you need historical data on CPU per Thread then you can also use Windows perfmon with % Processor Time & Thread Id data counters. You can download Process Explorer from the link below: http://technet.microsoft.com/en-us/sysinternals/bb896653 In our example, you can see that the Eclipse javaw.exe process is now using ~25% of total CPU utilization following the execution of our sample program. Step #2 – Launch Process Explorer Threads view The next step is to display the Threads view of the javaw.exe process. Simply right click on the javaw.exe process and select Properties. The Threads view will be opened as per below snapshot: - The first column is the Thread Id (decimal format) - The second column is the CPU utilization % used by each Thread - The third column is also another counter indicating if Thread is running on the CPU In our example, we can see our primary culprit is Thread Id #5996 using ~ 25% of CPU. Step #3 – Generate a JVM Thread Dump At this point, Process Explorer will no longer be useful. The goal was to pinpoint one or multiple Java Threads consuming most of the Java process CPU utilization which is what we achieved. In order to go the next level in your analysis you will need to capture a JVM Thread Dump. This will allow you to correlate the Thread Id with the Thread Stack Trace so you can pinpoint that type of processing is consuming such high CPU. JVM Thread Dump generation can be done in a few manners. If you are using JRockit VM you can simply use the jrcmd tool as per below example: Once you have the Thread Dump data, simply search for the Thread Id and locate the Thread Stack Trace that you are interested in. For our example, the Thread “Main Thread” which was fired from Eclipse got exposed as the primary culprit which is exactly what we wanted to demonstrate. "Main Thread" id=1 idx=0x4 tid=5996 prio=5 alive, native_blocked at org/ph/javaee/tool/cpu/HighCPUSimulator.main (HighCPUSimulator.java:31) at jrockit/vm/RNI.c2java(IIIII)V(Native Method) -- end of trace Step #4 – Analyze the culprit Thread(s) Stack Trace and determine root cause At this point you should have everything that you need to move forward with the root cause analysis. You will need to review each Thread Stack Trace and determine what type of problem you are dealing with. That final step is typically where you will spend most of your time and problem can be simple such as infinite looping or complex such as garbage collection related problems. In our example, the Thread Dump did reveal the high CPU originates from our sample Java program around line 31. As expected, it did reveal the looping condition that we engineered on purpose for this tutorial. for (int i = 0; i < NB_ITERATIONS; i++) { // Perform some String manipulations to slowdown and expose looping process... String data = DATA_PREFIX + i; } I hope this tutorial has helped you understand how you can analyze and help pinpoint root cause of Java CPU problems on Windows OS. Please stay tuned for more updates, the next article will provide you with a Java CPU troubleshooting guide including how to tackle that last analysis step along with common problem patterns.

Following on from the previous post on operator overloading I'm going to be looking at Implicit Conversions, and how we can combine them to with operator overloading to do some really neat things, including one way of creating a multi-parameter conversion. So what's an "Implicit Conversion" when it's at home? So lets start with some basic Scala syntax, if you've spent any time with Scala you've probably noticed it allows you to do things like: (1 to 4).foreach(println) // print out 1 2 3 4 Ever wondered how it does this? Lets make things more explicit, you could rewrite the above code as: val a : Int = 1 val b : Int = 4 val myRange : Range = a to b myRange.foreach(println) Scala is creating a Range object directly from two Ints, and a method called to. So what's going on here? Is this just a sprinkling of syntactic sugar to make writing loops easier? Is to just a keyword in like def or val? The answers to all this is no, there's nothing special going on here. to is simply a method defined in the RichInt class, which takes a parameter and returns a Range object (specifically a subclass of Range called Inclusive). You could rewrite it as the following if you really wanted to: val myRange : Range = a.to(b) Hang on though, RichInt may have a "to" method but Int certainly doesn't, in your example you're even explicitly casting your numbers to Ints Which brings me nicely on to the subject of this post, Implicit Conversions. This is how Scala does this. Implicit Conversions are a set of methods that Scala tries to apply when it encounters an object of the wrong type being used. In the case of the to example there's a method defined and included by default that will convert Ints into RichInts. So when Scala sees 1 to 4 it first runs the implicit conversion on the 1 converting it from an Int primitive into a RichInt. It can then call the to method on the new RichInt object, passing in the second Int (4) as the parameter. Hmm, think I understand, how's about another example? Certainly. Lets try to improve our Complex number class we created in the previous post. Using operator overloading we were able to support adding two complex numbers together using the + operator. eg. class Complex(val real : Double, val imag : Double) { def +(that: Complex) = new Complex(this.real + that.real, this.imag + that.imag) def -(that: Complex) = new Complex(this.real - that.real, this.imag - that.imag) override def toString = real + " + " + imag + "i" } object Complex { def main(args : Array[String]) : Unit = { var a = new Complex(4.0,5.0) var b = new Complex(2.0,3.0) println(a) // 4.0 + 5.0i println(a + b) // 6.0 + 8.0i println(a - b) // 2.0 + 2.0i } } But what if we want to support adding a normal number to a complex number, how would we do that? We could certainly overload our "+" method to take a Double argument, ie something like... def +(n: Double) = new Complex(this.real + n, this.imag) Which would allow us to do... val sum = myComplexNumber + 8.5 ...but it'll break if we try... val sum = 8.5 + myComplexNumber To get around this we could use an Implicit Conversion. Here's how we create one. object ComplexImplicits { implicit def Double2Complex(value : Double) = new Complex(value,0.0) } Simple! Although you do need to be careful to import the ComplexImplicits methods before they can be used. You need to make sure you add the following to the top of your file (even if your Implicits object is in the same file)... import ComplexImplicits._ And that's the problem solved, you can now write val sum = 8.5 + myComplexNumber and it'll do what you expect! Nice. Is there anything else I can do with them? One other thing I've found them good for is creating easy ways of instantiating objects. Wouldn't it be nice if there were a simpler way of creating one of our complex numbers other than with new Complex(3.0,5.0). Sure you could get rid of the new by making it a case class, or implementing an apply method. But we can do better, how's about just (3.0,5.0) Awesome, but I'd need some sort of multi parameter implicit conversion, and I don't really see how that's possible!? The thing is, ordinarily (3.0,5.0) would create a Tuple. So we can just use that tuple as the parameter for our implicit conversion and convert it into a Complex. how we might go about doing this... implicit def Tuple2Complex(value : Tuple2[Double,Double]) = new Complex(value._1,value._2); And there we have it, a simple way to instantiate our Complex objects, for reference here's what the entire Complex code looks like now. import ComplexImplicits._ object ComplexImplicits { implicit def Double2Complex(value : Double) = new Complex(value,0.0) implicit def Tuple2Complex(value : Tuple2[Double,Double]) = new Complex(value._1,value._2); } class Complex(val real : Double, val imag : Double) { def +(that: Complex) : Complex = (this.real + that.real, this.imag + that.imag) def -(that: Complex) : Complex = (this.real - that.real, this.imag + that.imag) def unary_~ = Math.sqrt(real * real + imag * imag) override def toString = real + " + " + imag + "i" } object Complex { val i = new Complex(0,1); def main(args : Array[String]) : Unit = { var a : Complex = (4.0,5.0) var b : Complex = (2.0,3.0) println(a) // 4.0 + 5.0i println(a + b) // 6.0 + 8.0i println(a - b) // 2.0 + 8.0i println(~b) // 3.60555 var c = 4 + b println(c) // 6.0 + 3.0i var d = (1.0,1.0) + c println(d) // 7.0 + 4.0i } }

So, I've been teaching myself Scala recently, and it's a very interesting language. One of the nice things I like about it, is it's support for creating DSLs, domain specific languages. A domain specific language - or at least my understanding of it - is a language that is written specifically for one problem domain. One example would be SQL, great for querying relational databases, useless for creating first person shooters. Of course Scala itself is not a DSL, it's a general purpose language. However it does offer several features that allow you to simulate a DSL, in particular operator overloading, and implicit conversions. In this post I'm going to focus on the first of these... Operator Overloading So what's operator overloading? Well operators are typically things such as +, -, and !. You know those things you use to do arithmetic on numbers, or occasionally for manipulating Strings. Well, operator overloading - just like method overloading - allows you to redefine their behaviour for a particular type, and give them meaning for your own custom classes. Hang on a minute! I'm sure someone once told me operator overloading was evil? Indeed, this is quite a controversial topic. It's considered far too open for abuse by some, and was so maligned in C++ that the creators of Java deliberately disallowed it (excepting "+" for String concatenation). I'm of a slightly different opinion, used responsibly it can be very useful. For example lots of different objects support a concept of addition, so why not just use an addition operator? Lets say you were developing a complex number class, and you want to support addition. Wouldn't it be nicer to write... Complex result = complex1 + complex2; ...rather than... Complex result = complex1.add(complex2); The first example is much more natural don't you think? So Scala allows you to overload operators then? Well, not really. In fact, technically not at all. So all this is just a tease? This is the most stupid blog post I've ever read. Scala's rubbish. I'm going back to Algol 68. Wait a second, I've not finished. You see Scala doesn't support operator overloading, because it doesn't have operators! Scala doesn't have operators? You've gone mad, I write stuff like "sum = 2 + 3" all the time, and what about all those funny list operations? "::", and ":/". They look like operators to me! Well they're not. The thing is, Scala has a rather relaxed attitude to what you can name a method. When you write... sum = 2 + 3, ...you're actually calling a method called + on a RichInt type with a value of 2. You could even rewrite it as... sum = 2.+(3) ...if you really really wanted to. Aha, I got it. So how do you go about overloading an operator then? Simple, it's exactly the same as writing a normal method. Here's an example. class Complex(val real : Double, val imag : Double) { def +(that: Complex) = new Complex(this.real + that.real, this.imag + that.imag) def -(that: Complex) = new Complex(this.real - that.real, this.imag - that.imag) override def toString = real + " + " + imag + "i" } object Complex { def main(args : Array[String]) : Unit = { var a = new Complex(4.0,5.0) var b = new Complex(2.0,3.0) println(a) // 4.0 + 5.0i println(a + b) // 6.0 + 8.0i println(a - b) // 2.0 + 2.0i } } Ok that's nice, what if I wanted a "not" operator though, ie something like a "!" That's a unary prefix operator, and yes scala can support these, although in a more limited fashion than an infix operator like "+" Only four operators can be supported in this fashion, +, -, !, and ~. You simply need to call your methods unary_! or unary_~, etc. Here's how you might add a "~" to calculate the magnitude of a Complex number to our complex number class class Complex(val real : Double, val imag : Double) { // ... def unary_~ = Math.sqrt(real * real + imag * imag) } object Complex { def main(args : Array[String]) : Unit = { var b = new Complex(2.0,3.0) prinln(~b) // 3.60555 } } So that's all pretty simple, but please use responsibly. Don't create methods called "+" unless your class really does something that could be interpreted as addition. And never ever redefine the binary shift left operator "<<" as some sort of substitute for println. It's not clever and you'll make the Scala gods angry. Hope you found that useful. Next up I'll cover implicit conversions. Another nice feature of Scala that really allows you to write your code in a more natural way

On the surface these methods do the same thing in Java; Thread.yield(), Thread.sleep(0), Object.wait(0,1) and LockSupport.parkNanos(1) They all wait a sort period of time, but how much that is varies a surprising amount and between platforms. Timing a short delay The following code times how long it takes to repeatedly call those methods. import java.util.concurrent.locks.LockSupport; public class Pausing { public static void main(String... args) throws InterruptedException { int repeat = 10000; for (int i = 0; i < 3; i++) { long time0 = System.nanoTime(); for (int j = 0; j < repeat; j++) Thread.yield(); long time1 = System.nanoTime(); for (int j = 0; j < repeat; j++) Thread.sleep(0); long time2 = System.nanoTime(); synchronized (Thread.class) { for (int j = 0; j < repeat/10; j++) Thread.class.wait(0, 1); } long time3 = System.nanoTime(); for (int j = 0; j < repeat/10; j++) LockSupport.parkNanos(1); long time4 = System.nanoTime(); System.out.printf("The average time to yield %.1f μs, sleep(0) %.1f μs, " + "wait(0,1) %.1f μs and LockSupport.parkNanos(1) %.1f μs%n", (time1 - time0) / repeat / 1e3, (time2 - time1) / repeat / 1e3, (time3 - time2) / (repeat/10) / 1e3, (time4 - time3) / (repeat/10) / 1e3); } } } On Windows 7 The average time to yield 0.3 μs, sleep(0) 0.6 μs, wait(0,1) 999.9 μs and LockSupport.parkNanos(1) 1000.0 μs The average time to yield 0.3 μs, sleep(0) 0.6 μs, wait(0,1) 999.5 μs and LockSupport.parkNanos(1) 1000.1 μs The average time to yield 0.2 μs, sleep(0) 0.5 μs, wait(0,1) 1000.0 μs and LockSupport.parkNanos(1) 1000.1 μs On RHEL 5.x The average time to yield 1.1 μs, sleep(0) 1.1 μs, wait(0,1) 2003.8 μs and LockSupport.parkNanos(1) 3.8 μs The average time to yield 1.1 μs, sleep(0) 1.1 μs, wait(0,1) 2004.8 μs and LockSupport.parkNanos(1) 3.4 μs The average time to yield 1.1 μs, sleep(0) 1.1 μs, wait(0,1) 2005.6 μs and LockSupport.parkNanos(1) 3.1 μs In summary If you want to wait for a short period of time, you can't assume that all these methods do the same thing, nor will be the same between platforms.

You may be using JSON to transfer data (we were using it in our message queue). While this is good, it has the only benefit of being human-readable. If you don’t care about readability, you’d probably want to use a more efficient serialization mechanism. Multiple options exist: protobuf, MessagePack, protostuff, java serialization. The easiest of them to use is java serialization, but it is less efficient (with both memory and time) than the other solutions. There are some benchmarks that will help you choose the most efficient solution, but if you want it to be easy and almost drop-in replacement to your JSON solution, MessagePack might be the best option. I made a simple test to compare the JSON output to the MessagePack output in terms of size: 2300 vs 150 bytes for a simple message. Pretty good reduction, and if the messages are a lot, it’s a must to optimize. However, you need to register all classes in the message pack. There are two options: use @Message on all the objects in the serialized graph. This is a bit tedious, especially if you already have a lot of classes that are transferred. You have to go through the whole graph you can manually register all classes with the mesagpack. Again tedious, because you also have to register all classes that the message class contains as a field (recursively) That’s why I wrote the following code to loop all our message classes, and register them with the message pack on startup. It partly relies on spring classes, but if you are not using Spring, you can replace them: private MessagePack serializer = new MessagePack(); private ClassMapper classMapper = new DefaultClassMapper(); @PostConstruct public void init() { // we need to find all messages, and register their classes, and also all their fields' recursively ClassPathScanningCandidateComponentProvider provider = new ClassPathScanningCandidateComponentProvider(false); Set classes = provider.findCandidateComponents("com.foo.bar.messages"); // hacking MessagePack to allow Set handling Field fld = ReflectionUtils.findField(MessagePack.class, "registry"); ReflectionUtils.makeAccessible(fld); TemplateRegistry registry = (TemplateRegistry) ReflectionUtils.getField(fld, serializer); registry.register(Set.class, new SetTemplate(new AnyTemplate(registry))); registry.registerGeneric(Set.class, new GenericCollectionTemplate(registry, SetTemplate.class)); try { for (BeanDefinition def : classes) { Class clazz = Class.forName(def.getBeanClassName()); registerHierarcy(clazz, serializer, Sets.>newHashSet()); } } catch (ClassNotFoundException e) { throw new IllegalStateException(e); } } private void registerHierarcy(Class clazz, MessagePack serializer, Set> handledClasses) { if (!isEligibleForRegistration(clazz)) { return; } Class currentClass = clazz; while (currentClass != null && !currentClass.isEnum() && currentClass != Object.class) { for (Field field : currentClass.getDeclaredFields()) { registerHierarcy(field.getType(), serializer, handledClasses); // type parameters Type type = field.getGenericType(); if (type instanceof ParameterizedType) { for (Type typeParam : ((ParameterizedType) type).getActualTypeArguments()) { // avoid circular generics references, resulting in stackoverflow Class typeParamClass = (Class) typeParam; if (!handledClasses.contains(typeParamClass)) { handledClasses.add(typeParamClass); registerHierarcy(typeParamClass, serializer, handledClasses); } } } } currentClass = currentClass.getSuperclass(); } try { serializer.register(clazz); } catch (Exception ex) { logger.warn("Problem registering class " + clazz, ex.getMessage()); } } private boolean isEligibleForRegistration(Class clazz) { return !(clazz.isAnnotationPresent(Entity.class) || clazz == Class.class || Type.class.isAssignableFrom(clazz) || clazz.isInterface() || clazz.isArray() || ClassUtils.isPrimitiveOrWrapper(clazz) || clazz == String.class || clazz == Date.class || clazz == Object.class); }

What is Zebra? The Zebra is a JavaScript library that follows easy OOP concepts and implements a rich set of various self-made UI components. There are a huge number of attractive and powerful WEB UI frameworks on the market. Most of them are stuck to HTML, DOM and CSS. They build UIs by coloring DOM with CSS and manipulating the HTML DOM tree. Zebra UI is different. It is not based on HTML, DOM or CSS. Zebra UI components are implemented and rendered from scratch as a number of widgets organized in hierarchy. How does Zebra manage to build and render UIs on the web? The essential thing required by Zebra abstraction is the existence of a "Canvas-like " component. The "Canvas" component has to have set of graphical methods to paint lines, simple shapes, text, images and be able to catch input (keyboard, mouse, etc) events. For instance Java AWT/SWING, Eclipse SWT, .NET or other platforms will supply "Canvas-like" components. But the new HTML5 standard embeds "Canvas" element. This element is supported by most of the modern browsers and is what makes it possible and reasonable to "drop" Zebra UI into a web context. See Zebra demo http://zebra.gravitysoft.org The table below lists the most significant Zebra UI components, managers view, etc. Components marked by orange background are still in development: The Zebra Java to JavaScript converter is out of context in this article, nevertheless it plays key role in porting Java-based UI components onto the web. For demo purposes, a slightly outdated version of the Java to JavaScript converter is available: http://j2js.gravitysoft.org Zebra JavaScript Easy OOP concept An attractive, easy for use, and understandable programming model is very important in context of having supportable, extendable, elegant code. This is one of the painful problems in web development. Zebra introduces easy OOP concepts as a foundation. ~10.5kb of Zebra code helps to do the following: To get more information about Zebra OOP see the cheat sheet: Zebra easy OOP concept cheat sheet Zebra "Hello WEB" application The Zebra JavaScript demo indents to show an internal window with a "Hello web" title. The window is opened by pressing a button: // create canvas and store root panel in local variable var c = new zCanvas(html5Canvas), r = c.root; // create button var b = new Button("Hello WEB"); // set border layout manager r.setLayout(new BorderLayout()); // add button to center of root panel r.add(Layout.CENTER, b); // register the button event listener that opens external // window every time the button has been pressed b._(function() { var w = new Window("Hello WEB"); w.setSize(200,200); w.show(); }); Zebra UI features via JavaScript code snippets Zebra UI component design follows approaches similar to traditional Java AWT/SWING, .NET, and Eclipse SWT. UI components are organized as a hierarchy where some components are laid out on others. But Zebra does many things much more quickly and easily. Below are some Zebra UI features equpped with code snippets: Compound components. Zebra UI components often are built as a combination of several other components laid out on a panel. For instance the "Button" component is a panel that keeps a child component as its label. By default title is "Label", but developers can set other UI components as a button label. Take a look at the snippets below: // by default button uses label component as its content var button = new Button("Label"); // set image as the button label var button = new Button(new ImagePan("b.png")); // set image+label as the button content. The image+label is also // compound component that consists of image and label var bContent = new Panel(new FlowLayout()); bContent.add(new ImagePan()); bContent.add(new Label()); var button = new Button(bContent); The "BorderPan" UI component is one more example of a compound component. It consists of a label and content panel. Take a look at the snippets below: // border panel uses simple "Label" as its title by default var bp = new BorderPanel("Label", new Panel()); // border panel uses another border panel as its title var bp = new BorderPanel(new BorderPan("Label"), new Panel()); // border panel uses "Checkbox" as its title var bp = new BorderPanel(new Checkbox("Check me"), new Panel()); Full control over UI component rendering. Zebra UI components painting is fully in developers hands. Zebra calls "paint", "update", "paintOnTop" UI component methods with a graphical context as an input argument. These methods form a UI component face. Passed graphical context gives developers number of elementary methods to draw and fill primitive shapes, paint text, and images. The "paint" method defines the UI component "face": // create inner class instance with redefined component "face" var myEyesCandyComponent = new Panel([ function paint(g) { // paint what ever you want using passed graphical context g.drawLine(...); g.drawRect(...); g.fillArc(...); } ]); The "update" method forms a component background: // create inner class instance and override background // rendering with a custom implementation var myEyesCandyComponent = new Panel([ function update(g) { ... } ]); The "paintOnTop" method may be used, for instance, to render a focus indicator over the component "face": // render rectangle around component if the component holds focus var myEyesCandyComponent = new Panel([ function paintOnTop(g) { if (this.hasFocus()) g.drawRect(0,0, this.width, this.height); } ]); Zebra paint manager takes care of triggering the re-painting of invalidated-"dirty" areas. UI components are designed to make direct "repaint" method execution unnecessary. But custom UI components should call thr "repaint([x, y, w, h])" method every time a new dirty area has appeared. The "repaint" method execution triggers paint manager to recalculate current dirty area and schedule repainting when it is possible: var MyCustomComponent = Class(Panel, [ function setBorderColor(c) { this.color = c; this.repaint(); // inform paint manager the component // has to be completely re-painted } ]); Neat events handling. When it is possible, Zebra event handling follows a declarative pattern (see next feature bullet). It allows developers avoiding listeners registration and simplifies the event handling concept. To catch an event: Express an intention to get the desired event type by inheritance an appropriate interface. Interface is like a marker. Implement function(s) to treat desired event. For instance, imagine a custom component that needs to handle mouse events. Express the intention by implementing the "MouseListener" interface: // let Zebra know that the component wants getting mouse // events by implementing"MouseListener" interface var MyComponent = new Class(Panel, MouseListener, []); Suppose "mouse button has pressed" and "mouse cursor has entered the component" event types have to be handled. To do it, declare the following functions correspondently: var MyComponent = new Class(Panel, MouseListener, [ function mousePressed(e) { // handle mouse pressed event here }, function mouseEntered(e) { // handle mouse entered event here } ]); Global event handling. The special Zebra event manager keeps track of all events that have occurred for all instantiated UI components. The manager can be utilized to register global event listeners that get all events of the given type. For instance, listening to focus events globally can be done this way: // instantiate "FocusListener" interface to express that focus // events have to be catched zebra.ui.events.addListener(new FocusListener([ function focusLost(e) { ... }, function focusGained(e) { ... } ])); Catching children components events. The Parent UI component can listen to input events that have happened in its children by implementing the "ChildrenListener" interface and adding appropriate method(s) as follows: // implements "ChildrenListener" interface to express intention // to get children events var MyComponent = new Class(Panel, ChildrenListener, [ function childInputEvent(e){ // handle children events here } ]); Composite component (event transparent children). Often compound UI components have to prevent their children components from getting any events. Children components are becoming event transparent. This can be done by implementing "Composite" interface and adding the "catchInput" method. The method should return "true" if the passed as argument child has to be event transparent. For instance: var MyComponent = new Class(Panel, Composite, [ function catchInput(kid){ // return true if the given kid has to be event transparent return true; } ]); Declarative pattern. "First inherit an interface to express an intention to do something and then implement required method(s)". This approach allows Zebra to decouple various functional parts from each other. Extending in this context means adding new declarative patterns instead of overloading Zebra UI classes with new code and APIs. For instance, imagine we need to change the mouse cursor type every time the mouse pointers enter a UI component. Zebra UI components don’t know how to control mouse cursor type. It's the cursor manager ("zebra.ui.cursor") that does it: // inherits "CursorInfo" interface to let cursor manager know // the component controls mouse cursor var p = new Panel(CursorInfo, [ // declare method that returns mouse cursor type for the component function getCursorType(x, y){ return Cursor.WAIT; } ]); HTML5 Canvas transformation operations can be applied to UI. Rotation, zoom in, zoom out and other graphical transformation effects can be applied to Zebra UI: // zoom UI in 1.3 times vertically // and horizontally zCanvas.scale(1.3, 1.3); // rotate zCanvas.rotate(0.3); ... // set back to initial stat zCanvas.scale(null); zCanvas.rotate(null, null); Look and feel customization. A lot of Zebra UI components' visual characteristics are defined by a special properties file. If the properties file is not custom enough, a UI Wizard class can be implemented. The instance of the class is notified about all instantiated UI components by calling a "customize" method. It helps to customize UI components' look and feel "on the fly". For instance, let’s define own wizard class to color all label components with red background: // declare custom Wizard class var MyCustomWizard = Class(Wizard, [ // the method is called every time a new component has been // instantiated function customize(id, comp) { // customize just instantiated label component background if (id == Wizard.LABEL) comp.setBackground(Fill.red); } ]); Then setup your wizard with the properties file as follow: ... # specify wizard to be used for UI customization wizard = MyCustomWizard() Layered architecture. Zebra Canvas is the root panel that consists of a number of layers. Layers are a standard Zebra UI "Panel" that are stretched over the whole Canvas surface. Zebra holds layers as a stack. Every time an event occurs the Zebra event manager "asks" (starting from top to bottom layer) who wants to take control. The first met layer that grabs control is selected as the target. The picture below explains it: Let’s develop a layer that freezes (blocks any interaction) and un-freezes UI components by pressing the "CTRL + SHIFT + ALT" keys combination. Freezing is indicated by dimming UI components: // declare custom layer class that inherits "BaseLayer" class var Freezer = Class(BaseLayer, [ function () { this.$super("FREEZER"); // call super with unique layer ID this.isActive = false; // set background to be 100% transparent this.setBackground(null); }, function layerKeyPressed(code, mask) { var rm = KeyEvent.CTRL + KeyEvent.SHIFT + KeyEvent.ALT; if ((rm & mask) == rm) { // CTRL+SHIFT+ALT keys combination has been pressed if (this.isActive) this.setBackground(null); else this.setBackground(new Fill(255,255,255, 0.7)); this.isActive = ! this.isActive; } }, // methods below indicate if the layer is in active state (take control) function isLayerActive(){ return this.isActive;}, function isLayerActiveAt(x,y){return this.isActive; } ]); ... // add the layer to zebra canvas zCanvas.add(new Freezer()); The result of the layer work is demonstrated below: Un-frozen UI Frozen UI Layout management. Layout specifies a rule that says how to shape a number of child components on the given panel. Rule-based positioning is much better than absolute locations and fixed size usage. Gained advantages are the same as "vector vs pixel graphics". Zebra layout managers are independent from the Zebra UI package and can be re-used to layout other objects, for instance HTML elements. Let’s see how, for instance, diagonal layouts can be implemented. The custom layout manager lays out children components aligning its top left corners to diagonal: // declare diagonal layout manager var DiagonalLayout = Class(Layout, [ // "layout" method positions and shapes visible // children of target component function layout(target) { var x = 0, y = 0; for (var i=0; i

Here are some notes on adding Camel (from v2.3+) exception handling to a JavaDSL route. There are various approaches/options available. These notes cover the important distinctions between approaches... Default handling The default mode uses the DefaultErrorHandler strategy which simply propagates any exception back to the caller and ends the route immediately. This is rarely the desired behavior, at the very least, you should define a generic/global exception handler to log the errors and put them on a queue for further analysis (during development, testing, etc). onException(Exception) .to("log:GeneralError?level=ERROR") .to("activemq:queue:GeneralErrorQueue"); Try-catch-finally This approach mimics the Java for exception handling and is designed to be very readable and easy to implement. It inlines the try/catch/finally blocks directly in the route and is useful for route specific error handling. from("direct:start") .doTry() .process(new MyProcessor()) .doCatch(Exception.class) .to("mock:error"); .doFinally() .to("mock:end"); onException This approach defines the exception clause separately from the route. This makes the route and exception handling code more readable and reusable. Also, the exception handling will apply to any routes defined in its CamelContext. onException(Exception.class) .to("mock:error"); from("direct:start") .process(new MyProcessor()) .to("mock:end"); Handled/Continued These APIs provide valuable control over the flow. Adding handled(true) tells Camel to not propagate the error back to the caller (should almost always be used). The continued(true) tells Camel to resume the route where it left off (rarely used, but powerful). These can both be used to control the flow of the route in interesting ways, for example... from("direct:start") .process(new MyProcessor()) .to("mock:end"); //send the exception back to the client (rarely used, clients need a meaningful response) onException(ClientException.class) .handled(false) //default .log("error sent back to the client"); //send a readable error message back to the client and handle the error internally onException(HandledException.class) .handled(true) .setBody(constant("error")) .to("mock:error"); //ignore the exception and continue the route (can be dangerous, use wisely) onException(ContinuedException.class) .continued(true); Using a processor for more control If you need more control of the handler code, you can use an inline Processor to get a handle to the exception that was thrown and write your own handler code... onException(Exception.class) .handled(true) .process(new Processor() { public void process(Exchange exchange) throws Exception { Exception exception = (Exception) exchange.getProperty(Exchange.EXCEPTION_CAUGHT); //log, email, reroute, etc. } }); Summary Overall, the exception handling is very flexible and can meet almost any scenario you can come up with. For the sake of focusing on the basics, many advanced features haven't been covered here. For more details, see these pages on Camel's site... http://camel.apache.org/error-handling-in-camel.html http://camel.apache.org/try-catch-finally.html http://camel.apache.org/exception-clause.html

In a previous post I demonstrated how EclipseLink MOXy can be leveraged to produce both XML and JSON representations of your domain model. The same metadata is used for both representations and MOXy applies it to leverage the capabilities of the media type. In this post I'll focus on how null is handled in each of these representations. Domain Model By default a JAXB (JSR-222) implementation will not include a mapped field/property with a null value in the output. If you want the null value represented then you simply add the following annotation @XmlElement(nillable=true). package blog.json.nillable; import javax.xml.bind.annotation.*; @XmlRootElement public class Customer { private String firstName; private String middleName; private String lastName; public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } public String getMiddleName() { return middleName; } public void setMiddleName(String middleName) { this.middleName = middleName; } @XmlElement(nillable=true) public String getLastName() { return lastName; } public void setLastName(String lastName) { this.lastName = lastName; } } Demo In the demo code below we will set both the middleName and lastName properties to null. Since we have mapped these properties differently in the domain model we will examine the output to see the impact of using @XmlElement(nillable=true). package blog.json.nillable; import javax.xml.bind.*; public class Demo { public static void main(String[] args) throws Exception { Customer customer = new Customer(); customer.setFirstName("Jane"); customer.setMiddleName(null); customer.setLastName(null); JAXBContext jc = JAXBContext.newInstance(Customer.class); Marshaller marshaller = jc.createMarshaller(); marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); // Output XML marshaller.marshal(customer, System.out); // Output JSON marshaller.setProperty("eclipselink.media-type", "application/json"); marshaller.marshal(customer, System.out); } } XML Output By default JAXB implementations do not include null values in the output, so there is no element corresponding to the middleName property. Since we annotated the lastName property with @XmlElement(nillable=true) and it had a null value, it is represented in the output. In XML an element with a null value is represented by adding the xsi:nil="true" attribute. Jane JSON Output Just like in the XML output, an entry for the middleName property does not appear in the JSON output. Since we annotated the lastName property with @XmlElement(nillable=true) and it had a null value, it is represented in the output. In JSON a null value is represented with null. { "customer" : { "firstName" : "Jane", "lastName" : null } }

Do you know what day of the week was the day you were born? Monday or maybe Saturday? Well, perhaps you know that. Everybody knows the day he’s born on, but do you know what day was the 31st of January in 1883? No? Well, there must be some method to determine any day in any century. We know that 2012 started at Sunday. After we know that, it’s easy to determine what day is the 2nd of January. It should be Monday. But things get a little more complex if we try to guess some date distant from January the 1st. Indeed 1st of Jan was on Sunday, but what day is 9th of May the same year. This is far more difficult to say. Of course we can go with a brute force approach and count from 1/Jan till 9/May, but that is quite slow and error prone. So what would we do if we had to code a program that answers this question? The easiest way is to use a library. Almost every major library has built-in functions that can answer what day is on a given date. Such are date() in PHP or getDate() in JavaScript. But the question remains: How these library functions know the answer and how can we code such library functions if our library doesn’t support such functionality? There must be some algorithm to help us. Overview Because months have different number of days, and most of them aren’t divisible by 7 without a remainder, months begin on different days of the week. Thus, if January begins on Sunday, the month of February the same year will begin on Wednesday. Of course, in common years February has 28 days, which fortunately is divisible by 7 and thus February and March both begin on the same day, which is great, but isn’t true for leap years. What Do We Know About the Calendar First thing to know is that each week has exactly 7 days. We also know that a common year has 365 days, while a leap year has one day more – 366. Most of the months have 30 or 31 days, but February has only 28 days in common years and 29 in leap years. Because 365 mod 7 = 1 in a common year each year begins exactly on the next day of the preceding year. Thus if 2011 started on Saturday, 2012 starts on Sunday. And yet again, that is because 2011 is not a leap year. What else do we know? Because a week has exactly seven days only February (with its 28 days in a common year) is divisible by 7 (28 mod 7 = 0) and has exactly four weeks in it. Thus in a common year February and March start on a same day. Unfortunately that is not true about the other months. All these things we know about the calendar are great, so we can make some conclusions. Although eleven of the months have either 30 or 31 days they don’t start on a same day, but some of the months do appear to start on a same day just because the number of days between them is divisible by 7 without a remainder. Let’s take a look on some examples. For instance September has 30 days, as does November, while October, which is in between them has 31 days. Thus 30+30+31 makes 91. Fortunately 91 mod 7 = 0. So for each year September and December start on the same day (as they are after February they don’t depend on leap years). The same thing occurs to April and July and the good news is that in leap years even January starts on the same day as April and July. Now we know that there are some relations between months. Thus, if we know somehow that the 13th of April is Monday, we’ll be sure that 13th of July is also Monday. Let’s see now a summary of these observations. We can also refer to the following diagram. For leap years there are other corresponding months. Let’s take a look at the following image. Another way to get the same information is the following table. We also know that leap years happen to occur once every four years. However, if there is a common year like the year 2001, which will be the next year that is common and starts and corresponds exactly on 2001? Because of leap years we can have a year starting on one of the seven days of the week and to be either leap or common. This means just 14 combinations. Following these observations we can refer to the following table. You can clearly see the pattern “6 4 2 0” Here’s the month table. Columns 2 and 3 differs only for January and February. Clearly the day table is as follows: Now let’s go back to the algorithm. Using these tables and applying a simple formula, we can calculate what day was on some given date. Here are the steps of this algorithm. Get the number for the corresponding century from the centuries table; Get the last two digits from the year; Divide the number from step 2 by 4 and get it without the remainder; Get the month number from the month table; Sum the numbers from steps 1 to 4; Divide it by 7 and take the remainder; Find the result of step 6 in the days table; Implementation First let’s take a look at a simple and practical example of the example above and then the code. Let’s answer the question from the first paragraph of this post. What day was on January 31st, 1883? Take a look at the centuries table: for 1800 – 1899 this is 2. Get the last two digits from the year: 83. Divide 83 by 4 without a remainder: 83/4 = 20 Get the month number from the month table: Jan = 0. Sum the numbers from steps 1 to 4: 2 + 83 + 20 + 0 = 105. Divide it by 7 and take the remainder: 105 mod 7 = 0 Find the result of step 6 in the days table: Sunday = 0. The following code in PHP implements the algorithm above. function get_century_code($century) { // XVIII if (1700 <= $century && $century <= 1799) return 4; // XIX if (1800 <= $century && $century <= 1899) return 2; // XX if (1900 <= $century && $century <= 1999) return 0; // XXI if (2000 <= $century && $century <= 2099) return 6; // XXII if (2100 <= $century && $century <= 2199) return 4; // XXIII if (2200 <= $century && $century <= 2299) return 2; // XXIV if (2300 <= $century && $century <= 2399) return 0; // XXV if (2400 <= $century && $century <= 2499) return 6; // XXVI if (2500 <= $century && $century <= 2599) return 4; // XXVII if (2600 <= $century && $century <= 2699) return 2; } /** * Get the day of a given date * * @param $date */ function get_day_from_date($date) { $months = array( 1 => 0,// January 2 => 3,// February 3 => 3,// March 4 => 6,// April 5 => 1,// May 6 => 4,// June 7 => 6,// July 8 => 2,// August 9 => 5,// September 10 => 0,// October 11 => 3,// November 12 => 5,// December ); $days = array( 0 => 'Sunday', 1 => 'Monday', 2 => 'Tuesday', 3 => 'Wednesday', 4 => 'Thursday', 5 => 'Friday', 6 => 'Saturday', ); // calculate the date $dateParts = explode('-', $date); $century = substr($dateParts[2], 0, 2); $year = substr($dateParts[2], 2); // 1. Get the number for the corresponding century from the centuries table $a = get_century_code($dateParts[2]); // 2. Get the last two digits from the year $b = $year; // 3. Divide the number from step 2 by 4 and get it without the remainder $c = floor($year / 4); // 4. Get the month number from the month table $d = $months[$dateParts[1]]; // 5. Sum the numbers from steps 1 to 4 $e = $a + $b + $c + $d; // 6. Divide it by 7 and take the remainder $f = $e % 7; // 7. Find the result of step 6 in the days table return $days[$f]; } // Sunday echo get_day_from_date('31-1-1883'); Application This algorithm can be applied in many different cases although most of the libraries have built-in functions that can do that. The only problem besides that is that there are much more efficient algorithms that don’t need additional space (tables) of data. However this algorithm isn’t difficult to implement and it gives a good outlook of some facts in the calendar.

How to create a real-time face detection system using HTML5, JavaScript, and OpenCV, leveraging WebRTC for webcam access and WebSockets for client-server communication.

This post shows code examples in Python (2.7) for sending data to Graphite. Once you have a Graphite server setup, with Carbon running/collecting, you need to send it data for graphing. Basically, you write a program to collect numeric values and send them to Graphite's backend aggregator (Carbon). To send data, you create a socket connection to the graphite/carbon server and send a message (string) in the format: "metric_path value timestamp\n" `metric_path`: arbitrary namespace containing substrings delimited by dots. The most general name is at the left and the most specific is at the right. `value`: numeric value to store. `timestamp`: epoch time. messages must end with a trailing newline. multiple messages maybe be batched and sent in a single socket operation. each message is delimited by a newline, with a trailing newline at the end of the message batch. Example message: "foo.bar.baz 42 74857843\n" Let's look at some (Python 2.7) code for sending data to graphite... Here is a simple client that sends a single message to graphite. Code: #!/usr/bin/env python import socket import time CARBON_SERVER = '0.0.0.0' CARBON_PORT = 2003 message = 'foo.bar.baz 42 %d\n' % int(time.time()) print 'sending message:\n%s' % message sock = socket.socket() sock.connect((CARBON_SERVER, CARBON_PORT)) sock.sendall(message) sock.close() Here is a command line client that sends a single message to graphite: Usage: $ python client-cli.py metric_path value Code: #!/usr/bin/env python import argparse import socket import time CARBON_SERVER = '0.0.0.0' CARBON_PORT = 2003 parser = argparse.ArgumentParser() parser.add_argument('metric_path') parser.add_argument('value') args = parser.parse_args() if __name__ == '__main__': timestamp = int(time.time()) message = '%s %s %d\n' % (args.metric_path, args.value, timestamp) print 'sending message:\n%s' % message sock = socket.socket() sock.connect((CARBON_SERVER, CARBON_PORT)) sock.sendall(message) sock.close() Here is a client that collects load average (Linux-only) and sends a batch of 3 messages (1min/5min/15min loadavg) to graphite. It will run continuously in a loop until killed. (adjust the delay for faster/slower collection interval): #!/usr/bin/env python import platform import socket import time CARBON_SERVER = '0.0.0.0' CARBON_PORT = 2003 DELAY = 15 # secs def get_loadavgs(): with open('/proc/loadavg') as f: return f.read().strip().split()[:3] def send_msg(message): print 'sending message:\n%s' % message sock = socket.socket() sock.connect((CARBON_SERVER, CARBON_PORT)) sock.sendall(message) sock.close() if __name__ == '__main__': node = platform.node().replace('.', '-') while True: timestamp = int(time.time()) loadavgs = get_loadavgs() lines = [ 'system.%s.loadavg_1min %s %d' % (node, loadavgs[0], timestamp), 'system.%s.loadavg_5min %s %d' % (node, loadavgs[1], timestamp), 'system.%s.loadavg_15min %s %d' % (node, loadavgs[2], timestamp) ] message = '\n'.join(lines) + '\n' send_msg(message) time.sleep(DELAY) Resources: Graphite Docs Graphite Docs - Getting Your Data Into Graphite Installing Graphite 0.9.9 on Ubuntu 12.04 LTS Installing and configuring Graphite END