in continuation of my earlier blog jsoup: nice way to do html parsing in java, in this blog i will compare jsoup with other similar framework, htmlunit . apparently both of them are good html parsing frameworks and both can be used for web application unit testing and web scraping. in this blog, i will explain how htmlunit is better suited for web application unit testing automation and jsoup is better suited for web scraping. typically web application unit testing automation is a way to automate webtesting in junit framework. and web scraping is a way to extract unstructured information from the web to a structured format. i recently tried 2 decent web scraping tools, webharvy and mozenda . for any good html parsing tools to click, they should support either xpath based or css selector based element access. there are lot of blogs comparing each one like, why css locators are the way to go vs xpath , and css selectors and xpath expressions . htmlunit htmlunit is a powerful framework, where you can simulate pretty much anything a browser can do like click events, submit events etc and is ideal for web application automated unit testing. xpath based parsing is simple and most popular and htmlunit is heavily based on this. in one of my application, i wanted to extract information from the web in a structured way. htmlunit worked out very well for me on this. but the problem starts when you try to extract structured data from modern web applications that use jquery and other ajax features and use div tags extensively. htmlunit and other xpath based html parsers will not work with this. there is also a jsoup version that supports xpath based on jaxe n, i tried this as well, guess what? it also was not able to access the data from modern web applications like ebay.com. finally my experience with htmlunit was it was bit buggy or maybe i call it unforgiving unlike a browser, where in if the target web applications have missing javascripts, it will throw exceptions, but we can get around this, but out of the box it will not work. jsoup the latest version of jsoup goes extra length not to support xpath and will very well support css selectors . my experience was it is excellent for extracting structured data from modern web applications. it is also far forgiving if the web application has some missing javascripts. extracting xpath and css selector data in most of the browsers, if you point to an element and right click and click on “inspect element” it can extract the xpath information, i noticed firefox/firebug can also extract css selector path as shown below, htmlunit vs jsoup: extract css path and xpath in firebug i hope this blog helped.

This is a recurring question made by our MySQL Support customers: How can I audit the login attempts in MySQL? Logging all the attempts or just the failed ones is a very important task on some scenarios. Unfortunately there are not too many audit capabilities in MySQL Community so the first option to audit MySQL’s authentication process is to get all the information we need from logs. General Query Log The first option is the General Query Log. Let’s see an example: Enable the log: general_log_file = /var/log/mysql/mysql.log general_log = 1 User correctly authenticated: 121227 8:31:49 38 Connect root@localhost on 38 Query select @@version_comment limit 1 User not correctly authenticated: 121227 8:32:18 39 Connect root@localhost on 39 Connect Access denied for user 'root'@'localhost' (using password: YES) The problem of the General Query Log is that it will log everything so it can cause performance degradation and you will have to deal with very large files on high loaded servers. general_log variable is dynamic so a solution could be enabling and disabling the log just when it’s needed. Error log If you only care about failed attempts to login then there is another different and less problematic approach. From 5.5 it’s possible to log access denied messages to the error log. We just need to enable log_warnings with a value greater than 1: log_warnings = 2 Then check the error log: 121227 8:44:21 [Warning] Access denied for user 'root'@'localhost' (using password: YES) User Statistics If you are using Percona Server then there is a third option to get information about our users, the User Statistics. As with the previous options we can get the number of connections and failed connections made by a particular user but not the date and time of those attempts. Besides that information we can get other statistics that can be very useful if MySQL is running on a multi-tenant environment or we need to control how resources are used. Let’s seen an example, first we enable User Statistics in my.cnf: 5.5 userstat = 1 5.1 userstat_running = 1 Then we get the information about a particular user: mysql> select * from user_statistics where user='root'\G *************************** 1. row *************************** USER: root TOTAL_CONNECTIONS: 25 CONCURRENT_CONNECTIONS: 0 CONNECTED_TIME: 464 BUSY_TIME: 96 CPU_TIME: 19 BYTES_RECEIVED: 62869617 BYTES_SENT: 14520 BINLOG_BYTES_WRITTEN: 0 ROWS_FETCHED: 783051 ROWS_UPDATED: 1017714 TABLE_ROWS_READ: 1484751 SELECT_COMMANDS: 14 UPDATE_COMMANDS: 103 OTHER_COMMANDS: 3556 COMMIT_TRANSACTIONS: 0 ROLLBACK_TRANSACTIONS: 0 DENIED_CONNECTIONS: 2 LOST_CONNECTIONS: 16 ACCESS_DENIED: 0 EMPTY_QUERIES: 0 TOTAL_SSL_CONNECTIONS: 0 Here we can see that root has done 25 total connections. Two denied connections (bad password) and 16 lost connections (not closed properly). Apart from that information we get the connection time, bytes received and sent, rows accessed, commands executed and so on. Very valuable information. It is important to mention that these tables are stored in INFORMATION_SCHEMA and that means that after a mysqld restart all the information will be lost. So if you really need that information you should copy it to another table or export to a csv for further analysis. Conclusion We don’t have too many audit capabilities in MySQL Community so logging all events and then filter them with custom-made scripts is the best solution we have nowadays. If you are using Percona Server you can get more detailed information about what a particular user is doing. All options can be combined to meet your needs.

In a previous post I introduced how the @XmlAnyElement(lax=true) annotation could be used to create a very flexible mapping. In this post I'll dig a bit deeper into how it relates to both the @XmlRootElement and @XmlElementDecl. @XmlAnyElement (Root) The any field is annotated with @XmlAnyElement(lax=true). This means that for that field if an element is associated with a class via @XmlRootElement or @XmlElementDecl then an instance of the corresponding class will be used to populate the field if not the element will be set as an instance of org.w3c.dom.Element. package blog.anyelement; import java.util.List; import javax.xml.bind.annotation.*; @XmlRootElement @XmlAccessorType(XmlAccessType.FIELD) public class Root { @XmlAnyElement(lax = true) protected List createBar(Bar bar) { return new JAXBElement(new QName("bar"), Bar.class, bar); } } Input (input.xml) Below is the input document we'll use for this example. There are 3 elements that correspond to the any property. The first corresponds to the @XmlRootElement annotation on the Foo class. The second corresponds to the @XmlElementDecl annotation for the Bar class and the third does not correspond to any of the domain classes. Hello World A B Demo In the demo code below we will unmarshal the input document, then output the classes of the objects in the resulting any property and then marshal the payload object back to XML. package blog.anyelement; import java.io.File; import javax.xml.bind.*; public class Demo { public static void main(String[] args) throws Exception { JAXBContext jc = JAXBContext.newInstance(Root.class, Foo.class, ObjectFactory.class); Unmarshaller unmarshaller = jc.createUnmarshaller(); File xml = new File("src/blog/anyelement/input.xml"); Root payload = (Root) unmarshaller.unmarshal(xml); for(Object o : payload.any) { System.out.println(o.getClass()); } Marshaller marshaller = jc.createMarshaller(); marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); marshaller.marshal(payload, System.out); } } Output Below is the output from running the demo code. Note the classes corresponding to the objects in the any property. The foo element became an instance of the Foo class. The bar element became an instance of JAXBElement that holds an instance of Bar. The other element became an instance of org.w3c.dom.Element. class blog.anyelement.Foo class javax.xml.bind.JAXBElement class com.sun.org.apache.xerces.internal.dom.ElementNSImpl Hello World A B

Find out how MongoDB ranks with atomicity, consistency, isolation and durability (ACID).

It is easy to do - a few button clicks (generally) - but the button location is damn unintuitive. So, this is what you have got to do Go to "Help" menu item. Click on "About ..." button (why on earth should I click that when I am trying to un-install a plugin. By the way, the menu item just above "About ..." is "Install New Software ...". Would it have been too much pain to have a "Manage plugins" and / or "Un-install plugins" right underneath it?) A form opens up. At the bottom of it there is button "Installation details". Click that. (Again, why on earth would anyone think "Installation details" would have anything to do with un-installing stuff. I would have expected only a static display of stuff that are already installed.) Another multi tabbed form opens up (Anyone keeping count of the number of windows opened already. This is the 3rd window by now, including the parent editor window) which shows all the installed plugins. If you select any of the installed plugins, a button to "uninstall" becomes available. Click that and you should be able to un-install and after a restart everything should be fine. My interest in software and IT has always been much more than a 9 to 5 job (and I am sure there is a huge population that it holds equally true for). I have always wanted software to be efficient and beautiful apart from doing it's job. However, it took an excellent session on usability (which I joined only with casual curiosity but left with renewed interest in the subject and admiration for David Travis who delivered the course) to get me to start looking at all software with an "user's" perspective. And I was surprised with what I found and how it changed my coding. I have been using Eclipse and STS for years now (nearing a decade now) and I absolutely love these software. However when you start looking at them as a "user" and not only as a developer, there are quite a few usability opportunities of improvement that meets the eye. This article - apart from helping folks looking to un-install plugins in Eclipse - is also intended at folks who design Eclipse - just a humble request to consider this also as a usability improvement.

Why would we use a multi threaded application vs. a single threaded application? First we must define multithreading. Multithreading is a feature of an operating system that allows programs to run subcomponents or threads in parallel. Typically most applications only need to use one thread because they do not perform time consuming tasks. The use of multiple threads allows an application to distribute long running tasks so that they can be executed in parallel. This gives the user the perceived appearance that the application is working faster due to the fact that while one thread is waiting on an IO process the remaining tasks can make use of the available CPU. The allows working threads to execute in tandem so that they can be competed sooner. Multithreading Benefits Improved responsiveness — Users usually report improved responsiveness compared to single thread applications. Faster applications — Multiple threads can lead to improved application performance. Prioritization — Threads can be assigned a priority which would allow higher priority tasks to take precedence over lower priority tasks. Single Threading Benefits Programming and debugging —These activities are easier compared to multithreaded applications due to the reduced complexity Less Overhead — Threads add overhead to an application When developing multi-threaded applications, the following must be considered. Deadlocks occur when two threads hold a monitor that the other one requires. In essence each task is blocking the other and both tasks are waiting for the other monitor to be released. This forces an application to hang or deadlock. Resource allocation is used to prevent deadlocks because the system determines if approving the resource request will render the system in an unsafe state. An unsafe state could result in a deadlock. The system only approves requests that will lead to safe states. Thread Synchronization is used when multiple threads use the same instance of an object. The threads accessing the object can then be locked and then synchronized so that each task can interact with the static object on at a time.

Learn all about this Unified Modeling Language diagram in Java.

This article will provide you with a tutorial allowing you to determine how much and where Java heap space is retained from your active application Java threads. A true case study from an Oracle Weblogic 10.0 production environment will be presented in order for you to better understand the analysis process. We will also attempt to demonstrate that excessive garbage collection or Java heap space memory footprint problems are often not caused by true memory leaks but instead due to thread execution patterns and high amount of short lived objects. Background As you may have seen from my past JVM overview article, Java threads are part of the JVM fundamentals. Your Java heap space memory footprint is driven not only by static and long lived objects but also by short lived objects. OutOfMemoryError problems are often wrongly assumed to be due to memory leaks. We often overlook faulty thread execution patterns and short lived objects they “retain” on the Java heap until their executions are completed. In this problematic scenario: Your “expected” application short lived / stateless objects (XML, JSON data payload etc.) become retained by the threads for too long (thread lock contention, huge data payload, slow response time from remote system etc.) Eventually such short lived objects get promoted to the long lived object space e.g. OldGen/tenured space by the garbage collector As a side effect, this is causing the OldGen space to fill up rapidly, increasing the Full GC (major collections) frequency Depending of the severity of the situation this can lead to excessive GC garbage collection, increased JVM paused time and ultimately OutOfMemoryError: Java heap space Your application is now down, you are now puzzled on what is going on Finally, you are thinking to either increase the Java heap or look for memory leaks…are you really on the right track? In the above scenario, you need to look at the thread execution patterns and determine how much memory each of them retain at a given time. OK I get the picture but what about the thread stack size? It is very important to avoid any confusion between thread stack size and Java memory retention. The thread stack size is a special memory space used by the JVM to store each method call. When a thread calls method A, it “pushes” the call onto the stack. If method A calls method B, it gets also pushed onto the stack. Once the method execution completes, the call is “popped” off the stack. The Java objects created as a result of such thread method calls are allocated on the Java heap space. Increasing the thread stack size will definitely not have any effect. Tuning of the thread stack size is normally required when dealing with java.lang.stackoverflowerror or OutOfMemoryError: unable to create new native thread problems. Case study and problem context The following analysis is based on a true production problem we investigated recently. Severe performance degradation was observed from a Weblogic 10.0 production environment following some changes to the user web interface (using Google Web Toolkit and JSON as data payload) Initial analysis did reveal several occurrences of OutOfMemoryError: Java heap space errors along with excessive garbage collection. Java heap dump files were generated automatically (-XX:+HeapDumpOnOutOfMemoryError) following OOM events Analysis of the verbose:gc logs did confirm full depletion of the 32-bit HotSpot JVM OldGen space (1 GB capacity) Thread dump snapshots were also generated before and during the problem The only problem mitigation available at that time was to restart the affected Weblogic server when problem was observed A rollback of the changes was eventually performed which did resolve the situation The team first suspected a memory leak problem from the new code introduced. Thread dump analysis: looking for suspects… The first analysis step we did was to perform an analysis of the generated thread dump data. Thread dump will often show you the culprit threads allocating memory on the Java heap. It will also reveal any hogging or stuck thread attempting to send and receive data payload from a remote system. The first pattern we noticed was a good correlation between OOM events and STUCK threads observed from the Weblogic managed servers (JVM processes). Find below the primary thread pattern found: <10-Dec-2012 1:27:59 o'clock PM EST> <[STUCK] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "672" seconds working on the request which is more than the configured time of "600" seconds. As you can see, the above thread appears to be STUCK or taking very long time to read and receive the JSON response from the remote server. Once we found that pattern, the next step was to correlate this finding with the JVM heap dump analysis and determine how much memory these stuck threads were taking from the Java heap. Heap dump analysis: retained objects exposed! The Java heap dump analysis was performed using MAT. We will now list the different analysis steps which did allow us to pinpoint the retained memory size and source. 1. Load the HotSpot JVM heap dump 2. Select the HISTOGRAM view and filter by “ExecuteThread” * ExecuteThread is the Java class used by the Weblogic kernel for thread creation & execution * As you can see, this view was quite revealing. We can see a total of 210 Weblogic threads created. The total retained memory footprint from these threads is 806 MB. This is pretty significant for a 32-bit JVM process with 1 GB OldGen space. This view alone is telling us that the core of the problem and memory retention originates from the threads themselves. 3. Deep dive into the thread memory footprint analysis The next step was to deep dive into the thread memory retention. To do this, simply right click over the ExecuteThread class and select: List objects > with outgoing references. As you can see, we were able to correlate STUCK threads from the thread dump analysis with high memory retention from the heap dump analysis. The finding was quite surprising. 3. Thread Java Local variables identification The final analysis step did require us to expand a few thread samples and understand the primary source of memory retention. As you can see, this last analysis step did reveal huge JSON response data payload at the root cause. That pattern was also exposed earlier via the thread dump analysis where we found a few threads taking very long time to read & receive the JSON response; a clear symptom of huge data payload footprint. It is crucial to note that short lived objects created via local method variables will show up in the heap dump analysis. However, some of those will only be visible from their parent threads since they are not referenced by other objects, like in this case. You will also need to analyze the thread stack trace in order to identify the true caller, followed by a code review to confirm the root cause. Following this finding, our delivery team was able to determine that the recent JSON faulty code changes were generating, under some scenarios, huge JSON data payload up to 45 MB+. Given the fact that this environment is using a 32-bit JVM with only 1 GB of OldGen space, you can understand that only a few threads were enough to trigger severe performance degradation. This case study is clearly showing the importance of proper capacity planning and Java heap analysis, including the memory retained from your active application & Java EE container threads. Learning is experience. Everything else is just information I hope this article has helped you understand how you can pinpoint the Java heap memory footprint retained by your active threads by combining thread dump and heap dump analysis. Now, this article will remain just words if you don’t experiment so I highly recommend that you take some time to learn this analysis process yourself for your application(s). Please feel free to post any comment or question.

Hi! Merry Christmas everyone :) Quite recently I've seen a nice presentation about Google Guava and we came to the conclusion in our project that it could be really interesting to use the its Cache functionallity. Let us take a look at the regexp Pattern class and its compile function. Quite often in the code we can see that each time a regular expression is being used a programmer is repeatidly calling the aforementioned Pattern.compile() function with the same argument thus compiling the same regular expression over and over again. What could be done however is to cache the result of such compilations - let us take a look at the RegexpUtils utility class: RegexpUtils.java package pl.grzejszczak.marcin.guava.cache.utils; import com.google.common.cache.CacheBuilder; import com.google.common.cache.CacheLoader; import com.google.common.cache.LoadingCache; import java.util.concurrent.ExecutionException; import java.util.regex.Matcher; import java.util.regex.Pattern; import static java.lang.String.format; public final class RegexpUtils { private RegexpUtils() { throw new UnsupportedOperationException("RegexpUtils is a utility class - don't instantiate it!"); } private static final LoadingCache COMPILED_PATTERNS = CacheBuilder.newBuilder().build(new CacheLoader() { @Override public Pattern load(String regexp) throws Exception { return Pattern.compile(regexp); } }); public static Pattern getPattern(String regexp) { try { return COMPILED_PATTERNS.get(regexp); } catch (ExecutionException e) { throw new RuntimeException(format("Error when getting a pattern [%s] from cache", regexp), e); } } public static boolean matches(String stringToCheck, String regexp) { return doGetMatcher(stringToCheck, regexp).matches(); } public static Matcher getMatcher(String stringToCheck, String regexp) { return doGetMatcher(stringToCheck, regexp); } private static Matcher doGetMatcher(String stringToCheck, String regexp) { Pattern pattern = getPattern(regexp); return pattern.matcher(stringToCheck); } } As you can see the Guava's LoadingCache with the CacheBuilder is being used to populate a cache with a new compiled pattern if one is not found. Due to caching the compiled pattern if a compilation has already taken place it will not be repeated ever again (in our case since we dno't have any expiry set). Now a simple test GuavaCache.java package pl.grzejszczak.marcin.guava.cache; import com.google.common.base.Stopwatch; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import pl.grzejszczak.marcin.guava.cache.utils.RegexpUtils; import java.util.regex.Pattern; import static java.lang.String.format; public class GuavaCache { private static final Logger LOGGER = LoggerFactory.getLogger(GuavaCache.class); public static final String STRING_TO_MATCH = "something"; public static void main(String[] args) { runTestForManualCompilationAndOneUsingCache(1); runTestForManualCompilationAndOneUsingCache(10); runTestForManualCompilationAndOneUsingCache(100); runTestForManualCompilationAndOneUsingCache(1000); runTestForManualCompilationAndOneUsingCache(10000); runTestForManualCompilationAndOneUsingCache(100000); runTestForManualCompilationAndOneUsingCache(1000000); } private static void runTestForManualCompilationAndOneUsingCache(int firstNoOfRepetitions) { repeatManualCompilation(firstNoOfRepetitions); repeatCompilationWithCache(firstNoOfRepetitions); } private static void repeatManualCompilation(int noOfRepetitions) { Stopwatch stopwatch = new Stopwatch().start(); compileAndMatchPatternManually(noOfRepetitions); LOGGER.debug(format("Time needed to compile and check regexp expression [%d] ms, no of iterations [%d]", stopwatch.elapsedMillis(), noOfRepetitions)); } private static void repeatCompilationWithCache(int noOfRepetitions) { Stopwatch stopwatch = new Stopwatch().start(); compileAndMatchPatternUsingCache(noOfRepetitions); LOGGER.debug(format("Time needed to compile and check regexp expression using Cache [%d] ms, no of iterations [%d]", stopwatch.elapsedMillis(), noOfRepetitions)); } private static void compileAndMatchPatternManually(int limit) { for (int i = 0; i < limit; i++) { Pattern.compile("something").matcher(STRING_TO_MATCH).matches(); Pattern.compile("something1").matcher(STRING_TO_MATCH).matches(); Pattern.compile("something2").matcher(STRING_TO_MATCH).matches(); Pattern.compile("something3").matcher(STRING_TO_MATCH).matches(); Pattern.compile("something4").matcher(STRING_TO_MATCH).matches(); Pattern.compile("something5").matcher(STRING_TO_MATCH).matches(); Pattern.compile("something6").matcher(STRING_TO_MATCH).matches(); Pattern.compile("something7").matcher(STRING_TO_MATCH).matches(); Pattern.compile("something8").matcher(STRING_TO_MATCH).matches(); Pattern.compile("something9").matcher(STRING_TO_MATCH).matches(); } } private static void compileAndMatchPatternUsingCache(int limit) { for (int i = 0; i < limit; i++) { RegexpUtils.matches(STRING_TO_MATCH, "something"); RegexpUtils.matches(STRING_TO_MATCH, "something1"); RegexpUtils.matches(STRING_TO_MATCH, "something2"); RegexpUtils.matches(STRING_TO_MATCH, "something3"); RegexpUtils.matches(STRING_TO_MATCH, "something4"); RegexpUtils.matches(STRING_TO_MATCH, "something5"); RegexpUtils.matches(STRING_TO_MATCH, "something6"); RegexpUtils.matches(STRING_TO_MATCH, "something7"); RegexpUtils.matches(STRING_TO_MATCH, "something8"); RegexpUtils.matches(STRING_TO_MATCH, "something9"); } } } We are running a series of tests and checking the time of their execution. Note that the results of these tests are not precise due to the fact that the application is not being run in isolation so numerous conditions can affect the time of the execution. We are interested in showing some degree of the problem rather than showing the precise execution time. For a given number of iterations (1,10,100,1000,10000,100000,1000000) we are either compiling 10 regular expressions or using a Guava's cache to retrieve the compiled Pattern and then we match them against a string to match. These are the logs: pl.grzejszczak.marcin.guava.cache.GuavaCache:34 Time needed to compile and check regexp expression [1] ms, no of iterations [1] pl.grzejszczak.marcin.guava.cache.GuavaCache:40 Time needed to compile and check regexp expression using Cache [35] ms, no of iterations [1] pl.grzejszczak.marcin.guava.cache.GuavaCache:34 Time needed to compile and check regexp expression [1] ms, no of iterations [10] pl.grzejszczak.marcin.guava.cache.GuavaCache:40 Time needed to compile and check regexp expression using Cache [0] ms, no of iterations [10] pl.grzejszczak.marcin.guava.cache.GuavaCache:34 Time needed to compile and check regexp expression [8] ms, no of iterations [100] pl.grzejszczak.marcin.guava.cache.GuavaCache:40 Time needed to compile and check regexp expression using Cache [3] ms, no of iterations [100] pl.grzejszczak.marcin.guava.cache.GuavaCache:34 Time needed to compile and check regexp expression [10] ms, no of iterations [1000] pl.grzejszczak.marcin.guava.cache.GuavaCache:40 Time needed to compile and check regexp expression using Cache [10] ms, no of iterations [1000] pl.grzejszczak.marcin.guava.cache.GuavaCache:34 Time needed to compile and check regexp expression [83] ms, no of iterations [10000] pl.grzejszczak.marcin.guava.cache.GuavaCache:40 Time needed to compile and check regexp expression using Cache [33] ms, no of iterations [10000] pl.grzejszczak.marcin.guava.cache.GuavaCache:34 Time needed to compile and check regexp expression [800] ms, no of iterations [100000] pl.grzejszczak.marcin.guava.cache.GuavaCache:40 Time needed to compile and check regexp expression using Cache [279] ms, no of iterations [100000] pl.grzejszczak.marcin.guava.cache.GuavaCache:34 Time needed to compile and check regexp expression [7562] ms, no of iterations [1000000] pl.grzejszczak.marcin.guava.cache.GuavaCache:40 Time needed to compile and check regexp expression using Cache [3067] ms, no of iterations [1000000] You can find the sources over here under the Guava/Cache directory or go to the url https://bitbucket.org/gregorin1987/too-much-coding/src

for those that are not familiar with sonar , ( i hope this post will make you at least try it or see it in action at http://nemo.sonarsource.org ) you can take a look at an earlier post i’ve written some time ago. in one sentence sonar is an open source platform that allows you to track and improve the quality of your source code. one of the key aspects when talking about software quality is the test coverage or code coverage which is how much of your source code is tested by unit tests. sonar integrates with the most popular open source code coverage tools ( jacoco , cobetura , emma ) and the well-known commercial clover by attlassian. by default it uses the jacoco (java code coverage) engine and you’ll shortly find out why before we move on, i’d like to give many kudos to evgeny mandrikov . this article is inspired by one of his older post s and its intention is to present a more updated comparison of the supported code coverage tools by sonar and point out some differences regarding their results and the way they work. recently sonar changed its default code coverage tool to jacoco and this post tries to explain the reasons behind that decision. some of the information is borrowed by evgeny’s post and the image is also taken from evgeny’s presentation about jacoco . so thanks a lot evgeny! now let’s go to the meat. for the comparison you’ll see, i’ve used the latest available sonar version 3.3, maven 2.2.1, java 1.6 and all analysis launched in a windows 7 machine (intel core i3-2120 cpu @ 3.30ghz) with 8gb ram. the projects were carefully selected ( a small, medium-sized and a large one – not that large as java code base but large enough to extract some results ). i ran five analysis for each open source code coverage tool ( i excluded the commercial clover from my comparison version ) and another five by disabling the code coverage mechanism. so that’s a total of 60 analysis ). in the following tables you can find some information about the code coverage tools and some basic metrics about the selected projects. pay attention to the date of the latest stable release. emma hasn’t been updated since dinosaurs era and cobertura is almost three years inactive. one might think that this isn’t an issue if they are stable and don’t need any new release. well, the truth is that both of them have bugs that frustrate end-users and there’s no one to fix them. on the other hand jacoco is continuously evolving and improving… the results of the analysis are displayed next. some important notices. emma doesn’t support branch coverage that’s why you’re not seeing any metrics. furthermore there are differences in the results of line and branch coverage, which are more concrete for larger projects. for instance in sonar jira plugin all three tools produce the same results whereas in sonar analysis and commons lang projects you can see that the numbers are not the same. now take a look at a graph that illustrates in a more readable way which tool is the fastest. it seems that emma and jacoco need the same amount of time to compute their metrics… but… as we already mentioned there’s a huge difference. there’s no branch coverage in emma reports. cobertura is always slower than jacoco so again the winner is jacoco. of course you can get even faster results by running a sonar analysis without computing code coverage metrics one last thing: jacoco, as the following figure shows is the only tool that analyses bytecode on-the-fly which is more . cobertura and emma run an offline analysis and use a class loader whereas jacoco has its own java agent for analysis code. this configuration allows jacoco to be very flexible, possible integrated with many other tools and frameworks and can be used with any language in a jvm environment. so, to sum up, if you’re using sonar ( if you don’t , you should ), then it strongly advisable to keep the default code coverage engine ( jacoco) , unless you have really important reasons for that. finally don’t forge to check sonar’s community 2013 unofficial survey and the upcoming book about sonar by manning publications. the release date is in about 3-4 months but you can get an early access version here . as always, feel free to comment or suggest improvements about the article and its content.

This article will describe the complete root cause analysis and solution of a HashMap High CPU problem (infinite looping) affecting a Weblogic 10.0 environment running on the Java HotSpot VM 1.5. This case study will again demonstrate this importance of mastering Thread Dump analysis skill and CPU correlation techniques such as Solaris prstat. Environment specifications Java EE server: Oracle Weblogic Portal 10.0 Middleware OS: Solaris 10 Java VM: Java HotSpot VM 1.5 Platform type: Portal application Monitoring and troubleshooting tools JVM Thread Dump (HotSpot format) Solaris prstat (CPU contributors analysis) Problem overview Problem type: High CPUobserved from our Weblogic production environment A high CPU problem was observed from our Solaris physical servers hosting a Weblogic Portal 10 environment. Users also reporting major slowdown of the portal application. Gathering and validation of facts As usual, a Java EE problem investigation requires gathering of technical and non-technical facts so we can either derived other facts and/or conclude on the root cause. Before applying a corrective measure, the facts below were verified in order to conclude on the root cause: What is the client impact? HIGH Recent change of the affected platform? No Any recent traffic increase to the affected platform? Yes How does this high CPU manifest itself? A sudden CPU increase was observed and is not going down; even after load goes down e.g. near zero level. Did an Oracle OSB recycle resolve the problem? Yes, but problem is returning after few hours or few days (unpredictable pattern) - Conclusion #1: The high CPU problem appears to be intermittent vs. pure correlation with load - Conclusion #2: Since high CPU remains after load goes down, this typically indicates either the presence of some infinite looping or heavy processing Threads Solaris CPU analysis using prstat Solaris prstat is a powerful OS command allowing you to obtain the CPU per process but more importantly CPU per Thread within a process. As you can see below from our case study, the CPU utilization was confirmed to go up as high as 100% utilization (saturation level). ## PRSTAT (CPU per Java Thread analysis) prstat -L -p 8223 1 1 PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID 8223 bea10 2809M 2592M sleep 59 0 14:52:59 38.6%java/494 8223 bea10 2809M 2592M sleep 57 0 12:28:05 22.3% java/325 8223 bea10 2809M 2592M sleep 59 0 11:52:02 28.3% java/412 8223 bea10 2809M 2592M sleep 59 0 5:50:00 0.3% java/84 8223 bea10 2809M 2592M sleep 58 0 2:27:20 0.2% java/43 8223 bea10 2809M 2592M sleep 59 0 1:39:42 0.2% java/41287 8223 bea10 2809M 2592M sleep 59 0 4:41:44 0.2% java/30503 8223 bea10 2809M 2592M sleep 59 0 5:58:32 0.2% java/36116 …………………………………………………………………………………… As you can see from above data, 3 Java Threads were found using together close to 100% of the CPU utilization. For our root cause analysis, we did focus on Thread #494 (decimal format) corresponding to 0x1ee (HEXA format). Thread Dump analysis and PRSTAT correlation Once the culprit Threads were identified, the next step was to correlate this data with the Thread Dump data (which was captured exactly at the same time as prstat). A quick search within the generated Thread Dump file did reveal the Thread Stack Trace (Weblogic Stuck Thread #125) for 0x1ee as per below. "[STUCK] ExecuteThread: '125' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=1 tid=0x014c5030 nid=0x1ee runnable [0x536fb000..0x536ffc70] at java.util.HashMap.get(HashMap.java:346) at org.apache.axis.encoding.TypeMappingImpl.getClassForQName(TypeMappingImpl.java:715) at org.apache.axis.encoding.TypeMappingDelegate.getClassForQName(TypeMappingDelegate.java:170) at org.apache.axis.encoding.TypeMappingDelegate.getClassForQName(TypeMappingDelegate.java:160) at org.apache.axis.encoding.TypeMappingImpl.getDeserializer(TypeMappingImpl.java:454) at org.apache.axis.encoding.TypeMappingDelegate.getDeserializer(TypeMappingDelegate.java:108) at org.apache.axis.encoding.TypeMappingDelegate.getDeserializer(TypeMappingDelegate.java:102) at org.apache.axis.encoding.DeserializationContext.getDeserializer(DeserializationContext.java:457) at org.apache.axis.encoding.DeserializationContext.getDeserializerForType(DeserializationContext.java:547) at org.apache.axis.encoding.ser.BeanDeserializer.getDeserializer(BeanDeserializer.java:514) at org.apache.axis.encoding.ser.BeanDeserializer.onStartChild(BeanDeserializer.java:286) at org.apache.axis.encoding.DeserializationContext.startElement(DeserializationContext.java:1035) at org.apache.axis.message.SAX2EventRecorder.replay(SAX2EventRecorder.java:165) at org.apache.axis.message.MessageElement.publishToHandler(MessageElement.java:1141) at org.apache.axis.message.RPCElement.deserialize(RPCElement.java:236) at org.apache.axis.message.RPCElement.getParams(RPCElement.java:384) at org.apache.axis.client.Call.invoke(Call.java:2467) at org.apache.axis.client.Call.invoke(Call.java:2366) at org.apache.axis.client.Call.invoke(Call.java:1812) Thread Dump analysis – HashMap.get() infinite loop condition! As you can see from the above Thread Stack Trace, the Thread is currently stuck in an infinite loop over a java.util.HashMap that originates from the Apache Axis TypeMappingImpl Java class. This finding was quite revealing. The 2 others Threads using high CPU also did reveal infinite looping condition within the same Apache Axis HashMap Object. Root cause: non Thread safe HashMap in Apache Axis 1.4 Additional research did reveal this known defect affecting Apache Axis 1.4; which is the version that our application was using. As you may already know, usage of non Thread safe / non synchronized HashMap under concurrent Threads condition is very dangerous and can easily lead to internal HashMap index corruption and / or infinite looping. This is also a golden rule for any middleware software such as Oracle Weblogic, IBM WAS, Red Hat JBoss which rely heavily on HashMap data structures from various Java EE and caching services. Such best practice is also applicable for any Open Source third party API such as Apache Axis. The most common solution is to use the ConcurrentHashMap data structure which is designed for that type of concurrent Thread execution context. Solution Our team did apply the proposed patch from Apache (synchronize the non Thread safe HashMap) which did resolve the problem. We are also currently looking at upgrading our application to a newer version of Apache Axis. Conclusion I hope this case study has helped you understand how to pinpoint the root cause of high CPU Threads and the importance of proper Thread safe data structure for high concurrent Thread / processing applications. Please don’t hesitate to post any comment or question. Find office supplies promo codes to save money for your department's bottom line, so you can spend more on the latest software.

This is the first post of a new series that will bring you frequent and short “question of the day” articles. This new format will complement my existing writing and is designed to provide you with fast answers for common problems and questions that I often get from work colleagues, IT clients and readers. Each of these QOTD posts will be archived and you will have access to the entire list via a separate page. I also encourage you to post your feedback, recommendations along with your own complement answer for each question. Now let’s get started with our first question of the day! Question: What is the difference between Oracle Weblogic thread states & attributes (Total, Standby, Active, idle, Hogging, Stuck)? Answer: Weblogic thread state can be a particularly confusing topic for Weblogic administrators and individuals starting to learn and monitor Weblogic threads. The thread monitoring section can be accessed for each managed server Managed server under the Monitoring > Threads tab. As you can see, the thread monitoring tab provides a complete view of each Weblogic thread along with its state. Now let’s review each section and state so you can properly understand how to assess the health. # Summary section Execute Thread Total Count: This is the total number of threads “created” from the Weblogic self-tuning pool and visible from the JVM Thread Dump. This value correspond to the sum of: Active + Standby threads Active Execute Threads: This is the number of threads “eligible” to process a request. When thread demand goes up, Weblogic will start promoting threads from Standby to Active state which will enable them to process future client requests Standby Thread Count: This is the number of threads waiting to be marked “eligible” to process client requests. These threads are created and visible from the JVM Thread Dump but not available yet to process a client request Hogging Thread Count: This is the number of threads taking much more time than the current execution time in average calculated by the Weblogic kernel Execute Thread Idle Count: This is the number of Active threads currently “available” to process a client request In the above snapshots, we have: Total of 43 threads, 29 in Standby state and 14 in Active state Out of the 14 Active threads, we have 1 Hogging thread and 7 Idle threads e.g. 7 threads “available” for request processing Another way to see the situation: we have a total of 7 threads currently “processing” client request with 1 out of 7 in Hogging state (e.g. taking more time than current calculated average) # Thread matrix This matrix gives you a view of each thread along with its current state. There is one more state that you must also understand: STUCK: A stuck thread is identified by Weblogic when it is taking more time than the configured stuck thread time (default is 600 seconds). When facing slowdown conditions, you will normally see Weblogic threads transitioning from the Hogging state followed by STUCK, depending how long these threads remain stuck executing their current request

in this article i deal with a different nosql database called mongodb a mature nosql engine born outside the .net world to clarify the concept of id in a typical no sql database. installation of mongo is really simple, just download, uncompress, locate the bin folder, and type this from an administrator console prompt to install mongo as service 1: mongod --install --logpath c:xxxx --dbpath c:yyyy you can find plenty of installation guide on the internet, but with the above install command you create a windows service that will automatically start mongodb on your machine using specified datafolder. now you should download the c# driver to connect from .net code, but if you like using linq you can install fluent mongo directly with nuget. figure 1: install fluent mongo with nuget. fluent mongo is a library that gave little linq capability over standard drivers, but adding a nuget reference to fluentmongo you get automatically a reference to the official drivers. now you are ready to insert your first record in mongodb with the above code. mongoserver server = mongoserver.create(); mongodatabase databasetest = server.getdatabase("test"); var untyped = databasetest.getcollection("untyped"); untyped.save(new bsondocument { { "name", "untyped1" } }); bsondocument seconddocument = bsondocument.parse("{name: 'untyped2', blabla: 'bla bla value'}"); untyped.save(seconddocument); in line1 i create a connection to mongodb server passing no parameter to connect to local mongodb server, then i obtain a reference to a mongodatabase object called “test” with mongoserver::getdatabase() method and finally i get a reference to a collection named “untyped” with the mongngodatabase::getcollection () method. this is quite similar to a sql server or other sql database, you have a server, the server contains several databases, and each database is composed by tables; in the same way mongo is divided into server/database/collection where a collection contains document. mongodb stores data in json format and to insert data inside a collection you can simply create a bsondocument, an object defined by c# driver assembly that is capable to represent a document composed by a series of key-value pair. to initialize a bsondocument you can pass a icollection (line 4) or if you feel more confortable with string json representation, you can user bsondocument.parse() to specify the document directly with a json string. after you inserted the above documents you can use mongovue to see what is contained inside the database. figure 2: use mongovue to see what is inside the database the interesting aspect is that each document has an unique id, even if i did not specified any special property in the code. this is a standard behavior for nosql databases, if you did not specify any id property the database engine will create unique id on his own to idendify the docuemnt. the id is a key factor for mongo and other nosql storage, if you try to store a document directly inside the collection specifying json content you will get an error. untyped.save("{name: 'json', attribute:'attribute content'}"); the mongocollection object contains a save object that accepts a string, but the above call will fail with the error subclass must implement getdocumentid . previous code works because one of the specific functionality that a bsondocument implements is the ability to manage id generation, but plain json does not have this capability. if you need to know the id generated by the database, you can query the bsondocument for its unique id after it was saved in a mongo collection . (remember that the id is not available until you save the document). bsondocument seconddocument = bsondocument.parse("{name: 'untyped2', blabla: 'bla bla value'}"); object id; type idtype; iidgenerator generator; untyped.save(seconddocument); seconddocument.getdocumentid(out id, out idtype, out generator); basically you are asking to your bsondocument to return you the generated id, as well as the type of the id and the generator that mongo used to generate that specific id. the result is represented into this snippet figure 3: the three object that you got with a call to getdocumentid: id, idtype and the generator. as you can see the id is an instance of type mongodb.bson.objectid, based on bsonvalue base class and the generator is an instance of objectidgenerator. this type of id is specific to mongo, and the documentation states that an objectid is a bson objectid is a 12-byte value consisting of a 4-byte timestamp (seconds since epoch), a 3-byte machine id, a 2-byte process id, and a 3-byte counter. note that the timestamp and counter fields must be stored big endian unlike the rest of bson. this is because they are compared byte-by-byte and we want to ensure a mostly increasing order. if you want to have a generator that creates integer id, like identity column in sql server , you will find that it is simply not available out of the box, because an int value is not guarantee to be unique if you use sharding. sharding is a technique that permits to partition data into different physical instances, so each instance should generates ids that are unique across all instances and this prevents the use of a simple int32 id. clearly in .net world a guid is guarantee to be unique and is more .net oriented, so mongo db has a guid id generator, that can be specified with the above snippet of code.. bsondocument thirddocument = bsondocument.parse("{name: 'untyped3', anotherproperty: 'xxxxxxxxxxxxxxxxxxxxxxx'}"); var id2 = mongodb.bson.serialization.idgenerators.guidgenerator.instance.generateid(untyped, thirddocument); thirddocument.setdocumentid(id2); the key is using the guidgenerator (in the mongodb.bson.serialization.idgenerators namespace) to generate a valid mongoid guid value, then call the setdocumentid method of bsondocument to manually set the id and not relay on automatic id generation. if you look at the db you will find that the document with guid id has really a different id type. figure 4: the document with guid id is represented in a different way in mongovue, but as you can verify there is no problem in having documents with different id types in the same collection. this demonstrates that a no sql database has a concept of document id that is similar to the concept of id of a standard sql server, you can use a native id generation of the engine that generates a valid id during insertion or you can assign your own id to the document, but basically the whole concept of id is more engine-related and has no business meaning, so i strongly discourage to use anything that has a business meaning as id of a document. noone prevents you to insert in the document a property called “myid” or something else that has a business meaning and can be used as logical id and let the engine handle the internal id by itself.

In analyzing social network, one common problem is how to detecting communities, such as groups of people who knows or interacting frequently with each other. Community is a subgraph of a graph where the connectivity are unusually dense. In this blog, I will enumerate some common algorithms on finding communities. First of all, community detection can be think of graph partitioning problem. In this case, a single node will belong to no more than one community. In other words, community does not overlap with each other. High Betweenness Edge Removal The intuition is that members within a community are densely connected and have many paths to reach each other. On the other hand, nodes from different communities requires inter-community links to reach each other, and these inter-community links tends to have high betweenness score. Therefore, by removing these high-betweenness links, the graph will be segregated into communities. Algorithm: For each edge, compute the edge-betweenness score Remove the edge with the highest betweenness score Until there are enough segregation However, while this method achieve good result, it is very slow and not work effectively when there are more than couple thousand nodes with dense edges. Hierarchical Clustering This is a very general approach of detecting communities. Some measure of distance (or similarities) is defined and computed between every pair of nodes first. Then classical hierarchical cluster technique can be applied. The distance should be chosen such that it is small between members within a community and big between members of different community. Random Walk Random walk can be used to compute the distance between every pair of nodes node-B and node-C. Lets focus on undirected graph for now. A random walker starts at node-B, throw a dice and has beta probability that it randomly pick a neighbor to visit based on the weight of links, and with 1 - beta probability that it will teleport back to the original node-v. At an infinite number of steps, the probability distribution of landing on node-w will be high if node-B and node-C belongs to the same community. The intuition here is that the random walker tends to be trapped within the community so all nodes that has high probability distribution tends to be within the same community as node-B (where the random walker is started). Notice that the pick of beta is important. If it is too big (close to 1), then the probability after converging is independent of the starting node (ie: the probability distribution only reflect the centrality of each node but not the community of the starting node). If beta is too small (close to zero), then the walker will die down too quick before it fully explore the community's connectivity. There is an analytical solution to this problem. Lets M be the transition matrix before every pair of nodes. V represents the probability distribution of where the random walkers is. The "distance" between node-B and every other nodes is the eigenvector of M. We can repeat the same to find out distance of all pairs of nodes, and then feed the result to a hierarchical clustering algorithm. Label Propagation The basic idea is that nodes observe its neighbors and set its own label to be the majority of its neighbors. Nodes are initially assigned with a unique label. In each round, each node examine the label of all its neighbors are set its own label to be the majority of its neighbors, when there is a tie, the winner is picked randomly. Until there is no more change of label assignments Modularity Optimization Within a community, the probability of 2 nodes having a link should be higher than if the link is just formed randomly within the whole graph. probability of random link = deg(node-B) * deg(node-C) / (N * (N-1)) The actual link = Adjacency matrix[B, C] Define com(B) to be community of node-B, com(C) to be community of node-C So a utility function "Modularity" is created as follows ... sum_over_v_w((actual - random) * delta(com(B), com(C))) Now we examine communities that can be overlapping. ie: A single node can belong to more than one community. Finding Clique Simple community detection usually starts with cliques. Clique is a subgraph whether every node is connected to any other node. In a K-Clique, there are K nodes and K^2 links between them. However, communities has a looser definition, we don't require everyone to know every other people within the community, but we need them to know "enough" (maybe a certain percentage) of other people in the community. K-core is more relaxed definition, it requires the nodes of the K-core to have connectivity to at least K other members. There are some less popular relaxation, K-Clan requires every node to connect with every other members within K steps (path length less than K). K-plex requires the node to connect to (N - K) members in the node where N total number of members within the K-plex. The community is defined as the found K-core, or K-clan, or K-plex. K-Clique Percolation Another popular way of finding community is by rolling across adjacent K-Clique. Two K-Clique is adjacent if they share K-1 nodes. K is a parameter that we need to pick based on how dense we expect the community should have. The algorithm is illustrated in following diagram. K-Clique percolation is a popular way to identify communities which can potentially be overlapping with each other.

over the past year, i have built my own coding conventions for c#. i also always manage to convince my colleagues to follow my coding convention if they don’t already have one. i’m a real freak about following coding conventions; if i see someone modifying one of my files and not following my conventions, i might have trouble sleeping at night (ok, not that much, but…). with the help of the great visual studio add-on resharper , it’s easy to format code with rules. you only need to press ctrl-e/ctrl-c to format a document. resharper is a must have tool for visual studio. for the past two years, since the release of the windows phone platform, i have been using the xaml language to program my user interfaces. finding coding conventions for c# is pretty easy, but for xaml, it was a bit more of a challenge. my first move was to check the default microsoft projects, but i concluded that even they are a bit messy even for today. here is an example of a windows store grid app project: first of all, there are not any empty lines, and secondly the button has attributes on separate lines, but for the textblock elements, the attributes are on the same lines without any order. with time, i developed my own xaml coding convention that i would like to share. one of the reasons that i developed my own xaml coding convention is i don’t like to use the properties window, because it is hard to have an overview of the properties that are not set to default. my coding convention is resumed in 5 points: 1- put empty lines between elements. don’t be afraid to put empty lines. it makes reading the code easier. my exceptions are the grid.columndefinition and grid.rowdefinitions, because they only have one line attribute. 2- put one attribute per line. 3- order the attributes alphabetically. some will argue that height and width should be side by side or on the adjacent line, but i still prefer the alphabetical order, because it is much easier to read when you know what order your definitions are in. also, if there is an element with many attributes, it is much easier to check whether an attribute is missing. 4- put the attached properties at the beginning and in an alphabetic order. the grid.column / grid.row are the classic examples. 5- definition of styles can be less strict. when i’m creating styles with expression blend, i tend to leave them as-is when they are big. it is more about saving time than anything else. however, when a style is small, i don’t put empty lines and i put the properties in an alphabetic order like this: conclusion it might not be the perfect solution for you, but if you do not have one, my convention is a good start especially if you are sharing code with colleagues. my motto about coding convention is the following: it is better to have a coding convention than not having one! happy coding!

what is actually good, clean code? why does it matter? marry poppendieck has, in her excellent talk deliberate practice in software development ( slides ), quoted a couple of leading figures of our industry on what is clean code. ( emphasis is mine.) bjarne stroustrup, inventor of c++: i like my code to be elegant and efficient. the logic should be straightforward and make it hard for bugs to hide, the dependencies minimal to ease maintenance, error handling complete according to an articulated strategy , and performance close to optimal so as not to tempt people to make the code messy with unprincipled optimizations. clean code does one thing well. grady booch, author of object-oriented analysis and design with applications: clean code is simple and direct. clean code reads like well-written prose. clean code never obscures the designers’ intent but rather is full of crisp abstractions and straightforward lines of control. “big” dave thomas, founder of oti and godfather of the eclipse strategy: clean code can be read, and enhanced by a developer other than its original author . it has unit and acceptance tests. it has meaningful names. it provides one way rather than many ways for doing one thing. it has minimal dependencies, which are explicitly defined, and provides a clear and minimal api. code should be literate since, depending on the language, not all necessary information can be expressed clearly in code alone. michael feathers, author of working effectively with legacy code: i could list all of the qualities that i notice in clean code, but there is one overarching quality that leads to all of them. clean code always looks like it was written by someone who cares. there is nothing obvious that you can do to make it better . all of those things were thought about by the code’s author, and if you try to imagine improvements, you are led back to where you are, sitting in appreciation of the code someone left for you—code written by someone who cared deeply about the craft. ward cunningham, inventor of wiki and fit, co-inventor of extreme programming. the force behind design patterns. smalltalk and oo thought leader. the godfather of all those who care about code. you know you are working with clean code when each routine you read turns out to be pretty much what you expected . you can call it beautiful code when the code also makes it look like the language was made for the problem. summary clean code is easily accessible to others (straightforward, clear intent, good abstractions, no surprises, good names) – this is absolutely the most mentioned point is made for the real-world, i.e. has a clear error-handling strategy (my current project has been burnt a lot by the lack of this so my emphasis is little subjective ) the author clearly cares for the software and other developers (which implies both readability and maintainability) is minimal (does one thing, has minimal dependencies) is good at what it does

Here are some simple steps to get you fully started with Quartz Scheduler on MySQL database using Groovy. The script below will allow you to quickly experiment different Quartz configuration settings using an external file. First step is to setup the database with tables. Assuming you already have installed MySQL and have access to create database and tables. bash> mysql -u root -p sql> create database quartz2; sql> create user 'quartz2'@'localhost' identified by 'quartz2123'; sql> grant all privileges on quartz2.* to 'quartz2'@'localhost'; sql> exit; bash> mysql -u root -p quartz2 < /path/to/quartz-dist/docs/dbTables/tables_mysql.sql The tables_mysql.sql can be found from Quartz distribution download, or directly from their source here. Once the database is up, you need to write some code to start up the Quartz Scheduler. Here is a simply Groovy script quartzServer.groovy that will run as a tiny scheduler server. // Run Quartz Scheduler as a server // Author: Author: Zemian Deng, Date: 2012-12-15_16:46:09 @GrabConfig(systemClassLoader=true) @Grab('mysql:mysql-connector-java:5.1.22') @Grab('org.slf4j:slf4j-simple:1.7.1') @Grab('org.quartz-scheduler:quartz:2.1.6') import org.quartz.* import org.quartz.impl.* import org.quartz.jobs.* config = args.length > 0 ? args[0] : "quartz.properties" scheduler = new StdSchedulerFactory(config).getScheduler() scheduler.start() // Register shutdown addShutdownHook { scheduler.shutdown() } // Quartz has its own thread, so now put this script thread to sleep until // user hit CTRL+C while (!scheduler.isShutdown()) { Thread.sleep(Long.MAX_VALUE) } And now you just need a config file quartz-mysql.properties that looks like this: # Main Quartz configuration org.quartz.scheduler.skipUpdateCheck = true org.quartz.scheduler.instanceName = DatabaseScheduler org.quartz.scheduler.instanceId = NON_CLUSTERED org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate org.quartz.jobStore.dataSource = quartzDataSource org.quartz.jobStore.tablePrefix = QRTZ_ org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount = 5 # JobStore: JDBC jobStoreTX org.quartz.dataSource.quartzDataSource.driver = com.mysql.jdbc.Driver org.quartz.dataSource.quartzDataSource.URL = jdbc:mysql://localhost:3306/quartz2 org.quartz.dataSource.quartzDataSource.user = quartz2 org.quartz.dataSource.quartzDataSource.password = quartz2123 org.quartz.dataSource.quartzDataSource.maxConnections = 8 You can run the Groovy script as usual bash> groovy quartzServer.groovy quartz-mysql.properties Dec 15, 2012 6:20:26 PM com.mchange.v2.log.MLog INFO: MLog clients using java 1.4+ standard logging. Dec 15, 2012 6:20:27 PM com.mchange.v2.c3p0.C3P0Registry banner INFO: Initializing c3p0-0.9.1.1 [built 15-March-2007 01:32:31; debug? true; trace:10] [main] INFO org.quartz.impl.StdSchedulerFactory - Using default implementation for ThreadExecutor [main] INFO org.quartz.core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl [main] INFO org.quartz.core.QuartzScheduler - Quartz Scheduler v.2.1.6 created. [main] INFO org.quartz.core.QuartzScheduler - JobFactory set to: org.quartz.simpl.SimpleJobFactory@1a40247 [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Using thread monitor-based data access locking (synchronization). [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - JobStoreTX initialized. [main] INFO org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.1.6) 'DatabaseScheduler' with instanceId 'NON_CLUSTERED' Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally. NOT STARTED. Currently in standby mode. Number of jobs executed: 0 Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 5 threads. Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is not clustered. [main] INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler 'DatabaseScheduler' initialized from the specified file : 'quartz-mysql.properties' from the class resource path. [main] INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler version: 2.1.6 Dec 15, 2012 6:20:27 PM com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 1hge16k8r18mveoq1iqtotg|1486306, debugUnreturnedConnectionStackTraces -> fals e, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1hge16k8r18mveoq1iqtotg|1486306, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://localhost:3306/quartz2, lastAcquisitionFailureDefaultUser -> null, maxAdministrativeTaskTime -> 0 , maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 8, maxStatements -> 0, maxStatementsPerConnection -> 120, minPoolSize -> 1, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, pref erredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ] [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Freed 0 triggers from 'acquired' / 'blocked' state.[main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Recovering 0 jobs that were in-progress at the time of the last shut-down. [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Recovery complete. [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Removed 0 'complete' triggers. [main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Removed 0 stale fired job entries. [main] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED started. ... CTRL+C [Thread-6] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED shutting down. [Thread-6] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED paused. [Thread-6] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED shutdown complete. That's a full run of above setup. Go ahead and play with different config. Read http://quartz-scheduler.org/documentation/quartz-2.1.x/configuration for more details. Here I will post couple more easy config that will get you started in a commonly used config set. A MySQL cluster enabled configuration. With this, you can start one or more shell terminal and run different instance of quartzServer.groovy with the same config. All the quartz scheduler instances should cluster themselve and distribute your jobs evenly. # Main Quartz configuration org.quartz.scheduler.skipUpdateCheck = true org.quartz.scheduler.instanceName = DatabaseClusteredScheduler org.quartz.scheduler.instanceId = AUTO org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate org.quartz.jobStore.dataSource = quartzDataSource org.quartz.jobStore.tablePrefix = QRTZ_ org.quartz.jobStore.isClustered = true org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount = 5 # JobStore: JDBC jobStoreTX org.quartz.dataSource.quartzDataSource.driver = com.mysql.jdbc.Driver org.quartz.dataSource.quartzDataSource.URL = jdbc:mysql://localhost:3306/quartz2 org.quartz.dataSource.quartzDataSource.user = quartz2 org.quartz.dataSource.quartzDataSource.password = quartz2123 org.quartz.dataSource.quartzDataSource.maxConnections = 8 Here is another config set for a simple in-memory scheduler. # Main Quartz configuration org.quartz.scheduler.skipUpdateCheck = true org.quartz.scheduler.instanceName = InMemoryScheduler org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount = 5 Now, if you need more fancy UI management of Quartz, give MySchedule a try.

Regardless of software process management type (waterfall, scrum, iterative etc.), there are some main rules which should be considered about software process management. In this post we will mention most important of these practices theorically: 1. Define roles and tasks of team members: For productivity, roles of team members should be defined clearly. These roles may be project manager, team lead, develeoper, tester etc. Furthermore, authorizations and responsibilities of these roles should be defined very clearly. Task-assignment based development should also be applied for avoiding redundant effords and chaos. 2. Define meeting types: Meetings are very important if we are talking about software process management and should be defined in detail (meeting participants, context, average duration etc.). Team members should also obey meeting rules. This will bring more productive meeting and development process, and will avoid losing time unnecessarily. 3. Define documentation strategy: Every single software process has a documentation strategy, even agile or extreme methods (backlogs, lessons learned info, bug items etc.). These documentation types should be defined clearly (document physical properties, standard sections, document update period, version numbering etc.) for consistency, understandability and more effective software production. 4. Define software quality metrics and measure them: There are so many metric types (line of code, cyclomatic complexity etc.) in software world. According to the properties of your software type, some of these should be chosen to measure quality and growth of code. These info may be discussed periodically and production quality will be increased after assessments. 5. Perform issue/requirement tracking: Requirement management or issue tracking is one of the key points in software development. They determine the scope of software, also supports a traceability base for functional tests. Those issues/requirements are preferred to be saved and managed with useful tools which have more features than text editing. 6. Perform version controlling: Version controlling is also crutial in production. This should include code and other documents. It will support co-working on code, returning to older versions of code and will ease versioning and dependency management. Versioning strategy (version numbering,versioning periods etc.) should also be determined clearly to obtain consistency. 7. Perform testing: Testing is one of the main phases of software development. Unit testing must be performed for every type of software, except a few exceptions like some user interface code. Other testing types (system, user, integration etc.) should be defined clearly and applied consistently as defined. This will increase quality of production and decrease bugs. 8. Perform dependency management: As software projects grow, more external libraries (jar, dll, ...) or projects (external projects or internal company projects) are included. If those items are added to projects imprecisely, later updates or version changes will bring chaos and so much time consumption. Dependency management strategies and tools should be used for efficient productivity. 9. Perform code reviews frequently: Code review brings high quality code. First of all it enforces developer to produce better code, because it will be controlled by others. Besides, a junior developer will learn better coding quickly by corrections of senior developers. So, code review is a partial type of pair programming and it enhances productivity. 10. Save "lessons learned" info for your projects: Even if there are experienced staff in projects, there may be unforeseen events which may obstruct or retard development process. This can be a complex item configuration, error, production experience etc. Those happenings are highly preferred to be written in "lessons learned" documents and shared in public locations. This will avoid recurrence of time loss and provide more productive software process management.

Okey, another one of my experiments. I came to know a few days ago that Google officially announced how you can now use your Google Drive account as a web host. So, I decided to make use of this awesome new feature and tried to create and host my profile on Google Drive itself. In this post, I am simply going to tell you what you need to do to get up and running with Google Drive as your web host in its simplest form. Things that you need to do to setup Google Drive as your web host Go to Google Drive Of course, you gotta have a Google account to begin in the first place and then just type in drive.google.com Create a public folder This is important. If you want to use Google Drive as a web host, you need to keep your files in a public directory. Create a new folder and then simply choose the 'share' button and give it public access. Lets assume my folder is called 'ryan' Dumping your static files Lets say that your site has a lot of html pages, css stylesheets, jQuery, javascript, images - the usual stuff. Just add all your files in the same directory structure in which they are present in your local system to the public folder that you just created. So, all my files are dumped into the public folder called 'ryan' in my google drive. Naming convention Ensure that at least one file in the public folder is named as index.html. This will act as the landing page for your site. What to publish Now comes an important step. In order to share your site, you need to share a link to the public folder. Initially I thought that you would need to share the link to the index.html but that did not work as expected. So what you need to do is, note the google drive file-id of your public folder. To do that, just open your public folder in google drive. By default it lists all the files in that folder. In the url at the top, you will see that it contains a long string of garbled aphanumeric characters after the last slash. That is your folder id. Copy it. Creating a url to publish Now all you need is a url that points to your folder. To create your url, simply append the file id to the following string - http://googledrive.com/host/ For eg. The raw link to my profile on google drive is http://googledrive.com/host/0B87xZT3F0rDcZlVONVRVY3QtZFE And that pretty much does it. Clicking on the above URL serves the index.html page of your public folder. Other Tips If you want to use a neat little name instead of the long string of garbled alphanumeric characters that appear at the end of the url, go to Gdriv.es and convert your folder id custom name. That's what I did for my profile, and now it looks like this - http://gdriv.es/ryansukale. Much more neat i'd say. Sometimes its just easier to download Google Drive for the desktop, edit your files right in your local machine and then watch them automatically get uploaded. When multiple pages are interlinked, I have observed that the url in the address bar remains the same. So, I guess bookmarking will be an issue in this case.

The command pattern is a behavioral design pattern in which an object is used to represent and encapsulate all the information needed to call a method at a later time. More about this pattern. I adore this pattern. If this pattern had a paypal account, I would donate it money on a regular basis. In general, the notion of encapsulating the method call into an object (like the functor sin C++) is an incredibly powerful idea, because is separate the idea of selecting what to invoke and when to invoke it. Commands are used pretty much every where, WPF is probably the most obvious place, because it actually have the notion of Command as a base class that you are supposed to be using. Other variations, like encapsulating a bunch of code to be executed later (job / task), or just being able to isolate a complex behavior into its own object, is also very useful. I base quite a lot of my architectural advice on the notion that you can decompose a system to a series of commands that you can compose and shuffle at will. Recommendation: Use it. Often. In fact, if you go so far as to say that the only reason we have classes is to have a nice vehicle for creating commands, you wouldn’t be going far enough. Okay, I am kidding, but I really like this pattern, and it is a useful one quite often. The thing that you want to watch for are commands that are too granular. IncrementAgeCommand that is basically wrapping Age++ is probably too much, for example. Commands are supposed to be doing something meaningful from the scope of the entire application. Let's continue the conversation in this other post: "The Command Pattern, Not Completely in Fashion"