In another blog post we explained how you can use a new feature of Mule 3.3 to cache data in your Mule flows. Here we look at how to configure Mule to use EHCache to handle the caching part, rather than storing the data in the default InMemoryObjectStore. Let’s get going. First let’s start by saying that there are millions of different ways to do this... We’ve taken the route of configuring everything through Spring. So just because you configured your EHCache differently, does not mean it’s wrong or ours is better. We prefer this way since Mule integrates very nicely with Spring. Now we have settled that, let’s make a list of what we need to do: Define cache manager Define cache factory bean Create a custom object store Define a Mule caching strategy The first job is to define a cache manager and cache factory bean. Spring provides two very handy classes for this, specific for EHCache: EhCacheManagerFactoryBean and EhCacheFactoryBean. The cache manager just needs to be defined. However, on the cache factory bean you can configure all EHCache details such as time to live, time to idle, when to overflow on disk, the eviction policy, and much more. For more information, you can check the API here or the EHCache website. Also, from the cache factory bean, you need to refer back to the cache manager. An example is shown in the following gist: Once the cache and the cache manager are configured, we need to define a custom object store that uses EHCache to store and retrieve the data. This is very easy to do, we just need to create a new class that implements the standard Mule’s ObjectStore interface, and use EHCache to do the operations. A working custom EHCache object store is shown in the following gist: package com.ricston.cache; import java.io.Serializable; import net.sf.ehcache.Ehcache; import net.sf.ehcache.Element; import org.mule.api.store.ObjectStore; import org.mule.api.store.ObjectStoreException; public class EhcacheObjectStore implements ObjectStore { private Ehcache cache; @Override public synchronized boolean contains(Serializable key) throws ObjectStoreException { return cache.isKeyInCache(key); } @Override public synchronized void store(Serializable key, T value) throws ObjectStoreException { Element element = new Element(key, value); cache.put(element); } @SuppressWarnings("unchecked") @Override public synchronized T retrieve(Serializable key) throws ObjectStoreException { Element element = cache.get(key); if (element == null) { return null; } return (T) element.getValue(); } @Override public synchronized T remove(Serializable key) throws ObjectStoreException { T value = retrieve(key); cache.remove(key); return value; } @Override public boolean isPersistent() { return false; } public Ehcache getCache() { return cache; } public void setCache(Ehcache cache) { this.cache = cache; } } As you can clearly see, this object store encapsulates an EHCache instance. This should be set before we start using this object store. As you can imagine, we will do this through Spring. The next step is to configure a caching strategy which uses our brand new EHCache object store. The caching strategy using our custom object store, and in the object store, we are using Spring to inject the cache defined earlier in this blog post. The rest in Mule can be exactly the same as in the other blog post we explained before. So here we have shown you how we can use EHCache as the caching engine for the cache scopes provided by Mule 3.3. A reason why you would do this is that with EHCache, you have a very good and proven caching product with a ton of settings that you can exploit and tune for your application. As a side note, if you have issues with EHCache classloading in Mule, place the EHCache jars inside $MULE_HOME/lib/user rather than in your application. Enjoy.

The ListView is a widget used extensively in Android applications to display data in a structured fashion. It is also a view that can be quite tricky to optimize, and making your applications feel less jittery is extremely important. When developing an application, a developer has to keep in mind the wide range of hardware on which Android runs. Nowadays most high end phones and tablets are equipped with at least a dual core processor, which can handle unoptimized code at tolerable speeds. However, a large percentage of Android devices are still using single core processors with limited memory. This blog aims to deliver one form of optimization that developers can place in their code at next to no effort. The ViewHolder pattern is a pattern that a developer can use to increase the speed at which their ListView renders data. The reason for this improvement is that the number of times which the findViewById method is invoked is drastically reduced, existing views do not have to be garbage collected and new views do not have to be inflated. A typical ListView adapter contains the following method signature: public View getView (int position, View convertView, ViewGroup parent) The ‘position’ integer stores the position of the current item within the adapter’s data set. The ‘convertView’ field is a reference to the old view that we are planning to reuse. This is the field that we use to implement our optimisation. It is extremely important to use this variable. Not doing so eventually leads to an OutOfMemoryException. This happens because internally a ListView keeps a reference to views it has already seen. Not reusing the ‘convertView’ field will continuously add new views to the ListView, causing a noticeable slowdown of your application and eventually lead to your application crashing. The ‘parent’ field is a reference to the parent view that this view is eventually attached to. Consider the following class: public class Person { private String name; private String surname; private Bitmap image; public Person(String name, String surname) { this.name = name; this.surname = surname; } public String getName() { return name; } public String getSurname() { return surname; } public Bitmap getImage() { return image; } public void setName(String name) { this.name = name; } public void setSurname(String surname) { this.surname = surname; } public void setImage(Bitmap image) { this.image = image; } } Now we want to print a list of persons, all of whom are paired with an image. We need one ImageView for the image, and two TextViews for the name and surname fields. Our ViewHolder class would look something like the following and reside as a static inner class inside our Activity (or Fragment): static class ViewHolder { private TextView nameTextView; private TextView surnameTextView; private ImageView personImageView; } We then implement our own ArrayAdapter for our list of persons that we want to display: private class PersonsAdapter extends ArrayAdapter { ... @Override public View getView(int position, View convertView, ViewGroup parent) { ViewHolder holder; if (convertView == null) { convertView = mInflater.inflate(R.layout.list_entry, null); holder = new ViewHolder(); holder.nameTextView = (TextView) convertView.findViewById(R.id.person_name); holder.surnameTextView = (TextView) convertView.findViewById(R.id.person_surname); holder.personImageView = (ImageView) convertView.findViewById(R.id.person_image); convertView.setTag(holder); } else { holder = (ViewHolder) convertView.getTag(); } Person person = getItem(position); holder.nameTextView.setText(person.getName()); holder.surnameTextView.setText(person.getSurname()); //holder.personImageView.setImageBitmap(person.getImage()); return convertView; } } Note that I have purposefully commented out the loading of bitmaps in the ListView. This is something that has given a lot of trouble to developers and might be something we could address in a future blog post. Given the relatively small heap size that Android applications are allowed to access, developers need to take several steps to display multiple images on screen in an efficient manner. Downsampling of images is just one example of many possible optimizations.

When putting together data sets to play around with, one of the more boring tasks is stripping out characters that you’re not interested in and more often than not those characters are white spaces. Since I’ve been building data sets using Clojure I wanted to write a function that would do this for me. I started out with the following string: (def word " with a little bit of space we can make it through the night ") which I wanted to format in such a way that there would be a maximum of one space between each word. I start out by using the trim function but that only removes white space from the beginning and end of a string: > (clojure.string/trim word) "with a little bit of space we can make it through the night" I wanted to get rid of the space in between ‘a’ and ‘little’ as well so I wrote the following code to split on a space and filter out any excess spaces that still remained before joining the words back together: > (clojure.string/join " " (filter #(not (clojure.string/blank? %)) (clojure.string/split word #" "))) "with a little bit of space we can make it through the night" I wanted to try and make it a bit easier to read by using the thread last (->>) macro but that didn’t work as well as I’d hoped because clojure.string/split doesn’t take the string in as its last parameter: > (->> (clojure.string/split word #" ") (filter #(not (clojure.string/blank? %))) (clojure.string/join " ")) "with a little bit of space we can make it through the night" I worked around it by creating a specific function for splitting on a space: (defn split-on-space [word] (clojure.string/split word #"\s")) which means we can now chain everything together nicely: > (->> word split-on-space (filter #(not (clojure.string/blank? %))) (clojure.string/join " ")) "with a little bit of space we can make it through the night" I couldn’t find a cleaner way to do this but I’m sure there is one and my googling just isn’t up to scratch so do let me know in the comments!

In the previous post, we have seen an example where dataProvider attribute has been used3 to test methods with different sets of input data for the same test method. TestNG provides another attribute dataProviderClass in conjunction with dataProvider to fetch the input data for the test methods from an external class. The actual class that holds input data is set to the dataProviderClass attribute and datProvider by itself holds the method name where the input data is actually fetched. Here is a quick example to show how to use dataProviderClass and dataProvide attribute Code Service Class ? view source print? 01.package com.skilledmonster.example; 02./** 03.* Simple calculator service to demonstrate TestNG Framework 04.* 05.* @author Jagadeesh Motamarri 06.* @version 1.0 07.*/ 08.public interface CalculatorService { 09.int sum(int a, int b); 10.int multiply(int a, int b); 11.int div(int a, int b); 12.int sub(int a, int b); 13.} Service Implementation Class ? view source print? 01.package com.skilledmonster.example; 02./** 03.* Simple calculator service implementation to demonstrate TestNG Framework 04.* 05.* @author Jagadeesh Motamarri 06.* @version 1.0 07.*/ 08.public class SimpleCalculator implements CalculatorService { 09.public int sum(int a, int b) { 10.return a + b; 11.} 12.public int multiply(int a, int b) { 13.return a * b; 14.} 15.public int div(int a, int b) { 16.return a / b; 17.} 18.public int sub(int a, int b) { 19.return a - b; 20.} 21.} Data Provider Class ? view source print? 01.package com.skilledmonster.common; 02.import org.testng.annotations.DataProvider; 03./** 04.* Data Provider class for TestNG test cases 05.* 06.* @author Jagadeesh Motamarri 07.* @version 1.0 08.*/ 09.public class TestNGDataProvider { 10./** 11.* Data Provider for testing sum of 2 numbers 12.* 13.* @return 14.*/ 15.@DataProvider 16.public static Object[][] testSumInput() { 17.return new Object[][] { { 5, 5 }, { 10, 10 }, { 20, 20 } }; 18.} 19./** 20.* Data Provider for testing multiplication of 2 numbers 21.* 22.* @return 23.*/ 24.@DataProvider 25.public static Object[][] testMultipleInput() { 26.return new Object[][] { { 5, 5 }, { 10, 10 }, { 20, 20 } }; 27.} 28.} Finally, test class that uses dataProviderClass attribute to feed the input data for the test methods ? package com.skilledmonster.example; import org.testng.Assert; import org.testng.annotations.BeforeClass; import org.testng.annotations.Test; import com.skilledmonster.common.TestNGDataProvider; /** * Example to demonstrate use of dataProviderClass and dataProvide attributes of TestNG framework * * @author Jagadeesh Motamarri * @version 1.0 */ public class TestNGAnnotationTestDataProviderExample { public CalculatorService service; @BeforeClass public void init() { System.out.println("@BeforeClass: The annotated method will be run before the first test method in the current class is invoked."); System.out.println("init service"); service = new SimpleCalculator(); } @Test(dataProviderClass = TestNGDataProvider.class, dataProvider = "testSumInput") public void testSum(int a, int b) { System.out.println("@Test : testSum()"); int result = service.sum(a, b); Assert.assertEquals(result, a + b); } @Test(dataProviderClass = TestNGDataProvider.class, dataProvider = "testMultipleInput") public void testMultiple(int a, int b) { System.out.println("@Test : testMultiple()"); int result = service.multiply(a, b); Assert.assertEquals(result, a * b); } } Output As shown in the above console output, each of the testSum() and testMutiple() methods are invoked with different sets of input data using an external class with dataProviderClass attribute. Advantage More flexibility and re-usability of commonly used data across several test classes. Download Download TestNG DataProvider Example

Recently, a customer made a query on how to perform large dataset retrieval in Mule. The documentation page briefly explains how this may be achieved, however there is no working example on how to do this as far as I can tell. This blog post aims to explain in detail how large dataset retrieval works in Mule by giving an example. The customer wanted to transfer items from one database to another by performing a batch select and then a batch insert. The ‘batch insert’ part is pretty straightforward and is done automatically by Mule when the payload is of type List. However, the batch select is mastered in a different way. In order to retrieve all the records, we will use the Batch Manager to compute the ID ranges for the next batch of records to be retrieved. This is provided out of the box with Mule EE. We start by defining the database which will be used throughout the example to retrieve and insert records. For simplicity’s sake we are going to use the Derby in-memory database. NOTE: the records should be identified by a key which is unique and in a sequential numeric order. CREATE TABLE table1(KEY1 INTEGER GENERATED BY DEFAULT AS IDENTITY(START WITH 1) NOT NULL PRIMARY KEY, KEY2 VARCHAR(255)); CREATE TABLE table2(KEY1 VARCHAR(255), KEY2 VARCHAR(255)); INSERT INTO table1(KEY2) VALUES ('TEST1'); INSERT INTO table1(KEY2) VALUES ('TEST2'); INSERT INTO table1(KEY2) VALUES ('TEST3'); INSERT INTO table1(KEY2) VALUES ('TEST4'); INSERT INTO table1(KEY2) VALUES ('TEST5'); INSERT INTO table1(KEY2) VALUES ('TEST6'); INSERT INTO table1(KEY2) VALUES ('TEST7'); INSERT INTO table1(KEY2) VALUES ('TEST8'); INSERT INTO table1(KEY2) VALUES ('TEST9'); INSERT INTO table1(KEY2) VALUES ('TEST10'); As explained before, the select query is based on the ID ranges that are computed by the Batch Manager when nextBatch() is called. This will return a map with the lower and upper ids to be selected. In our case, we are storing this map into a flow variable named ‘boundaries’. After configuring the database and the JDBC connector, we need to configure the Batch Manager. This consists of specifying the idStore (which is a text file), which the BatchManager uses to store the starting point for the next batch. Moreover, on the Batch Manager, we need to configure the batch size and the starting point. In the documentation, you would find a reference to the noArgsWrapper. Its job is to invoke the nextBatch() method on the Batch Manager. However we find this very confusing and misleading, thus instead, we use a simple MEL expression which calls the nextBatch() directly. Now we have to configure the main flow where we perform the batch select. Given that the records are retrieved in batches, the flow has to be called multiple times until all of the records are retrieved. To solve this, we created a composite source so that at the end of the flow, if we haven’t retrieved all the records, we re-trigger the same flow using the VM queue. Once the current batch is finished, we need to call competeBatch() to instruct the batch manager that we’re done from the current batch, and ready to process the next. If this is not done, the Batch Manager will still consider the previous batch as ‘processing’. Furthermore, we have to check whether we have retrieved all of the records so we can stop processing. We do this by checking the size of the payload that is returned from the JDBC outbound endpoint. If the payload size is ’0′ (no more records to be retrieved), we have to call the completeBatch() method with ‘-1′, instructing the Batch Manager that all of the batch is complete. We must also set the starting point for next batch to ’0′. This is required so that when the flow is triggered again from the HTTP inbound endpoint, the flow will start processing from the first record. If the batch is not complete, we call the completeBatch() method (from the BatchManager class) with the current upperId. This sets the new starting point for the next batch to be processed. Finally we end the flow with a VM outbound on ‘batch’ which triggers the main flow to process the next batch of records. app.registry.seqBatchManager.completeBatch(-1); app.registry.seqBatchManager.setStartingPointForNextBatch(0); app.registry.seqBatchManager.completeBatch(flowVars.boundaries.upperId); A complete Mule configuration of the main flow shown here below.

Here are some free HTML WYSIWYG editors based on the Mozilla Gecko Engine.

I wanted to do some date manipulation in Clojure recently and figured that since clj-time is a wrapper around Joda Time it’d probably do the trick. The first thing we need to do is add the dependency to our project file and then run lein reps to pull down the appropriate JARs. The project file should look something like this: project.clj (defproject ranking-algorithms "0.1.0-SNAPSHOT" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :dependencies [[org.clojure/clojure "1.4.0"] [clj-time "0.6.0"]]) Now let’s load the clj-time.format namespace into the REPL since we know we’ll be parsing dates: > (require '(clj-time [format :as f])) The string that I want to convert into a date looks like this: (def string-date "18 September 2012") The first thing we should do is check whether there is an existing formatter that we can use by evaluating the following function: > (f/show-formatters) ... :hour-minute 06:45 :hour-minute-second 06:45:22 :hour-minute-second-fraction 06:45:22.473 :hour-minute-second-ms 06:45:22.473 :mysql 2013-09-20 06:45:22 :ordinal-date 2013-263 :ordinal-date-time 2013-263T06:45:22.473Z :ordinal-date-time-no-ms 2013-263T06:45:22Z :rfc822 Fri, 20 Sep 2013 06:45:22 +0000 ... There are a lot of different built in formatters but unfortunately I couldn’t find one that exactly matched our date format so we’ll have to write our own one. For that we’ll need to refresh our knowledge of Java date formatting: We end up with the following formatter: > (f/parse (f/formatter "dd MMM YYYY") string-date) # It took me much longer than it should have to remember that ‘MMM’ is the pattern to match a short form of a month but it’s just the same as what we’d have to do in Java but with some neat wrapper functions.

Even among the large IT/development publications, I see inconsistencies in the use of the word front-end. Is it hyphenated or not?

This post comes from Alberto Pose at the MuleSoft blog. Introduction Suppose that you have a Maven project and you want to download Node.js modules previously uploaded to NPM. One way of doing that without running node is by using the npm-maven-plugin. It allows the user to download the required Node modules without running node.js: It is completely implemented on the JVM. Getting Started First of all, you will need to add the Mule Maven repo to your pom.xml file: mulesoft-releases MuleSoft Repository https://repository.mulesoft.org/releases/ After doing that, you will need to add the following to the build->plugin section of your pom.xml file: org.mule.tools.javascript npm-maven-plugin 1.0 generate-sources fetch-modules colors:0.5.1 jshint:0.8.1 Then just execute: mvn generate-sources and that’s it! One more thing… By default, the modules can be found in src/main/resources/META-INF but that path can be changed setting the ‘outputDirectory’ parameter. Also, module transitivity is taken into account. That means that it will download all the required dependencies before downloading the specified module. Show me the code! The source code can be found here. Feel free to fork the project and propose changes to it. Happy (Node.js and Maven) hacking!

When I’m teaching training courses (as I was this week at Skills Matter) or advising clients on the requirements-side of software development (which I’m doing a lot of just now), I talk about a model I call the “3 Styles of Agile.” Incredibly, I’ve never blogged about this -- although the model is hidden inside a couple of articles over the years. So now the day has come… I don’t claim the “3 Styles Model” is the way it should be, I only claim that it is the way I find the world. While “doing Agile” on the code side of software development always comes back to the same things (stand-up meetings, test/behavior driven development, code review/pair programming, stories, boards, etc.) the requirements side is very very variable. The advice that is given is very variable and the degree to which that advice is compatible with corporate structures and working is very variable. However, I find three reoccurring styles in which the requirements side operates and interfaces to development. I call these styles: Iterative, Incremental and Evolutionary, and I usually draw this diagram: I say style because I’m looking for a neutral word. I think you can use Scrum, XP and Kanban (or any other method) in any of the three styles. That said, I believe Kanban is a better fit for evolutionary while Scrum/XP are a better fit for Iterative and Incremental. I try not to be judgmental, I know a lot of Agile folk will see Evolutionary as superior, they may even consider Evolutionary to be the only True Agile but actually I don’t think that is always the case. There are times when the other styles are “right.” Let me describe the three styles: Iterative In this style the development team is doing lots of good stuff like: stand up meetings, planning meetings, short iterations or Kanban flow, test driven development, code review, refactoring, continuous integration and so on. I say they are doing it but it might be better to say “I hope they are doing” because quite often some bit or other is missing. That’s not important for this model. The key thing is the dev team are doing it! In this model, requirements arrive in a requirements document en mass. In fact, the rest of the organization carries on as if nothing has changed, indeed this may be what the organization wants. In this model you hear people say things like “Agile is a delivery mechanism” and “Agile is for developers." The requirement document may even have been written by a consultant or analyst who is now gone. The document is “thrown over the fence” to another analyst or project manager who is expected to deliver everything (scope, features) within some fixed time frame for some budget. Delivery is most likely one “big bang” at the end of the project (when the team may be dissolved). In order to do this they use a bacon slicer. I’ve written about this before and called it Salami Agile. The requirements document exists and the job of the “Product Owner” is to slice off small pieces for the team to do every iteration. The development team is insulated from the rest of the organization. There is probably still a change review board and any increase scope is seen as a problem. I call this iterative because the team is iterating but that’s about it. This is the natural style of large corporations, companies with annual budgets, senior managers who don’t understand IT and in particular banks. Incremental This style is mostly the same as Iterative, it looks similar to start with. The team are still (hopefully) doing good stuff and iterating. There is still a big requirements document, the organization still expects it all delivered and it is still being salami sliced. However, in this model, the team is delivering the software to customers. At the very least, they are demonstrating the software and listening to feedback. More likely, they are deploying the software and (potential) users can start using it today. As a result, the customer/users give feedback about what they want in the software. Sometimes this is an extra feature and functionality (scope creep!) and sometimes it is about removing things that were requested (scope retreat!). The “project” is still done in the traditional sense that everything in the document is “done,” but now some things are crossed out rather than ticked. Plus some additional stuff might be done over and above the requirements document. I call this incremental because the customers/users/stakeholders are seeing the thing grow in increments -- and hopefully early value is being delivered. I actually believe this is the most common style of software development -- whether that work is called Agile, waterfall or anything else. However, in some environments this is seen as wrong, wrong because the upfront requirements are “wrong” or because multiple deliveries need to be made, or because the team aren’t delivering everything they were originally asked to deliver. Evolutionary Here again the development team are iterating much as before. However, this time there is no requirements document. Work has begun with just an idea. Ideally I would want to see a goal, an objective, an aim, which will guide work and help inform what should be done -- and this goal should be stated in a single sentence, a paragraph at most. But sometimes even this is missing, for better or worse. In this model the requirements guy and developers both start at the beginning. They brainstorm some ideas and select something to do. While Mr. Requirements runs off to talk to customers and stakeholders about what the problem is and what is needed, the tech team (maybe just one person) gets started on the best idea so far. Sometime soon (2 weeks tops) they get back together. Mr. Requirements talks about what he has found and the developers demonstrate what they have built. They talk some more and decide what to do next. With that done, the developers gets on with building and Mr. Requirements gets on his bike again, he shows what has been built and talks to people -- some people again and some new people. As soon as possible the team starts to push software out to users and customers to use. This delivers value and provides feedback. And so it goes. It finishes, if it finishes, when the goal is met to the organization decided to use its resources somewhere else. Evolutionary style is most at home in Palo Alto, Mountain View, and anywhere else that start-ups are the norm. Evolutionary is actually a lot more common than is recognized but it is called maintenance or “bug fixing” and seen as something that shouldn’t exist. Having set out the three styles I’ll leave discussion of how to use the model and why you might use each style to another entry. If you want to know more about each model and how I see Agile as spectrum have a look my 2011 “The Agile Spectrum” from ACCU Overload or the recently revised (expanded but unfinished) version by the same title: “Agile Spectrum” (the 2013 version I suppose, online only).

One of the problems with memory-mapped files is that you can’t actually map beyond the end of the file. So you can’t use that to extend your file. I had a thought about and set out to check out what happens when I create a sparse file, a file that only take space when you write to it, and at the same time, map it. As it turns out, this actually works pretty well in practice. You can do so without any issues. Here is how it works: using (var f = File.Create(path)) { int bytesReturned = 0; var nativeOverlapped = new NativeOverlapped(); if (!NativeMethod.DeviceIoControl(f.SafeFileHandle, EIoControlCode.FsctlSetSparse, IntPtr.Zero, 0, IntPtr.Zero, 0, ref bytesReturned, ref nativeOverlapped)) { throw new Win32Exception(); } f.SetLength(1024*1024*1024*64L); } This creates a sparse file that is 64 GB in size. Then we can map it normally: using (var mmf = MemoryMappedFile.CreateFromFile(path)) using (var memoryMappedViewAccessor = mmf.CreateViewAccessor(0, 1024*1024*1024*64L)) { for (long i = 0; i < memoryMappedViewAccessor.Capacity; i += buffer.Length) { memoryMappedViewAccessor.WriteArray(i, buffer, 0, buffer.Length); } } And then we can do stuff to it. And that includes writing to yet-unallocated parts of the file. This also means that you don’t have to worry about writing past the end of the file, the OS will take care of all of that for you. Happy happy, joy joy, etc. There is one problem with this method, however. It means that you have a 64 GB file, but you don’t have that much allocated. What that means in turn is that you might not have that much space available for the file. Which brings up an interesting question, what happens when you are trying to commit a new page, and the disk is out of space? Using file I/O you would get an I/O error with the right code. But when using memory mapped files, the error would actually turn up during access, which can happen pretty much anywhere. It also means that it is a Standard Exception Handling error in Windows, which requires special treatment. To test this out, I wrote the following so it would write to a disk that had only about 50 GB free. I wanted to know what would happen when it ran out of space. That is actually something that happens, and we need to be able to address this issue robustly. The kicker is that this might actually happen at any time, so that would really result is some… interesting behavior with regards to robustness. In other words, I don’t think that this is a viable option, it is a really cool trick, but I don’t think it is a very well thought out option. By the way, the result of my experiment was that we had an effectively a frozen process. No errors, nothing, just a hung. Also, I am pretty sure that WriteArray() is really slow, but I’ll check this out at another pointer in time.

In this post we are going to learn how we can use NHibernate in ASP.NET MVC application. What is NHibernate: ORMs(Object Relational Mapper) are quite popular this days. ORM is a mechanism to map database entities to Class entity objects without writing a code for fetching data and write some SQL queries. It automatically generates SQL Query for us and fetch data behalf on us. NHibernate is also a kind of Object Relational Mapper which is a port of popular Java ORM Hibernate. It provides a framework for mapping an domain model classes to a traditional relational databases. Its give us freedom of writing repetitive ADO.NET code as this will be act as our database layer. Let’s get started with NHibernate. How to download: There are two ways you can download this ORM. From nuget package and from the source forge site. Nuget - http://www.nuget.org/packages/NHibernate/ Source Forge-http://sourceforge.net/projects/nhibernate/ Creating a table for CRUD: I am going to use SQL Server 2012 express edition as a database. Following is a table with four fields Id, First Name, Last name, Designation. Creating ASP.NET MVC project for NHibernate: Let’s create a ASP.NET MVC project for NHibernate via click on File-> New Project –> ASP.NET MVC 4 web application. Installing NuGet package for NHibernate: I have installed nuget package from Package Manager console via following Command. It will install like following. NHibertnate configuration file: Nhibernate needs one configuration file for setting database connection and other details. You need to create a file with ‘hibernate.cfg.xml’ in model Nhibernate folder of your application with following details. NHibernate.Connection.DriverConnectionProvider NHibernate.Driver.SqlClientDriver Server=(local);database=LocalDatabase;Integrated Security=SSPI; NHibernate.Dialect.MsSql2012Dialect Here you have got different settings for NHibernate. You need to selected driver class, connection provider as per your database. If you are using other databases like Orcle or MySQL you will have different configuration. ThisNHibernate ORM can work with any databases. Creating a model class for NHibernate: Now it’s time to create model class for our CRUD operations. Following is a code for that. Property name is identical to database table columns. namespace NhibernateMVC.Models { public class Employee { public virtual int Id { get; set; } public virtual string FirstName { get; set; } public virtual string LastName { get; set; } public virtual string Designation { get; set; } } } Creating a mapping file between class and table: Now we need a xml mapping file between class and model with name “Employee.hbm.xml” like following in Nhibernate folder. Creating a class to open session for NHibernate I have created a class in models folder called NHIbernateSession and a static function it to open a session for NHibertnate. using System.Web; using NHibernate; using NHibernate.Cfg; namespace NhibernateMVC.Models { public class NHibertnateSession { public static ISession OpenSession() { var configuration = new Configuration(); var configurationPath = HttpContext.Current.Server.MapPath(@"~\Models\Nhibernate\hibernate.cfg.xml"); configuration.Configure(configurationPath); var employeeConfigurationFile = HttpContext.Current.Server.MapPath(@"~\Models\Nhibernate\Employee.hbm.xml"); configuration.AddFile(employeeConfigurationFile); ISessionFactory sessionFactory = configuration.BuildSessionFactory(); return sessionFactory.OpenSession(); } } } Listing: Now we have our open session method ready its time to write controller code to fetch data from the database. Following is a code for that. using System; using System.Web.Mvc; using NHibernate; using NHibernate.Linq; using System.Linq; using NhibernateMVC.Models; namespace NhibernateMVC.Controllers { public class EmployeeController : Controller { public ActionResult Index() { using (ISession session = NHibertnateSession.OpenSession()) { var employees = session.Query().ToList(); return View(employees); } } } } Here you can see I have get a session via OpenSession method and then I have queried database for fetching employee database. Let’s create a new for this you can create this via right lick on view on above method.We are going to create a strongly typed view for this. Our listing screen is ready once you run project it will fetch data as following. Create/Add: Now its time to write add employee code. Following is a code I have written for that. Here I have used session.save method to save new employee. First method is for returning a blank view and another method with HttpPost attribute will save the data into the database. public ActionResult Create() { return View(); } [HttpPost] public ActionResult Create(Employee emplolyee) { try { using (ISession session = NHibertnateSession.OpenSession()) { using (ITransaction transaction = session.BeginTransaction()) { session.Save(emplolyee); transaction.Commit(); } } return RedirectToAction("Index"); } catch(Exception exception) { return View(); } } Now let’s create a create view strongly typed view via right clicking on view and add view. Once you run this application and click on create new it will load following screen. Edit/Update: Now let’s create a edit functionality with NHibernate and ASP.NET MVC. For that I have written two action result method once for loading edit view and another for save data. Following is a code for that. public ActionResult Edit(int id) { using (ISession session = NHibertnateSession.OpenSession()) { var employee = session.Get(id); return View(employee); } } [HttpPost] public ActionResult Edit(int id, Employee employee) { try { using (ISession session = NHibertnateSession.OpenSession()) { var employeetoUpdate = session.Get(id); employeetoUpdate.Designation = employee.Designation; employeetoUpdate.FirstName = employee.FirstName; employeetoUpdate.LastName = employee.LastName; using (ITransaction transaction = session.BeginTransaction()) { session.Save(employeetoUpdate); transaction.Commit(); } } return RedirectToAction("Index"); } catch { return View(); } } Here in first action result I have fetched existing employee via get method of NHibernate session and in second I have fetched and changed the current employee with update details. You can create view for this via right click –>add view like below. I have created a strongly typed view for edit. Once you run code it will look like following. Details: Now it’s time to create a detail view where user can see the employee detail. I have written following logic for details view. public ActionResult Details(int id) { using (ISession session = NHibertnateSession.OpenSession()) { var employee = session.Get(id); return View(employee); } } You can add view like following via right click on actionresult view. now once you run this in browser it will look like following. Delete: Now its time to write delete functionality code. Following code I have written for that. public ActionResult Delete(int id) { using (ISession session = NHibertnateSession.OpenSession()) { var employee = session.Get(id); return View(employee); } } [HttpPost] public ActionResult Delete(int id, Employee employee) { try { using (ISession session = NHibertnateSession.OpenSession()) { using (ITransaction transaction = session.BeginTransaction()) { session.Delete(employee); transaction.Commit(); } } return RedirectToAction("Index"); } catch(Exception exception) { return View(); } } Here in the above first action result will have the delete confirmation view and another will perform actual delete operation with session delete method. When you run into the browser it will look like following. That’s it. It’s very easy to have crud operation with NHibernate. Stay tuned for more.

activity recognition gives our android device the ability to detect a number of our physical activities like walking, riding a bicycle, driving a car or standing idle. all that can be detected by simply using an api to access google play services , an increasingly crucial piece of software available to all android versions. as in the article on geofencing , we will download the sample app ( activityrecognition.zip ) at the android developer’s site and start playing with it, eventually modifying parts of it to fit our purposes. we will show here only the most relevant code sections. the first thing to note is that we need a specific permission to use activity recognition: as with geofencing or location updates, we use the api to request google play services to analyse our data and provide us with the results. the chain of method calls for requesting updates is similar to that of geofencing: make sure that google play services is available. as an activity recognition client, request a connection. once connected, location services calls back the onconnected() method in our app. proceed with the updates request via a pending intent pointing to an intentservice we have written. google location services sends out its activity recognition updates as intent objects, using the pendingintent we provided. get and process the updates in our intentservice’s onhandleintent() method. the sample app writes all the updates in a log file, and that is ok if we like that sort of thing … though a closer look at the data makes us realize that most of it is garbage. do we really need to know that we have a 27 percent chance of being driving a vehicle and a 7 percent chance of riding a bicycle when we are in fact sitting idle at our desk? not really. what we want is the most significant data, and in this case, that would be the most probable activity: //.. import com.google.android.gms.location.activityrecognitionresult; import com.google.android.gms.location.detectedactivity; /** * service that receives activityrecognition updates. it receives updates * in the background, even if the main activity is not visible. */ public class activityrecognitionintentservice extends intentservice { //.. /** * called when a new activity detection update is available. */ @override protected void onhandleintent(intent intent) { //... // if the intent contains an update if (activityrecognitionresult.hasresult(intent)) { // get the update activityrecognitionresult result = activityrecognitionresult.extractresult(intent); detectedactivity mostprobableactivity = result.getmostprobableactivity(); // get the confidence % (probability) int confidence = mostprobableactivity.getconfidence(); // get the type int activitytype = mostprobableactivity.gettype(); /* types: * detectedactivity.in_vehicle * detectedactivity.on_bicycle * detectedactivity.on_foot * detectedactivity.still * detectedactivity.unknown * detectedactivity.tilting */ // process } } } instead of writing the updates to a log file, it is simpler to just store them in memory (e.g. in a static list in a dedicated class) and display them to the user of our app. one way to do this would be by using a fragment to display the updates on top of a google map. as commented in previous articles, fragments were introduced in honeycomb but are also available to older android versions through the support library . once we define our own xml layout for the actreconfragment and give it a transparent background (left to the reader as an exercise), we will get a nice overlaid display like this: since we have chosen to show the most probable activity to the users of our app, we need the display to be dynamic, like a live feed . for that, we can add a local broadcast in our service: //inside activityrecognitionintentservice 's onhandleintent intent broadcastintent = new intent(); // give it the category for all intents sent by the intent service broadcastintent.addcategory(activityutils.category_location_services); // set the action and content for the broadcast intent broadcastintent.setaction(activityutils.action_refresh_status_list); // broadcast *locally* to other components in this app localbroadcastmanager.getinstance(this).sendbroadcast(broadcastintent); we are using a localbroadcastmanager (included in android 3.0 and above, and in the support library v4 for early releases). apart from providing our own layout to position the activity detection panel on top of a map, the only new code snippet we wrote is the above local broadcast. for the remainder below, we have simply re-positioned the sample app’s code in a fragment and use in-memory storage of the activity updates instead of using a log file. the receiver on that local broadcast is in our fragment: //... public class actreconfragment extends fragment{ // intent filter for incoming broadcasts from the intentservice intentfilter mbroadcastfilter; // instance of a local broadcast manager private localbroadcastmanager mbroadcastmanager; //... /** * called when the corresponding map activity's * oncreate() method has completed. */ @override public void onactivitycreated(bundle savedinstancestate) { super.onactivitycreated(savedinstancestate); // set the broadcast receiver intent filer mbroadcastmanager = localbroadcastmanager.getinstance(getactivity()); // create a new intent filter for the broadcast receiver mbroadcastfilter = new intentfilter(activityutils.action_refresh_status_list); mbroadcastfilter.addcategory(activityutils.category_location_services); //... } /** * broadcast receiver that receives activity update intents * this receiver is local only. it can't read broadcast intents from other apps. */ broadcastreceiver updatelistreceiver = new broadcastreceiver() { @override public void onreceive(context context, intent intent) { // when an intent is received from the update listener intentservice, // update the display. updateactivityhistory(); } }; //... } live feed shots: once we have taken care of the display, we need to move on to other important aspects like what to do with those activity updates. the sample app gives us one example of that in the activityrecognitionintentservice : if( // if the current type is "moving" i.e on foot, bicycle or vehicle ismoving(activitytype) && // the activity has changed from the previous activity activitychanged(activitytype) // the confidence level for the current activity is >= 50% && (confidence >= 50)) { // do something useful } simply getting the most probable activity might be ok for displaying purposes, but might not be enough for an app to act on it and do something useful. we need to make sure that the type of activity and the corresponding confidence level (i.e. probability) are adequate for our purposes. while a detected activity type of “unknown” with a confidence level of 52% is next to useless, knowing that the user is moving in a vehicle as opposed to walking can be put to good use: increase the frequency of location updates, enlarge the map area of available points of interest, etc … activity recognition has been added as an experimental feature to this geofencing app . check it out and feel free to post any feedback.

Built upon Lucene, Solr provides fast, highly scalable, and easily maintainable full-text search capabilities. However, under the hood, Solr is really just a sophisticated token-matching engine. What’s missing? — Semantic Search! Consider three, somewhat silly documents: Yellow banana peels. A banana is a long yellow fruit. This mystery fruit is long and yellow and has a peel. Now what happens if you search for the term “banana?" Under normal circumstances you only get back the first and second document. But why shouldn’t you also get back the third document? It’s obviously talking about bananas! Semantic Search via Collaborative Filtering Colleague Doug Turnbull and I recently set about to right this wrong with help from a machine learning technique called collaborative filtering. Collaborative filtering is most often used as a basis for recommendation algorithms. For example, collaborative filtering algorithms were the central focus of the now-famous Netflix Prize which awarded $1Million to the team which could build the best movie recommendation engine. When dealing with recommendations, collaborative filtering works by mathematically identifying commonalities in groups of users based upon the movies that they enjoyed. Then, if you appear to fall in one of those groups, the recommendation engine will point you towards a movie that a) you haven’t watched and b) you are likely to enjoy. So what does this have to do with Semantic Search? Everything! In just the same way that certain users gravitate towards certain movies, certain words commonly co-occur in the same documents. When working with Semantic Search, rather than recommending user to movies that they would likely enjoy, we are going to identify words that are likely to belong in a given document, whether or not they actually occurred there. The math is exactly the same! Here’s how the process works: First we identify a text field of interest in our documents and extract the associated term-document matrix for external processing. Each element of this term-document matrix indicates the strength of a particular term within a particular document (where strength can be anything, but will likely be either term frequency or TF*IDF). Next, collaborative filtering is applied to the term-document matrix which effectively generates a pseudo-term-document matrix. This pseudo-term-document matrix is the same size and shape as the original term-document matrix and references the same terms and documents, but the numbers are slightly different. These new values indicate the strength that a particular term should have in a particular document once noisy data is removed. Finally, the high-scoring values in the pseudo-term-document matrix are mapped back to the associated terms. These terms are then injected back into Solr in a new field which can be used for Semantic Search. Demo Time! So let’s consider an example case. As in plenty of our previous posts, we will be using the Science Fiction Stack Exchange. Why? Because we’re all nerds and with such a familiar topic, we can quickly intuit whether or not a search is returning relevant results. In this data set, the field of interest is the Body field because it contains the contents of all questions and answers. So, now that we’ve decided upon our demo dataset, we’re ready run the analysis. If you’d like to follow along, then please take a look at our git repo. This repo contains the example SciFi data set, the Semantic Search code, and README to get you going. However I’m going execute everything from within Python: >>> from SemanticAnalyzer import * >>> stvc = SolrTermVectorCollector(field='Body',feature='tf',batchSize=1000) >>> tdc = TermDocCollection(source=stvc,numTopics=150) That last line takes a few minutes. If it’s in the AM where you are, grab a coffee. If it’s in the PM, grab a beer. Once that line completes, we will have successfully extracted the term-document matrix from Solr. Now let’s play with it for a bit. One of the cool side effects of this analysis is the ability to quickly find words that commonly occur together. Let’s give it an easy test; here are the 30 most highly correlated words with the word ‘vader’ (as in Darth Vader). >>> tdc.getRelatedTerms('vader',30) Did you notice that pause when you called the function? That was the collaborative filtering taking place. The results of that process have now been saved, so additional calls will return quite quickly. vader luke emperor darth palpatin anakin sith skywalk sidiou apprentic empir luca side star son forc turn kill death rule suit father question jedi command obi tarkin dark wan plan Hey, not bad! Everything here seems very reasonably connected with Mr. Vader. You may notice some odd spellings here; that’s because these are the indexed terms, therefore they are stemmed. Let’s try again with a different term; this time everyone’s favorite wizard: >>> tdc.getRelatedTerms('potter',30) harri potter voldemort wizard snape death magic jame love spell time rowl lili eater travel seri hous hand hogwart three find wormtail kill slytherin hallow secret deathli muggl order lord Again, pretty good! One last try, and we’ll make it a little more challenging – a vague adjective: >>> tdc.getRelatedTerms('dark',30) dark side jedi sith eater lord death mark snape magic curs evil forc luke mercuri cave yoda jame palpatin dagobah anakin black call wizard slytherin live light siriu matter voldemort Indeed, most of these terms are like a hall of fame of dark things from Star Wars and Harry Potter. Now since the word correlation has proven itself out, it’s time to generate the pseudo terms and post them back to Solr. >>> SolrBlurredTermUpdater(tdc,blurredField="BodyBlurred").pushToSolr(0.1) This line will probably see you to the end of your coffee or beer (it takes about 10 minutes on my machine). But once it’s done, you can start issuing searches to Solr. Solr Results Here’s an example of Semantic Search using Solr: http://localhost:8983/solr/select/?q=-Body:dark +BodyBlurred:dark The Body field contains the original text while the BodyBlurred contains the pseudo-terms. So this finds all documents that do not include the term dark, but presumably contain dark content. Take a look at the documents that come back: { Body: " In the John Carter movie (2012), he shows off some of his powers, like jumping abnormally high, but I have difficulty evaluating his strength. On the one side, he shows great strength, as when he kills a thark warrior with one hand, but he is also quite mistreated by them. He also seems helpless when he is strangled by Tars Tarkas. Why does the strength he shows seem so inconsistent? ", BodyBlurred: "tv great movi control kill consid hand dark side power long mutant fight machin light abil sauron wormtail hulk" }, { Body: " In the movies, the Nazgul ride black horses with armour. I was wondering if that is all they are, or do they have some sort of magic? Are they evil? ", BodyBlurred: "movi black magic dark demon engin hous aveng slytherin" }, { Body: " The remaining Black Brother from the prologue of A Game of Thrones is apparently the deserter who is beheaded in the beginning of the book. But how did he manage to get to Winterfell from the other side of The Wall? Or did the show throw me off track and in the book there weren't any survivors, so the deserter is someone else? ", BodyBlurred: "book watch black hole dark side plai long game demon engin light turn district" }, { Body: " Was this ever discussed in any episode, or as a side-plot somewhere? ", BodyBlurred: "episod dark side light" } Not bad – most of those topics are rather … dark. Though check out that last result. So … maybe there are still some improvements we can make! But you also have to remember that we’re dealing with word correlation here, and I can only guess that somewhere else in the corpus, dark side-plots and dark episodes were surely discussed. Speaking of word correlations, check out this gem: { Body: " You're correct, Enterprise is the only Star Trek that fits into both the original and the new 2009 movie timelines. From the perspective of the Enterprise characters, both are possible futures, given the over-arcing conceit of the show was a Temporal Cold War, so its future is in flux and could line up with either of the timelines we're familiar with, or with an entirely different future. ", BodyBlurred: "answer charact place klingon star trek design travel crew watch work movi happen enterpris featur futur exist origin 2009 chang altern timelin war to version event captain gener pictur tng creat iii galaxi theori return alter voyag entir fry turn kirk paradox biff doc marti feder 1955 starship 2015 class hero centuri tempor uss phoenix mirror river 800 ncc 1701 simon conner skynet alisha" } The original document involves Star Trek and time travel. And appropriately, the pseudo terms include Star Trek things and time-travel terms … but do you see anything funny? That’s right, Biff, Doc and Marti made their way into the pseudo terms, likely because of their role in the popular time-travel film “Back to the Future.” Speaking of the future … Future Work Semantic Search with Solr is hot right now. In the upcoming Dublin LuceneRevolution I know of at least three related talks that have been submitted (one of them my own); I have heard that MapR is working on a Solr Semantic Search/Recommendation engine built atop of their Hadoop offering; and I suspect that with Cloudera’s recent foray into Solr with Mark Miller, they will also be working on the same thing. What’s next for our work? Recommendations! Remember, that’s how we started this conversation. E-commerce recommendations is a simple extension of the work presented above. Given an inventory catalog (e.g., product title, description, etc.), and given a history of user purchases, we can build a search-aware recommendation engine. That is, when a customer searches for a particular item, they will receive results as usual, except that the results will be boosted with items that they are more likely to purchase. How? Because we know what type of customer they are and what products that type of customer is more likely to buy! Do you have a good case for Solr Semantic Search and Recommendation? We’d love to hear it, please contact us!

If you work with large, gridded datasets, you should probably be using NetCDF, the Network Common Data Form from Unidata: NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. Lots of high-end analysis software can be made to support NetCDF, and it is indispensable for working with gridded datasets that weigh in at tens of gigabytes or more. This brief post describes the easiest way to install the NetCDF libraries and the R ‘ncdf’ package on our favorite systems: CentOS, Ubuntu and Mac OSX. CentOS 6.x CentOS is the operating system of choice if you want a free, robust, open-source server to host your scientific analysis. It is basically an unbranded clone of Red Hat Enterprise Linux. The following instructions worked on CentOS 6.2. Installing System Libraries First, go to http://fedoraproject.org/wiki/EPEL and check for the latest version of the Extended Packages for Enterprise Linux (which contains NetCDF, HDF and many other useful packages). The latest version should be specified on this page. To download a local copy of EPEL and install NetCDF from it, just execute the following commands: sudo wget http://mirror.metrocast.net/fedora/epel/6/i386/epel-release-6-8.noarch.rpm sudo rpm -Uvh epel-release-6-8.noarch.rpm sudo yum --assumeyes install netcdf sudo yum --assumeyes install netcdf-devel Note that adding EPEL as a package archive in the second line does not automatically install all the packages in EPEL. We had to manually install netcdf and netcdf-dev. A list of other available packages is given at the EPEL wiki given above. Installing R Package ‘ncdf’ With the libraries in place, we can now install the ncdf package for our favorite statistical package — R. sudo wget http://cran.r-project.org/src/contrib/ncdf_1.6.6.tar.gz sudo R CMD INSTALL --configure-args="--with-netcdf-include=/usr/include --with-netcdf-lib=/usr/lib" ncdf_1.6.6.tar.gz Ubuntu 12.04 Ubuntu is easy to install and has a great user interface for linux systems. Ubuntu 12.0.4 is the most recent Long-Term-Stable release. Installing System Libraries The following instructions worked on Ubuntu 12.04 LTS. To install NetCDF libraries that allow reading, writing and manipulation, use apt-get, rather than downloading the source files and installing them yourself. To install, open a terminal and type: sudo apt-get install netcdf Installing R package ‘ncdf’ The base version of R on Ubuntu 12.04 slt is 2.14.1. Unfortunately, clicking the install button in RStudio and typing 'ncdf' will only work at the user level. The package will not be installed for all users or even show up in all of your RStudio projects. To install ncdf tools in the global library you must start up R as root and use the following command: install.packages(repos=c('http://cran.fhcrc.org/'),pkgs=c('ncdf'),lib="/usr/lib/R/site-library/") ‘http://cran.fhcrc.org/’ should be replaced by whichever CRAN mirror is closest to you. OSX 10.8.4 Macs run OSX, which is Unix based. The following instructions worked on OSX 10.8.4 — Mountain Lion. Installing System Libraries The absolute easiest way to install NetCDF on a mac requires Macports. Macports is a software package designed to make installing and compiling software easy. Macports .pkg and installation instructions are available here. Once Macports is installed, building and installing NetCDF libraries is a one-step job. sudo port install netcdf More details and instructions for installing Fortran and Python APIs are available here. Installing R Package ‘ncdf’ Having NetCDF command line tools is not required to use the ncdf R package. Simply download the package from CRAN (link), or by clicking on the “install packages” button in RStudio. This package allows reading, writing and manipulation of existing .nc files. However, the package’s ability to view the content of nc files before loading them into the R workspace is limited. For this reason, installing the NetCDF tools outlined in the first section of this post is extremely important. Command line tools such as “ncdump” are crucial to effectively working with NetCDF files.

ElasticSearch provides Java API, thus it executes all operations asynchronously by using client object.

So, I’ve been wanting to get back to playing with C# for a while, and finally have had the opportunity. I’ve also been wanting to play with the Task library in .NET and see if I could get it to do something interesting, well below is the result. The code below, running in a .NET 4 project, will run two SQL SELECT statements against the AdventureWorks2012 database. There are three tasks in here, ParallelTask 1 and 2, and a timing task. The Parallel task takes a Connection String and a query as inputs, and passes out a Status Message. One of the important points with a task is that the task has to be self contained. This is why the connection is instantiated within the task. I also added in a Timing task (ParallelTiming) so I could pass out a ping message. The whole thing is controlled by the code in the main section, which is used to start the three tasks, with their appropriate parameters. After this it awaits the tasks completing, then passes out the resulting return messages. Try it out; it’s good fun and all you need is SQL Server, AdventureWorks and something to build C# projects. You can download the code here Have fun! /// Parallel_SQL demonstration code /// From Nick Haslam /// http://blog.nhaslam.com /// 16/9/2013 using System; using System.Collections.Generic; using System.Data.SqlClient; using System.Linq; using System.Text; using System.Threading.Tasks; namespace Parallel_SQL { class Program { /// /// First Parallel task /// ///Connection string details ///Query to execute ///Status message to pass back /// static Task ParallelTask1(string sConnString, string sQuery, Action StatusMessage) { return Task.Factory.StartNew(() => { SqlConnection conn = new SqlConnection(sConnString); conn.Open(); StatusMessage(“Running Query”); SqlDataReader reader = null; SqlCommand sqlCommand = new SqlCommand(sQuery, conn); reader = sqlCommand.ExecuteReader(); while (reader.Read()) { StatusMessage(reader[0].ToString()); } return “Task 1 Complete”; }); } /// /// Second Parallel task /// ///Connection string details ///Query to execute ///Status message to pass back /// static Task ParallelTask2(string sConnString, string sQuery, Action StatusMessage) { return Task.Factory.StartNew(() => { SqlConnection conn = new SqlConnection(sConnString); conn.Open(); StatusMessage(“Running Query”); SqlDataReader reader = null; SqlCommand sqlCommand = new SqlCommand(sQuery, conn); reader = sqlCommand.ExecuteReader(); while (reader.Read()) { StatusMessage(reader[0].ToString()); } return “Task 2 Complete”; }); } /// /// Timing Task /// ///Milliseconds between ping ///Status message to pass back /// static Task ParallelTiming(int iMSPause, Action StatusMessage) { return Task.Factory.StartNew(() => { for (int i = 0; i < 10; i++) { System.Threading.Thread.Sleep(iMSPause); StatusMessage(“******************** PING ********************”); } return “Timing task done”; }); } static void Main(string[] args) { string sConnString = “server=.; Trusted_Connection=yes; database=AdventureWorks2012;”; try { var Task1Control = ParallelTask1(sConnString, “SELECT top 500 TransactionID FROM Production.TransactionHistory”, (update) => { Console.WriteLine(String.Format(“{0} – {1}”, DateTime.Now, update)); }); var Task2Control = ParallelTask2(sConnString, “SELECT top 500 SalesOrderDetailID FROM sales.SalesOrderDetail”, (update) => { Console.WriteLine(String.Format(“{0} – \t\t{1}”, DateTime.Now, update)); }); var TimingTaskControl = ParallelTiming(250, (update) => { Console.WriteLine(String.Format(“{0} – \t\t\t{1}”, DateTime.Now, update)); }); // Await Completion of the tasks Console.WriteLine(“Task 1 Status – {0}”, Task1Control.Result); Console.WriteLine(“Task 2 Status – {0}”, Task2Control.Result); Console.WriteLine(“Timing Task Status – {0}”, TimingTaskControl.Result); } catch (Exception e) { Console.WriteLine(e.ToString()); } Console.ReadKey(); } } }

When I was implementing the Elo Rating algorithm a few weeks ago one thing I needed to do was come up with a base ranking for each team. I started out with a set of teams that looked like this: (def teams #{ "Man Utd" "Man City" "Arsenal" "Chelsea"}) and I wanted to transform that into a map from the team to their ranking e.g. Man Utd -> {:points 1200} Man City -> {:points 1200} Arsenal -> {:points 1200} Chelsea -> {:points 1200} I had read the documentation of array-map, a function which can be used to transform a collection of pairs into a map, and it seemed like it might do the trick. I started out by building an array of pairs using mapcat: > (mapcat (fn [x] [x {:points 1200}]) teams) ("Chelsea" {:points 1200} "Man City" {:points 1200} "Arsenal" {:points 1200} "Man Utd" {:points 1200}) array-map constructs a map from pairs of values e.g. > (array-map "Chelsea" {:points 1200} "Man City" {:points 1200} "Arsenal" {:points 1200} "Man Utd" {:points 1200}) ("Chelsea" {:points 1200} "Man City" {:points 1200} "Arsenal" {:points 1200} "Man Utd" {:points 1200}) Since we have a collection of pairs rather than individual pairs we need to use the apply function as well: > (apply array-map ["Chelsea" {:points 1200} "Man City" {:points 1200} "Arsenal" {:points 1200} "Man Utd" {:points 1200}]) {"Chelsea" {:points 1200}, "Man City" {:points 1200}, "Arsenal" {:points 1200}, "Man Utd" {:points 1200} And if we put it all together we end up with the following: > (apply array-map (mapcat (fn [x] [x {:points 1200}]) teams)) {"Man Utd" {:points 1200}, "Man City" {:points 1200}, "Arsenal" {:points 1200}, "Chelsea" {:points 1200} It works but the function we pass to mapcat feels a bit clunky. Since we just need to create a collection of team/ranking pairs we can use the vector and repeat functions to build that up instead: > (mapcat vector teams (repeat {:points 1200})) ("Chelsea" {:points 1200} "Man City" {:points 1200} "Arsenal" {:points 1200} "Man Utd" {:points 1200}) And if we put the apply array-map code back in we still get the desired result: > (apply array-map (mapcat vector teams (repeat {:points 1200}))) {"Chelsea" {:points 1200}, "Man City" {:points 1200}, "Arsenal" {:points 1200}, "Man Utd" {:points 1200} Alternatively we could use assoc like this: > (apply assoc {} (mapcat vector teams (repeat {:points 1200}))) {"Man Utd" {:points 1200}, "Arsenal" {:points 1200}, "Man City" {:points 1200}, "Chelsea" {:points 1200} I also came across the into function which seemed useful but took in a collection of vectors: > (into {} [["Chelsea" {:points 1200}] ["Man City" {:points 1200}] ["Arsenal" {:points 1200}] ["Man Utd" {:points 1200}] ]) We therefore need to change the code to use map instead of mapcat: > (into {} (map vector teams (repeat {:points 1200}))) {"Chelsea" {:points 1200}, "Man City" {:points 1200}, "Arsenal" {:points 1200}, "Man Utd" {:points 1200} However, my favourite version so far uses the zipmap function like so: > (zipmap teams (repeat {:points 1200})) {"Man Utd" {:points 1200}, "Arsenal" {:points 1200}, "Man City" {:points 1200}, "Chelsea" {:points 1200} I’m sure there are other ways to do this as well so if you know any let me know in the comments.

TestNG method that is annotated with @BeforeClass annotation will be run before the first test method in the current class is invoked.

My last couple of blogs have talked about long polling and Spring's DeferredResult technique and to demonstrate these concepts I've shoehorned the code from my Producer Consumer project into a web application. Although the code demonstrates the points made by the blogs it does contain a large number of holes in its logic. Apart from the fact that in a real application you wouldn't use a simple LinkedBlockingQueue, but would choose JMS or some other industrial-strength messaging service, and the fact that only one user can get a hold of the match updates, there's also the problem that it spawns badly behaved threads that don't close down when the JVM terminates. You may wonder why this should be a problem… well to you, as a developer, it isn't really a problem at all, it's only a little bit of sloppy programming. But, to one of your operations guys it can make life unnecessarily difficult. The reason for this is that if you have too many badly behaved threads then typing Tomcat's shutdown.sh command will have very little effect and you have to savagely kill your web server by typing something like: ps -ef | grep java to get the pid and then kill -9 <> …and when you have a field of Tomcat web servers to restart all this extra kerfuffle that becomes a severe pain. When you type shutdown.sh you want Tomcat to stop. In my last couple of blogs the badly behaved threads I created had the following run() methods with the first of these, shown below, being really badly behaved: @Override public void run() { while (true) { try { DeferredResult result = resultQueue.take(); Message message = queue.take(); result.setResult(message); } catch (InterruptedException e) { throw new UpdateException("Cannot get latest update. " + e.getMessage(), e); } } } In this code I've used an infinite while(true), which means that the thread will just keep running and never terminate. @Override public void run() { sleep(5); // Sleep to allow the reset of the app to load logger.info("The match has now started..."); long now = System.currentTimeMillis(); List matchUpdates = match.getUpdates(); for (Message message : matchUpdates) { delayUntilNextUpdate(now, message.getTime()); logger.info("Add message to queue: {}", message.getMessageText()); queue.add(message); } start = true; // Game over, can restart logger.warn("GAME OVER"); } The second example, above, is also pretty badly behaved. It'll keep taking messages from MatchUpdates list and adding them to the message queue at the appropriate moment. Their only saving grace is that they may throw an InterruptedException, which if handled correctly would cause thread termination; however, this cannot be guaranteed. There's a quick fix for this, really… all you need to do is to ensure that any threads you create are daemon threads. The definition of a daemon thread is a thread that doesn't prevent the JVM from exiting when the program finishes but, the thread is still running. The usual example of a daemon thread is the JVM's garbage collection thread. To turn your threads into daemon threads you simply call: thread.setDaemon(true); ...and when you type shutdown.sh then, WHAM, all your threads will disappear. There is, however, a problem with this. What if one of your daemon's threads was doing something important and chopping it down in its prime, resulting in the loss of some pretty important data? What you need to do is to ensure that all your threads shut down gracefully, completing any work they may be currently undertaking. The rest of this blog demonstrates a fix for these errant threads, gracefully coordinating their shutdown by using a ShutdownHook. According to the documentation, a "shutdown hook is simply an initialized but unstarted thread. When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently." So, after reading the last sentence you may have guessed that what you need to do is to create a thread that has the responsibility of shutting down all your other threads and is passed to the JVM as a shutdown hook. All of this can be generically implemented in a couple of small classes and by performing some jiggery-pokery on your existing thread run() methods. The two classes to create are a ShutdownService and a Hook. The Hook class, which I'll demonstrate first, is used to link the ShutdownService to your threads. The code for Hook is as follows: public class Hook { private static final Logger logger = LoggerFactory.getLogger(Hook.class); private boolean keepRunning = true; private final Thread thread; Hook(Thread thread) { this.thread = thread; } /** * @return True if the daemon thread is to keep running */ public boolean keepRunning() { return keepRunning; } /** * Tell the client daemon thread to shutdown and wait for it to close gracefully. */ public void shutdown() { keepRunning = false; thread.interrupt(); try { thread.join(); } catch (InterruptedException e) { logger.error("Error shutting down thread with hook", e); } } } The Hook contains two instance variables: keepRunning and thread. thread is a reference to the thread that this instance of Hook is responsible for shutting down, while keepRunning tells the thread to… keep running. Hook has two public methods: keepRunning() and shutdown(). keepRunning() is called by the thread to figure out whether it should keep running, and shutdown() is called by the ShutdownService's shutdown hook thread to get your thread to shut down. This is the most interesting of the two methods. Firstly it sets the keepRunning variable to false. It then calls thread.interrupt() to interrupt the thread forcing it to throw an InterruptedException. Lastly, it calls thread.join() and waits for the thread instance to shutdown. Note that this technique relies on all your threads cooperating. If there's one badly behaved thread in the mix, then the whole thing could hang. To get around this problem add a timeout to thread.join(…). @Service public class ShutdownService { private static final Logger logger = LoggerFactory.getLogger(ShutdownService.class); private final List hooks; public ShutdownService() { logger.debug("Creating shutdown service"); hooks = new ArrayList(); createShutdownHook(); } /** * Protected for testing */ @VisibleForTesting protected void createShutdownHook() { ShutdownDaemonHook shutdownHook = new ShutdownDaemonHook(); Runtime.getRuntime().addShutdownHook(shutdownHook); } protected class ShutdownDaemonHook extends Thread { /** * Loop and shutdown all the daemon threads using the hooks * * @see java.lang.Thread#run() */ @Override public void run() { logger.info("Running shutdown sync"); for (Hook hook : hooks) { hook.shutdown(); } } } /** * Create a new instance of the hook class */ public Hook createHook(Thread thread) { thread.setDaemon(true); Hook retVal = new Hook(thread); hooks.add(retVal); return retVal; } @VisibleForTesting List getHooks() { return hooks; } } The ShutdownService is a Spring service that contains a list of Hook classes, and therefore by inference threads, that it is responsible for shutting down. It also contains an inner class ShutdownDaemonHook, which extends Thread. An instance of ShutdownDaemonHook is created during the construction of ShutdownService, which is then passed to the JVM as a shutdown hook by calling: Runtime.getRuntime().addShutdownHook(shutdownHook); The ShutdownService has one public method: createHook(). The first thing that this class does is to ensure that any thread passed to it is converted into a daemon thread. It then creates a new Hook instance, passing in the thread as the argument, before finally both storing the result in a list and returning it to the caller. The only thing left to do now is to integrate the ShutdownService into DeferredResultService and MatchReporter, the two classes that contain the badly behaved threads. @Service("DeferredService") public class DeferredResultService implements Runnable { private static final Logger logger = LoggerFactory.getLogger(DeferredResultService.class); private final BlockingQueue> resultQueue = new LinkedBlockingQueue<>(); private Thread thread; private volatile boolean start = true; @Autowired private ShutdownService shutdownService; private Hook hook; @Autowired @Qualifier("theQueue") private LinkedBlockingQueue queue; @Autowired @Qualifier("BillSkyes") private MatchReporter matchReporter; public void subscribe() { logger.info("Starting server"); matchReporter.start(); startThread(); } private void startThread() { if (start) { synchronized (this) { if (start) { start = false; thread = new Thread(this, "Studio Teletype"); hook = shutdownService.createHook(thread); thread.start(); } } } } @Override public void run() { logger.info("DeferredResultService - Thread running"); while (hook.keepRunning()) { try { DeferredResult result = resultQueue.take(); Message message = queue.take(); result.setResult(message); } catch (InterruptedException e) { System.out.println("Interrupted when waiting for latest update. " + e.getMessage()); } } System.out.println("DeferredResultService - Thread ending"); } public void getUpdate(DeferredResult result) { resultQueue.add(result); } } The first change to this class was to autowire in the Shutdown service instance. The next thing to do is to use the ShutdownService to create an instance of Hook after the creation of the thread but before thread.start() is called: thread = new Thread(this, "Studio Teletype"); hook = shutdownService.createHook(thread); thread.start(); The final change is to replace while(true) with: while (hook.keepRunning()) { … telling the thread when to quit the while loop and shutdown. You may have also noticed that there are a few System.out.println() calls thrown into the above code. There is a reason for this and it's because of the undetermined order in which the shutdown hook threads are executed. Remember that not only are your classes trying to shutdown gracefully, but other sub-systems and shutting down too. This means that my original code, which called logger.info(…) failed throwing the following exception: Exception in thread "Studio Teletype" java.lang.NoClassDefFoundError: org/apache/log4j/spi/ThrowableInformation at org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:159) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.log(Category.java:856) at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:382) at com.captaindebug.longpoll.service.DeferredResultService.run(DeferredResultService.java:75) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.ClassNotFoundException: org.apache.log4j.spi.ThrowableInformation at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559) ... 6 more This is because the logger has already been unloaded when I try to call it; hence the failure.Again, as the documentation states: "Shutdown hooks run at a delicate time in the life cycle of a virtual machine and should therefore be coded defensively. They should, in particular, be written to be thread-safe and to avoid deadlocks insofar as possible. They should also not rely blindly upon services that may have registered their own shutdown hooks and therefore may themselves in the process of shutting down. Attempts to use other thread-based services such as the AWT event-dispatch thread, for example, may lead to deadlocks." The MatchReport class has some very similar modifications. The major difference is that the hook.keepRunning() code is inside the run() method's for loop. public class MatchReporter implements Runnable { private static final Logger logger = LoggerFactory.getLogger(MatchReporter.class); private final Match match; private final Queue queue; private volatile boolean start = true; @Autowired private ShutdownService shutdownService; private Hook hook; public MatchReporter(Match theBigMatch, Queue queue) { this.match = theBigMatch; this.queue = queue; } /** * Called by Spring after loading the context. Will "kick off" the match... */ public void start() { if (start) { synchronized (this) { if (start) { start = false; logger.info("Starting the Match Reporter..."); String name = match.getName(); Thread thread = new Thread(this, name); hook = shutdownService.createHook(thread); thread.start(); } } } else { logger.warn("Game already in progress"); } } /** * The main run loop */ @Override public void run() { sleep(5); // Sleep to allow the reset of the app to load logger.info("The match has now started..."); long now = System.currentTimeMillis(); List matchUpdates = match.getUpdates(); for (Message message : matchUpdates) { delayUntilNextUpdate(now, message.getTime()); if (!hook.keepRunning()) { break; } logger.info("Add message to queue: {}", message.getMessageText()); queue.add(message); } start = true; // Game over, can restart logger.warn("GAME OVER"); } private void sleep(int deplay) { try { TimeUnit.SECONDS.sleep(10); } catch (InterruptedException e) { logger.info("Sleep interrupted..."); } } private void delayUntilNextUpdate(long now, long messageTime) { while (System.currentTimeMillis() < now + messageTime) { try { Thread.sleep(100); } catch (InterruptedException e) { logger.info("MatchReporter Thread interrupted..."); } } } } The ultimate test of this code is to issue a Tomcat shutdown.sh command half way through the match update sequence. As the JVM terminates it'll call the shutdown hook from the ShutdownDaemonHook class. As this class's run() method executes it loops throughout the list of Hook instances telling them to close down their respective threads. If you tail -f your server's log file (in my case catalina.out, but your Tomcat maybe configured differently to mine), you'll see the trail of entries shutting your server shutdown gracefully.