Caching may address new challenges when developing performing applications.

In theory everyone knows Hash Map is not Thread Safe and it shouldn’t be used in multi Threaded applications. But still people come out with their own theories that they can use HashMap in their context. Some say they are just reading the data and map is not written to a lot. Unfortunately none of these explanations holds good when one lands up in a synchronization issue. Normally most of the guys do not understand fundamentals around Java Memory Model and Concurrency .One cannot blame them for not knowing their fundamentals as its hard for people to visualize concurrent executions since from college days we are used to sequential executions of program. Enuf of blame game, now lets look into some code.Have a look at the code mentioned below and come out with all your theories of what can go wrong with it or theories which say its all correct. public class MapTestTask implements Runnable { private Map hashMap; private Object value = new Object(); public MapTestTask(Map map) { this.hashMap = map; } public void run() { hashMap.put(Thread.currentThread(), value); Object retrieved = hashMap.get(Thread.currentThread()); if (retrieved == null) { // Can it ever Happen } } } Now question is when we run multiple such Threads can we ever see retrieved as null. If we look from sequential point of view it can never happen.But when concurrency comes into picture this can happen. I will give you a code with which can reproduce this scenario. This is my sincere advise : do not over engineer when Concurreny is involved. Because none of the theories will stand the test of time in a concurrent environment. As a rule of thumb use Thread safe collections wherever concurrency is involved. Finally i will leave you with this interesting bug in Java which says HashMap can get into infinite loop when used in muti Threaded Environment which further reiterates my point that not all the possible scenarios can be visualized in a multi threaded environment and therefore one should rely on basics rather than trying a smart creativity which has high probability of landing into trouble at some point in future. Source Code for Reproducing package test; import java.util.Map; public class MapTestTask implements Runnable { private Map hashMap; private Object value = new Object(); public MapTestTask(Map map) { this.hashMap = map; } public void run() { hashMap.put(Thread.currentThread(), value); Object retrieved = hashMap.get(Thread.currentThread()); if (retrieved == null) { // Can it ever Happen System.out.println("Oh My God it can happen."); } } } package test; import java.util.Map; import java.util.HashMap; public class TestMap { public static void main(String[] args) throws InterruptedException { Map map = new HashMap(); int NUM_THREADS = 1000; Thread[] threads = new Thread[NUM_THREADS]; for (int i = 0; i < NUM_THREADS; i++) { threads[i] = new Thread(new MapTestTask(map)); } for (int i = 0; i < NUM_THREADS; i++) { threads[i].start(); } for (int i = 0; i < NUM_THREADS; i++) { threads[i].join(); } } } From http://pitfalls.wordpress.com/2008/06/29/concurrencyandhashmap/

Glimmer is a JRuby DSL that enables easy and efficient authoring of user-interfaces using the robust platform-independent Eclipse SWT library. Glimmer comes with built-in data-binding support to greatly facilitate synchronizing UI with domain models. The goal of the Glimmer project is to create a JRuby framework on top of Eclipse technologies to enable easy and efficient authoring of desktop applications by taking advantage of the Ruby language. With Glimmer having just become an Eclipse project, it's a good time to find out more. Philosophy Glimmer's design philosophy can be summarized as follows: Concise and DRY Asks for minimum info needed to accomplish task Convention over configuration As predictable as possible for existing SWT developers Conventions Since Glimmer relies on Ruby, it is different in its syntax and conventions from what typical Java SWT developers would expect: Method parentheses are optional Java-vs-Ruby example: show() => show Method names follow underscored syntax Java-vs-Ruby example: addListener => add_listener Classes are constructed using the new(...) method (as opposed to new keyword): Java-vs-Ruby example: new GridLayout() => GridLayout.new Download Please download Glimmer from RubyForge: https://rubyforge.org/projects/glimmer/ NOTE: Glimmer is moving to Eclipse.org. Please visit http://andymaleh.blogspot.com for up-to-date news on the move and the upcoming download location on the Eclipse website. Installation Extract the Glimmer zip file and follow the installation instructions in the README file. NOTE: While Glimmer is platform-independent, its functionality has only been verified on Windows. Feedback from Mac and Linux users would be greatly appreciated. Tutorial Let's start with a very simple Glimmer Hello World example: shell { label { text “Hello World!” } } This will render the following: [img_assist|nid=3586|title=|desc=|link=none|align=undefined|width=126|height=48] In the SWT library a shell represents an application's window. It acts as a frame around the application widgets, which are visual components that display information and/or enable interaction with the user. One widget that was used in the Hello World example is the label widget, which simply displays text on the screen. Shell is also considered a widget, except it is a special kind of widget called composite. The shell keyword, which declared the application's shell, was followed by a block of code encased in curly braces. This block contains the shell content declarations, such as the Hello World label. The label keyword was also followed by a block of code. However, this block contained a property declaration for the label, stating that the text value is “Hello World!” So, to declare a widget, simply state its name followed by a block of code. The block may specify property values or nest other widget declarations for composite widgets. Now, let's move on to a more advanced example: shell { text "User Profile" composite { layout GridLayout.new(2, false) group { text "Name" layout GridLayout.new(2, false) layout_data GridData.new(fill, fill, true, true) label {text "First"}; text {text "Bullet"} label {text "Last"}; text {text "Tooth"} } group { layout_data GridData.new(fill, fill, true, true) text "Gender" button(radio) {text "Male"; selection true} button(radio) {text "Female"} } group { layout_data GridData.new(fill, fill, true, true) text "Role" button(check) {text "Student"; selection true} button(check) {text "Employee"; selection true} } group { text "Experience" layout RowLayout.new layout_data GridData.new(fill, fill, true, true) spinner {selection 5}; label {text "years"} } button { text "save" layout_data GridData.new(right, center, true, true) } button { text "close" layout_data GridData.new(left, center, true, true) } } }.open This will render the following: [img_assist|nid=3587|title=|desc=|link=none|align=undefined|width=195|height=209] The example contains a variation of widgets from SWT: Composite: a widget that can simply contain other widgets and manage their layout Group: Similar to Composite except that it usually has a border and a title. Text Field: Enables user to type in text information Checkbox Button: Allows user to make a selection from different options Radio Button: Allows user to make a selection between options that are mutually exclusive Spinner: Enables user to type in numeric information or spinning a number selection by mouse Push Button: Enables user to initiate actions Given that Glimmer relies on the Eclipse SWT library, developers may consult the SWT API as a reference on all the widgets, including their properties and layout options: http://help.eclipse.org/stable/nftopic/org.eclipse.platform.doc.isv/reference/api/index.html Keep in mind the following rules when reading the SWT API: Any widget available in SWT, including custom widgets written by developers, can be accessed from Glimmer by downcasing/underscoring the widget's name (e.g. Composite -> composite, LabledText -> labeled_text) Properties available on SWT widgets are specified by listing them followed by their values, each on a line or separated by semicolons within the widget's block (e.g. label {text "Username:"; font some_font}) Property names are also downcased/underscored in Glimmer. SWT widgets must have a style value specified, which is a constant available on the “SWT” class. Glimmer generally hides that by relying on smart defaults. Here is a listing of the defaults configured in Glimmer: text: SWT::BORDER table: SWT::BORDER spinner: SWT::BORDER button: SWT::PUSH Nonetheless, to customize a widget, a style value may be optionally specified within parentheses after the widget name. For an example, “button(SWT::RADIO)” renders a radio button and “button(SWT::CHECK)” renders a checkbox button. Glimmer's syntax also has syntactic sugar for specifying the style. Simply state the name of the style in the standard Ruby downcased/underscored format without the “SWT::” prefix. For example, button(SWT::RADIO) becomes button(radio). SWT composite widgets, such as shell, composite, and group can have a layout manager that lays out child widgets according to a certain pattern without the need to specify the (x, y) position of each child widget explicitly. Layout managers come in many flavors, such as GridLayout, offering a grid-like layout; FillLayout, allowing child widgets to fill the whole available area; and RowLayout, rendering child widgets one after the other in a row by default. Glimmer is configured with smart defaults for layout managers too: shell: FillLayout composite: GridLayout with one column group: GridLayout with one column GridLayout is a particularly useful SWT layout, so I will go over it in a little more detail here. GridLayout allows you to lay widgets out in a grid similar to HTML tables. To instantiate a custom GridLayout, you must specify the number of columns and whether they are of equal width or not. Here is a block of code demonstrating a group box having a GridLayout with 2 columns of unequal width: group { layout GridLayout.new(2, false) } Now, suppose we add four elements to that group box: group { layout GridLayout.new(2, false) label {text "First"}; text {text "Bullet"} label {text "Last"}; text {text "Tooth"} } The specified GridLayout will lay out the child widgets in the grid from left to right and top to bottom: The label with the text “First” will go into the 1st column of the 1st row. The text box with the text “Bullet” will go into the 2nd column of the 1st row. The label with the text “Last” will go into the 1st column of the 2nd row. The text box with the text “Tooth” will go into the 2nd column of the 2nd row. The group was actually a part of the advanced example illustrated earlier. It was given a title (by specifying the text attribute,) and the widget declarations were written in a way that maps visually to how they appear on the screen. Notice how text box declarations are on the same line as the label declarations since both the label and text box go under the same row, which helps improve code readability and maintainability: group { text "Name" layout GridLayout.new(2, false) layout_data GridData.new(fill, fill, true, true) label {text "First"}; text {text "Bullet"} label {text "Last"}; text {text "Tooth"} } That renders the following: [img_assist|nid=3588|title=|desc=|link=none|align=undefined|width=89|height=73] Layout of specific widgets may be further customized by specifying layout data. For GridLayout, layout data is specified through GridData objects. For example, we may decide to have the text boxes in the previous example have a greater width: group { layout GridLayout.new(2, false) label {text "First"}; text { text "Bullet" layout_data GridData.new(100, default) } label {text "Last"}; text { text "Tooth" layout_data GridData.new(100, default) } } This renders the following: [img_assist|nid=3589|title=|desc=|link=none|align=undefined|width=125|height=73] The used GridData constructor takes two parameters: width hint and height hint. The width was set to 100 pixels for both text boxes. The height was kept at the default value (SWT::DEFAULT) For more details about GridLayout, GridData, and other layout managers, please refer to the SWT API documentation. So far we have covered how to construct user-interfaces that can display data and gather input from the user. Next, we will demonstrate how to perform work based on actions taken by the user. SWT widgets can be monitored for certain user-interface events, such as mouse clicks, focus gain and loss, and key presses. With the original SWT API, events can be monitored by adding listeners to widgets. For example, to monitor the push of a button, you would add a SelectionListener that does some work in its widgetSelected event method. With Glimmer, events can be monitored by declaring their name (following Ruby conventions) prefixed by “on” Here is an example of how to monitor button selection: import org.eclipse.swt.widgets.MessageBox @shell1 = shell { composite { button { text 'Save' on_widget_selected { message_box = MessageBox.new(@shell1.widget, SWT::NULL) message_box.text = 'Information' message_box.message = 'Saved!' message_box.open } } } } @shell1.open This renders the following: [img_assist|nid=3590|title=|desc=|link=none|align=undefined|width=170|height=145] On click of the button, a message box is opened to let the user know that the information entered is saved. MessageBox is a class from SWT that represents message dialogs. It was imported using the JRuby import method. Its constructor takes a parent and style. To obtain the parent, we assigned the shell object to a Ruby class variable @shell1. Since Glimmer wraps all SWT constructed objects with Glimmer decorators ( e.g. Shell is wrapped with RShell,) to obtain the SWT Shell class and pass it as the parent to the MessageBox constructor, the widget method was called (e.g. @shell1.widget.) In the original SWT API, MessageBox has setter methods to set its text and message attributes. However in JRuby, the developer has the option to set them following the Ruby attribute conventions (e.g. message_box.text = 'value') because JRuby automatically enhances all Java objects with methods that follow the Ruby convention. Another example that benefits from event monitoring is field validation on loss of focus. For example, let's say we are validating the ZIP code on an address form, and we would like to display an error message if its value does not have a valid ZIP code format (e.g. 12345 or 12345-1234,) here is how we would do it with Glimmer (please add the following code before the button in the previous example): import org.eclipse.swt.widgets.MessageBox @shell1 = shell { composite { label { text "ZIP Code" } text { on_focus_lost { |focus_event| zip_code = focus_event.widget.text unless zip_code =~ /^\d{5}([-]\d{4})?$/ message_box = MessageBox.new(@shell1.widget, SWT::NULL) message_box.text = 'Validation Error' message_box.message = 'Format must match ##### or #####-####' message_box.open focus_event.widget.set_focus end } } button { text 'Save' on_widget_selected { message_box = MessageBox.new(@shell1.widget, SWT::NULL) message_box.message = 'Saved!' message_box.open } } } } @shell1.open Here is what it produces: [img_assist|nid=3591|title=|desc=|link=none|align=undefined|width=346|height=151] Notice how the on_focus_lost block has a FocusEvent object as a parameter. This parameter may be specified optionally whenever some information is needed from the event object. Again, this maps to the focusLost method on the FocusListener class in the original SWT API, which also takes a FocusEvent object as a parameter. While widgets in the original SWT API have a setFocus event to grab the user interface focus, in JRuby set_focus may be used instead following the Ruby naming conventions. Now, in order to cleanly separate event-driven behavior from user-interface code, we can rely on Glimmer's data-binding support. Stay tuned for the next tutorial, which will cover data-binding and how to achieve clean code separation with the Model-View-Presenter pattern. References: Glimmer Eclipse Technology Project Proposal: http://www.eclipse.org/proposals/glimmer/ Glimmer Newsgroup: http://www.eclipse.org/newsportal/thread.php?group=eclipse.technology.glimmer Glimmer at RubyForge: http://rubyforge.org/projects/glimmer/ Author Blog: http://andymaleh.blogspot.com Andy Maleh (andy at obtiva.com), Senior Consultant, Obtiva Corp.

I have been searching for a method to dynamically route objects to databases at runtime using Hibernate and recently I found a solution which fit the bill.

Last few months I have seen too much code where a HashMap (without any extra synchronization) is used instead of a thread-safe alternative like the ConcurrentHashMap or the less concurrent but still thread-safe HashTable. This is an example of a HashMap used in a home grown cache (used in a multi-threaded environment): interface ValueProvider{V retrieve(K key);}public class SomeCache{private Map map = new HashMap();private ValueProvider valueProvider;public SomeCache(ValueProvider valueProvider){this.valueProvider = valueProvider;}public V getValue(K key){V value = map.get(key);if(value == null){value = valueProvider.get(key);if(value!=null)map.put(key,value);}return value;} There is much wrong with this innocent looking piece of code. There is no happens before relation between the put of the value in the map, and the get of the value. This means that a thread that receives the value from the cache, doesn’t need to see all fields if the value has publication problems (most non thread-safe structures have publication problems). The same goes for the value and the internals (the buckets for example) of the HashMap. This means that updates to the internals of the HashMap while putting, don’t need to be visible to a thread that does the get. So it could be that the state of the cache in main memory is not in an allowed state (some of the changes maybe are stuck in the cpu-cache), and the cache could start behaving erroneous and if you are lucky starts throwing exceptions. And last, but certainly not least, there also is a classic race problem: if 2 threads do a interleaved map.put, the internals of the HashMap can get in an inconsistent state. In most cases an application reboot/redeploy would be the only way to fix this problem. There are other problems with the cache behavior of this code as well. The items don’t have a timeout, so once a value gets in the cache, it stays in the cache. In practice this could lead to web-page that keeps displaying some value, even though in the main repository the value has been updated. An application reboot also is the only way to solve this problem. Using a Common Of The Shelf (COTS) cache would be a much saver solution, even though a new library needs to be added. It is important to realize that a HashMap can be used perfectly in a multi-threaded environment if extra synchronization is added. But without extra synchronization, it is a time-bomb waiting to go off.

The hardest part about learning Hbase (the open source implementation of Google's BigTable), is just wrapping your mind around the concept of what it actually is. I find it rather unfortunate that these two great systems contain the words table and base in their names, which tend to cause confusion among RDBMS indoctrinated individuals (like myself). This article aims to describe these distributed data storage systems from a conceptual standpoint. After reading it, you should be better able to make an educated decision regarding when you might want to use Hbase vs when you'd be better off with a "traditional" database. It's all in the terminology Fortunately, Google's BigTable Paper clearly explains what BigTable actually is. Here is the first sentence of the "Data Model" section: A Bigtable is a sparse, distributed, persistent multidimensional sorted map. Note: At this juncture I like to give readers the opportunity to collect any brain matter which may have left their skulls upon reading that last line. The BigTable paper continues, explaining that: The map is indexed by a row key, column key, and a timestamp; each value in the map is an uninterpreted array of bytes. Along those lines, the HbaseArchitecture page of the Hadoop wiki posits that: HBase uses a data model very similar to that of Bigtable. Users store data rows in labelled tables. A data row has a sortable key and an arbitrary number of columns. The table is stored sparsely, so that rows in the same table can have crazily-varying columns, if the user likes. Although all of that may seem rather cryptic, it makes sense once you break it down a word at a time. I like to discuss them in this sequence: map, persistent, distributed, sorted, multidimensional, and sparse. Rather than trying to picture a complete system all at once, I find it easier to build up a mental framework piecemeal, to ease into it... map At its core, Hbase/BigTable is a map. Depending on your programming language background, you may be more familiar with the terms associative array (PHP), dictionary (Python), Hash (Ruby), or Object (JavaScript). From the wikipedia article, a map is "an abstract data type composed of a collection of keys and a collection of values, where each key is associated with one value." Using JavaScript Object Notation, here's an example of a simple map where all the values are just strings: { "zzzzz" : "woot", "xyz" : "hello", "aaaab" : "world", "1" : "x", "aaaaa" : "y" } persistent Persistence merely means that the data you put in this special map "persists" after the program that created or accessed it is finished. This is no different in concept than any other kind of persistent storage such as a file on a filesystem. Moving along... distributed Hbase and BigTable are built upon distributed filesystems so that the underlying file storage can be spread out among an array of independent machines. Hbase sits atop either Hadoop's Distributed File System (HDFS) or Amazon's Simple Storage Service (S3), while a BigTable makes use of the Google File System (GFS). Data is replicated across a number of participating nodes in an analogous manner to how data is striped across discs in a RAID system. For the purpose of this article, we don't really care which distributed filesystem implementation is being used. The important thing to understand is that it is distributed, which provides a layer of protection against, say, a node within the cluster failing. sorted Unlike most map implementations, in Hbase/BigTable the key/value pairs are kept in strict alphabetical order. That is to say that the row for the key "aaaaa" should be right next to the row with key "aaaab" and very far from the row with key "zzzzz". Continuing our JSON example, the sorted version looks like this: { "1" : "x", "aaaaa" : "y", "aaaab" : "world", "xyz" : "hello", "zzzzz" : "woot" } Because these systems tend to be so huge and distributed, this sorting feature is actually very important. The spacial propinquity of rows with like keys ensures that when you must scan the table, the items of greatest interest to you are near each other. This is important when choosing a row key convention. For example, consider a table whose keys are domain names. It makes the most sense to list them in reverse notation (so "com.jimbojw.www" rather than "www.jimbojw.com") so that rows about a subdomain will be near the parent domain row. Continuing the domain example, the row for the domain "mail.jimbojw.com" would be right next to the row for "www.jimbojw.com" rather than say "mail.xyz.com" which would happen if the keys were regular domain notation. It's important to note that the term "sorted" when applied to Hbase/BigTable does not mean that "values" are sorted. There is no automatic indexing of anything other than the keys, just as it would be in a plain-old map implementation. multidimensional Up to this point, we haven't mentioned any concept of "columns", treating the "table" instead as a regular-old hash/map in concept. This is entirely intentional. The word "column" is another loaded word like "table" and "base" which carries the emotional baggage of years of RDBMS experience. Instead, I find it easier to think about this like a multidimensional map - a map of maps if you will. Adding one dimension to our running JSON example gives us this: { "1" : { "A" : "x", "B" : "z" }, "aaaaa" : { "A" : "y", "B" : "w" }, "aaaab" : { "A" : "world", "B" : "ocean" }, "xyz" : { "A" : "hello", "B" : "there" }, "zzzzz" : { "A" : "woot", "B" : "1337" } } In the above example, you'll notice now that each key points to a map with exactly two keys: "A" and "B". From here forward, we'll refer to the top-level key/map pair as a "row". Also, in BigTable/Hbase nomenclature, the "A" and "B" mappings would be called "Column Families". A table's column families are specified when the table is created, and are difficult or impossible to modify later. It can also be expensive to add new column families, so it's a good idea to specify all the ones you'll need up front. Fortunately, a column family may have any number of columns, denoted by a column "qualifier" or "label". Here's a subset of our JSON example again, this time with the column qualifier dimension built in: { // ... "aaaaa" : { "A" : { "foo" : "y", "bar" : "d" }, "B" : { "" : "w" } }, "aaaab" : { "A" : { "foo" : "world", "bar" : "domination" }, "B" : { "" : "ocean" } }, // ... } Notice that in the two rows shown, the "A" column family has two columns: "foo" and "bar", and the "B" column family has just one column whose qualifier is the empty string (""). When asking Hbase/BigTable for data, you must provide the full column name in the form ":". So for example, both rows in the above example have three columns: "A:foo", "A:bar" and "B:". Note that although the column families are static, the columns themselves are not. Consider this expanded row: { // ... "zzzzz" : { "A" : { "catch_phrase" : "woot", } } } In this case, the "zzzzz" row has exactly one column, "A:catch_phrase". Because each row may have any number of different columns, there's no built-in way to query for a list of all columns in all rows. To get that information, you'd have to do a full table scan. You can however query for a list of all column families since these are immutable (more-or-less). The final dimension represented in Hbase/BigTable is time. All data is versioned either using an integer timestamp (seconds since the epoch), or another integer of your choice. The client may specify the timestamp when inserting data. Consider this updated example utilizing arbitrary integral timestamps: { // ... "aaaaa" : { "A" : { "foo" : { 15 : "y", 4 : "m" }, "bar" : { 15 : "d", } }, "B" : { "" : { 6 : "w" 3 : "o" 1 : "w" } } }, // ... } Each column family may have its own rules regarding how many versions of a given cell to keep (a cell is identified by its rowkey/column pair) In most cases, applications will simply ask for a given cell's data, without specifying a timestamp. In that common case, Hbase/BigTable will return the most recent version (the one with the highest timestamp) since it stores these in reverse chronological order. If an application asks for a given row at a given timestamp, Hbase will return cell data where the timestamp is less than or equal to the one provided. Using our imaginary Hbase table, querying for the row/column of "aaaaa"/"A:foo" will return "y" while querying for the row/column/timestamp of "aaaaa"/"A:foo"/10 will return "m". Querying for a row/column/timestamp of "aaaaa"/"A:foo"/2 will return a null result. sparse The last keyword is sparse. As already mentioned, a given row can have any number of columns in each column family, or none at all. The other type of sparseness is row-based gaps, which merely means that there may be gaps between keys. This, of course, makes perfect sense if you've been thinking about Hbase/BigTable in the map-based terms of this article rather than perceived similar concepts in RDBMS's. And that's about it Well, I hope that helps you understand conceptually what the Hbase data model feels like. As always, I look forward to your thoughts, comments and suggestions.

When asking the question how does functional programming help me with concurrent programming? The standard response tends to be functional programming use immutable data structures, read-only data structures can be shared between threads without issues, end of problem. Except it isn’t. Immutable data structures have a different set of problems associated with them when working on concurrent problems. This post will examine what these problems are, and then show that this is just a special case of a more general set of problems when working with immutable data structures. Finally will start taking a look at how we solve some of these problems, but in a single thread environment first of all. First let’s frame the problem by looking at how imperative programs work with threads. In a classic imperative/OO languages programmers tend to use either instance or static member variables to send messages between threads, let’s look a fragment of C# that does something classically multi-threaded: object workQueueLock = new object(); Queue workQueue = new Queue(); // this method runs on its own thread private void Worker() { while (true) { // define a work item attempt to retrive work from the queue WorkItem item = null; lock (workQueueLock) { if (workQueue.Count > 0) { item = workQueue.Dequeue(); } } // check if we have work to do, otherwise sleep if (item != null) { // do some work } else { Thread.Sleep(QueuePollInterval); } } } // an even that resets our flag private void Some_Event(object sender, WorkEventArgs ea) { lock (workQueueLock) { workQueue.Enqueue(ea.WorkItem); } } I wouldn’t recommend you use this naive version of a work queue, but the above code is straight forward enough to understand easily and illustrate the typical way imperative programs communicate between threads. We have a member variable “workQueue” that controls stores the work to be done, the method “Worker” is designed to read from this queue and if there’s some work to do, do the work, otherwise sleep till it’s time to poll the queue again. We use “workQueue” again in “Some_event” to send a message to “Worker”, to enqueue some work for it to do. It’s easy to see that mutation of the variable “workQueue” is essential to get this to work; if we couldn’t change the content of “workQueue” then we couldn’t send the message. It’s also easy to see that we now have a huge number of implementation choices: Do we lock on the queue, or have separate lock? What’s the shortest possible time we can hold the lock for (to avoid other threads be blocked when they want to write to the queue)? How long do we sleep for before polling the queue? Too shorter time and we risk wasting too much processor time polling the queue, too longer time and risk that the queue because unreactive because the worker wastes too much time sleeping when there’s work to be done. In pure functional programming there are no variables or mutation, so the above scenario simply isn’t possible. Sure, F# isn’t a pure function language, actually most functional languages aren’t, so you can indeed use mutable data structures to implement something similar to the C# fragment we showed earlier, but that’s not the point we want to learn how to use immutable data structures. To fully understand the limitations of immutable data structures, let’s look at another C# example do something simpler. Imagine that we want compute a key of a value then store it in a member variable, a dictionary in this case, for later use: Dictionary myDict = new Dictionary(); public void ReceiveValue(string val) { myDict.Add(ComputerKey(val), val); } Now let’s think about how we can translate this into F#. Firstly, if don’t mind being dirty and mutable we can translate this fragment verbatim: type Store() = let myDict = new Dictionary() member x.ReceiveValue (value:string) = myDict.Add(x.ComputeKey value, value) However, if we don’t want to be mutable it’s not quite so straight forward. F# contains a type called “Map”, which is very similar to a Dictionary except that it is immutable. When you add a new item to a map you don’t change the map you create a new version of the map with the new key added. So here is how our store class would look to if we used an immutable “Map” data structure: type ComputeKeys(myDict:Map) = member x.ReceiveValue (value:string) = new ComputeKeys(myDict.Add(x.ComputeKey value, value)) The important thing to notice is that we now have no “let” definition where we store our dictionary; instead the dictionary is passed to the class constructor. So our constructor receives a “Map”, and when we use our “ReceiveValue” method we create a new instance of the “ComputerKeys” which contains the newly created value. I think the type signature really helps us understand what’s going on: type ComputeKeys = class end with member ReceiveValue : value:string -> ComputeKeys new : myDict:Map -> ComputeKeys end This is pretty much the revelation of immutable data structures, “let” definitions become merely short conveniences for values, not memory location that can be updated at a later data if we want to. These new values are all held on the threads stack, if were being pure and fully immutable that we have no memory locations that we can write them to. Okay let’s have a look at how we might use these two classes: /// wraps a Dictionary to provide /// some hashing and printing functions type Store() = // the dictionary that stores the values let myDict = new Dictionary() /// receive a value, hash it store it member x.ReceiveValue (value:string) = myDict.Add(x.ComputeKey value, value) /// computers the hash (a bit naff for now) member x.ComputeKey (value:string) = value.GetHashCode().ToString() /// prints the stored values override x.ToString() = let stringWriter = new StringWriter() for key in myDict.Keys do stringWriter.WriteLine("{0}: {1}", key, myDict.[key]) stringWriter.ToString() let useStore() = let store = new Store() store.ReceiveValue("One") store.ReceiveValue("Two") store.ReceiveValue("Three") printfn "%s" (store.ToString()) The mutable version needs little explanation, it is classical imperative programming, we create an instance of store then add values to our store, and finally we print them out. Now compare this with the immutable version: /// wraps a Map to provide /// some hashing and printing functions type ComputeKeys(myDict:Map) = /// receive a value, hash it, return the new value member x.ReceiveValue (value:string) = new ComputeKeys(myDict.Add(x.ComputeKey value, value)) /// computers the hash (a bit naff for now) member x.ComputeKey (value:string) = value.GetHashCode().ToString() /// prints values in the map override x.ToString() = myDict.Fold (fun key value acc -> Printf.sprintf "%s \r\n%s: %s" acc key value) "" let useComputeKeys() = let keysEmpty = new ComputeKeys(Map.empty) let keysOne = keysEmpty.ReceiveValue("One") let keysTwo = keysOne.ReceiveValue("Two") let keysThree = keysTwo.ReceiveValue("Three") printfn "%s" (keysThree.ToString()) The thing to notice here is how similar using the immutable ComputeKeys class is to using the Store class. We create an instance of the class, we add values to it, and then finally we print it. The only difference being that we need to catch the value returned from RecieveValue and use this value in the next step. Here we’ve used different names for each instance – to illustrate that each let binding is to a different instances, but we don’t need to do that we can reuse the same name to save inventing new names: let useComputeKeysAlt() = let keys = new ComputeKeys(Map.empty) let keys = keys.ReceiveValue("One") let keys = keys.ReceiveValue("Two") let keys = keys.ReceiveValue("Three") printfn "%s" (keys.ToString()) The take away from this is that programming with immutable data structures when we have one thread of execution is not that different to programming with imperative mutable structures, we just have to remember that every time we want to make a change we copy and add rather than update. Wrapping It Up In this inductor post we’ve looked at why mutation is important to classical concurrent programming, and indeed classical imperative programming. Then we looked at immutable data structures and compared they way that they work to mutable data structures. In the next post we’ll dig deeper into immutable data structures, to really get a feel for the programming possibilities they offer. Then in the post after that we’ll look at concurrent programming with immutable data structures and finally get to grips the problem we posed ourselves in the first couple of paragraphs of this post. Patience is a virtue and good things come to those who wait J.

The star schema represents data as a table of facts (measurable values) that are associated with the various dimensions of the fact. Common dimensions include time, geography, organization, product and the like. I'm working with some folks whose facts are a bunch of medical test results, and the dimensions are patient, date, and a facility in which the tests were performed. I got an email with the following situation: "a client who is processing gigs of incoming fact data each day and they use a host of C/C++, Perl, mainframe and other tools for their incoming fact processing and I've seriously considered pushing Python in their organization.". Here are my thoughts on using Python for data warehousing when you've got Gb of data daily. Small Dimensions The pure Python approach only works when your dimension will comfortably fit into memory -- not a terribly big problem with most dimensions. Specifically, it doesn't work well for those dimensions which are so huge that the dimensional model becomes a snowflake instead of a simple star. When dealing with a large number of individuals (public utilities, banks, medical management, etc.) the "customer" (or "patient") dimension gets too big to fit into memory. Special bridge-table techniques must be used. I don't think Python would be perfect for this, since this involves slogging through a lot of data one record at a time. However, Python is considerably faster than PL/SQL. I don't know how it compares with Perl. Any programming language will be faster than any SQL procedure, because there's no RDBMS overhead. For all small dimensions. Load the dimension values from the RDBMS into a dict with a single query. Read all source data records (ideally from a flat file); conform the dimension, tracking changes; write a result record with the dimension FK information to a flat file. Iterate through the dimension dictionary and persist the dimension changes. The details vary with the Slowly Changing Dimension (SCD) rules you're using. The conformance algorithm is is essentially the following: row= Dimension(...) ident= ( row.field, row.field, row.field, ... ) dimension.setdefault( ident, row ) In some cases (like the Django ORM) this is called the get-or-create query. The Dimension Bus For BIG dimensions, I think you still have to implement the "dimension bus" outlined in The Data Warehouse Toolkit. To do this in Python, you should probably design things to look something like the following. For any big dimensions. Use an external sort-merge utility. Seriously. They're way fast for data sets too large to fit into memory. Use CSV format files and the resulting program is very tidy. The outline is as follows: First, sort the source data file into order by the identifying fields of the big dimension (customer number, patient number, whatever). Second, query the big dimension into a data file and sort it into the same order as the source file. (Using the SQL ORDER BY may be slower than an external sort; only measurements can tell which is faster.) Third, do a "match merge" to locate the differences between the dimension and the source. Don't use a utility like diff, it's too slow. This is a simple key matching between two files. The match-merge loop looks something like this. src= sourceFile.next() dim= dimensionFile.next() try: while True: src_key = ( src['field'], src['field'], ... ) dim_key= ( dim['field'], dim['field'], ... ) if src_key < dim_key: # missing some dimension values update_dimension( src ) src= sourceFile.next() elif dim_key < src_key: # extra dimension values dim= dimensionFile.next() else: # src and dim keys match # check non-key attributes for dimension change. src= sourceFile.next() except StopIteration, e: # if source is at end-of-file, that's good, we're done. # if dim is at end of file, all remaining src rows are dimension updates. for src in sourceFile: update_dimension( src ) At the end of this pass, you'll accumulate a file of customer dimension adds and changes, which is then persisted into the actual customer dimension in the database. This pass will also write new source records with the customer FK. You can also handle demographic or bridge tables at this time, too. Fact Loading The first step in DW loading is dimensional conformance. With a little cleverness the above processing can all be done in parallel, hogging a lot of CPU time. To do this in parallel, each conformance algorithm forms part of a large OS-level pipeline. The source file must be reformatted to leave empty columns for each dimension's FK reference. Each conformance process reads in the source file and writes out the same format file with one dimension FK filled in. If all of these conformance algorithms form a simple OS pipe, they all run in parallel. It looks something like this. src2cvs source | conform1 | conform2 | conform3 | load At the end, you use the RDBMS's bulk loader (or write your own in Python, it's easy) to pick the actual fact values and the dimension FK's out of the source records that are fully populated with all dimension FK's and load these into the fact table. I've written conformance processing in Java (which is faster than Python) and had to give up on SQL-based conformance for large dimensions. Instead, we did the above flat-file algorithm to merge large dimensions. The killer isn't the language speed, it's the RDBMS overheads. Once you're out of the database, things blaze. Indeed, products like the syncsort data sort can do portions of the dimension conformance at amazing speeds for large datasets. Hand Wringing "But," the hand-wringers say, "aren't you defeating the value of the RDBMS by working outside it?" The answer is NO. We're not doing incremental, transactional processing here. There aren't multiple update transactions in a warehouse. There are queries and there are bulk loads. Doing the prep-work for a bulk load outside the database is simply more efficient. We don't need locks, rollback segments, memory management, threading, concurrency, ACID rules or anything. We just need to match-merge the large dimension and the incoming facts.

WSDL is a version of XML used to better work with web severs. In this post, we'll learn how to better use it alongside the Java language.

The current Java Persistence API (JPA) standard does not mandate JPA provider to support data type conversions through annotations, not even a with simple boolean field. For readers who are unfamiliar with JPA, what I mean is, to persist a boolean field, JPA expects the database data type to be integer, where value of "1" means true, and value of "0" means false. We simply just can't annotate the boolean field, specifying our own boolean field value, such as "True/False", "T/F", "Yes/No", "Y/N", "-1/0" and then let the JPA provider to convert those boolean field on the fly. As an example, I am expecting JPA will allow me to annotate a boolean field @boolean(trueValue="Yes", falseValue="No") private boolean enabled; For me, this is a very annoying limitation, espeically when you have to deliver applications on an existing database with hundreds of tables. After some research, I decided to create my own Java Annotation Processing Tool (APT) Compile Time Annotation, called @BooleanMagic, with a compile time Java APT preprocessing factory, which will automatically generate additional code to work around the issue. I've also made some changes on the original code: Fixed the bug of Null pointer exception, when JPA return null on the annotated field. Introduce a new properties call ifNull, which allows user to configure what to return if JPA returns null, it expect enum of org.jbpcc.util.jpa.ReturnType, which have values of ReturnType.True, ReturnType.FALSE, and ReturnType.Null,. The default value of ifNull is ReturnType.Null So here is an example, assuming we have model class defined as below: package org.jbpcc.domain.model; import javax.persistence.Entity; import javax.persistence.Id; import org.jbpcc.util.jpa.BooleanMagic; import org.jbpcc.util.jpa.BooleanMagic.ReturnType; @Entitypublic class SomeVO { @Id private Integer id; @BooleanMagic(trueValue = "Yes", falseValue = "No", columnName = "OVERDUED", ifNull = ReturnType.FALSE) private transient Boolean overdued; public Boolean isOverdued() { return overdued; } public void setOverdued(Boolean overdued) { this.overdued = overdued; } } Using Java APT with JPABooleanMagicConverter factory, the code above will be now be converted to: @Entitypublic class SomeVO { @Id private Integer id; private transient Boolean overdued; //--- Lines below are generated by JBPCC BooleanMagicConvertor PROCESSOR //--- START : @Column(name="OVERDUED") private String magicBooleanOverdued; public Boolean isOverdued() { if (this.magicBooleanOverdued == null) return false; return this.magicBooleanOverdued.equals("Yes") ? Boolean.TRUE : Boolean.FALSE; } public Boolean getOverdued() { if (this.magicBooleanOverdued == null) return false; return this.magicBooleanOverdued.equals("Yes") ? Boolean.TRUE : Boolean.FALSE; } public void setOverdued(Boolean trueFlag) { this.magicBooleanOverdued = trueFlag ? "Yes" : "No"; } //--- END //--- GENERATED BY JBPCC BooleanMagicConvertor PROCESSOR } I have put the annotation with it compile time process factory under my open source project - Java Batch Process control center, at http://code.google.com/p/jbpcc, I also posted an article at my blog detailing the usage of the annotation. I hope some readers will find the annotation and the APT preprocessing factory useful. Do share your thoughts and suggestions.

This turns a class name (like LineItem) into a nice string (like "line item") line_item = LineItem.new puts line_item.class.name.underscore.humanize.lowcase #spits out "line item"

# Array#to_ranges # Converts an array of values (which must respond to #succ) to an array of ranges. For example, # [3,4,5,1,6,9,8].to_ranges => [1,3..6,8..9] class Array def to_ranges array = self.compact.uniq.sort ranges = [] if !array.empty? # Initialize the left and right endpoints of the range left, right = self.first, nil array.each do |obj| # If the right endpoint is set and obj is not equal to right's successor # then we need to create a range. if right && obj != right.succ ranges << Range.new(left,right) left = obj end right = obj end ranges << Range.new(left,right) end ranges end end

using System; using System.Collections.Generic; using System.IO.Compression; using System.IO; using System.Collections; namespace Utilities { class Compression { public static byte[] Compress(byte[] data) { MemoryStream ms = new MemoryStream(); DeflateStream ds = new DeflateStream(ms, CompressionMode.Compress); ds.Write(data, 0, data.Length); ds.Flush(); ds.Close(); return ms.ToArray(); } public static byte[] Decompress(byte[] data) { const int BUFFER_SIZE = 256; byte[] tempArray = new byte[BUFFER_SIZE]; List tempList = new List(); int count = 0, length = 0; MemoryStream ms = new MemoryStream(data); DeflateStream ds = new DeflateStream(ms, CompressionMode.Decompress); while ((count = ds.Read(tempArray, 0, BUFFER_SIZE)) > 0) { if (count == BUFFER_SIZE) { tempList.Add(tempArray); tempArray = new byte[BUFFER_SIZE]; } else { byte[] temp = new byte[count]; Array.Copy(tempArray, 0, temp, 0, count); tempList.Add(temp); } length += count; } byte[] retVal = new byte[length]; count = 0; foreach (byte[] temp in tempList) { Array.Copy(temp, 0, retVal, count, temp.Length); count += temp.Length; } return retVal; } } }

SimpleDateFormat sdf = new SimpleDateFormat("dd-MM-yyyy"); Date date = sdf.parse(strDate); Calendar cal = Calendar.getInstance(); cal.setTime(date);

A simple time based cache build around a map store. import java.util.Map; import java.util.WeakHashMap; /** * Simple time-based cache. */ public class SimpleCache { private long maxAge; private Map store; /** * Instanciate a cache with max age of 1 hour and a WeakHashMap as store. * @see java.util.WeakHashMap */ public SimpleCache() { this.maxAge = 1000 * 60 * 60; this.store = new WeakHashMap(); } /** * @param maxAge maximum age of an entry in milliseconds * @param store map to hold entries */ public SimpleCache(long maxAge, Map store) { this.maxAge = maxAge; this.store = store; } /** * Cache an object. * @param key unique identifier to retrieve object * @param value object to cache */ public void put(Object key, Object value) { store.put(key, new Item(value)); } /** * Fetch an object. * @param key unique identifier to retrieve object * @return an object or null in case it isn't stored or it expired */ public Object get(Object key) { Item item = getItem(key); return item == null ? null : item.payload; } /** * Fetch an object or store and return output of callback. * @param key unique identifier to retrieve object * @param block code executed when object not in cache * @return an object */ public synchronized Object get(Object key, Callback block) { Item item = getItem(key); if (item == null) { Object value = block.execute(); item = new Item(value); store.put(key, item); } return item.payload; } /** * Remove an object from cache. * @param key unique identifier to retrieve object */ public void remove(Object key) { store.remove(key); } /** * Get an item, if it expired remove it from cache and return null. * @param key unique identifier to retrieve object * @return an item or null */ private Item getItem(Object key) { Item item = (Item) store.get(key); if (item == null) { return null; } if (System.currentTimeMillis() - item.birth > maxAge) { store.remove(key); return null; } return item; } /** * Value container. */ private static class Item { long birth; Object payload; Item(Object payload) { this.birth = System.currentTimeMillis(); this.payload = payload; } } /** * A visitor interface. */ public static interface Callback { Object execute(); } } And a couple of junit tests: import java.util.HashMap; import junit.framework.TestCase; public class SimpleCacheTest extends TestCase { public void testPutGet () { SimpleCache c = new SimpleCache(Long.MAX_VALUE, new HashMap()); c.put("key1", "value1"); assertEquals("value1", c.get("key1")); c.put("key1", "value1.0"); assertEquals("value1.0", c.get("key1")); c.put("key2", "value2"); assertEquals("value2", c.get("key2")); assertEquals("value1.0", c.get("key1")); } public void testMaxAge () throws InterruptedException { SimpleCache c = new SimpleCache(1000, new HashMap()); c.put("key1", "value1"); assertEquals("value1", c.get("key1")); Thread.sleep(1500); assertNull(c.get("key1")); c.put("key2", "value2"); Thread.sleep(750); c.put("key3", "value3"); Thread.sleep(750); assertNull(c.get("key2")); assertNotNull(c.get("key3")); Thread.sleep(750); assertNull(c.get("key3")); } public void testRemove () { SimpleCache c = new SimpleCache(Long.MAX_VALUE, new HashMap()); c.remove("key"); assertNull(c.get("key")); c.put("key", "value"); assertNotNull(c.get("key")); c.remove("key"); assertNull(c.get("key")); } public void testCallBack () { SimpleCache c = new SimpleCache(Long.MAX_VALUE, new HashMap()); assertEquals("value1", c.get("key1", new SimpleCache.Callback() { public Object execute() { return "value1"; } })); assertEquals("value1", c.get("key1")); // again with a new callback (value) c.get("key1", new SimpleCache.Callback() { public Object execute() { return "value2"; } }); assertEquals("value1", c.get("key1")); } }

class Array def swap!(a,b) self[a], self[b] = self[b], self[a] self end end You can now do stuff like.. [1,2,3,4].swap!(2,3) # = [1,2,4,3] etc.. Many thanks to Sam Stephenson and technoweenie for their suggestions.