Download the source code for this blog entry here: ClientSideCacheInvalidation.zip TL;DR? Please scroll down to the bottom of this article to review the summary. I ran into a problem not long ago where some JSON results from an AJAX call to an ASP.NET MVC JsonResult action were being cached by the browser, quite intentionally by design, but were no longer up-to-date, and without devising a new approach to route manipulation or any of the other fundamental infrastructural designs for the endpoints (because there were too many) our hands were tied. The caching was being done using the ASP.NETOutputCacheAttribute on the action being invoked in the AJAX call, something like this (not really, but this briefly demonstrates caching): [OutputCache(Duration = 300)] public JsonResult GetData() { return Json(new { LastModified = DateTime.Now.ToString() }, JsonRequestBehavior.AllowGet); } @model dynamic @{ ViewBag.Title = "Home"; } Home Reload @section scripts { } Since we were using a generalized approach to output caching (as we should), I knew that any solution to this problem should also be generalized. My first thought was in the mistaken assumption that the default [OutputCache] behavior was to rely on client-side caching, since client-side caching was what I was observing while using Fiddler. (Mind you, in the above sample this is not the case, it is actually server-side, but this is probably because of the amount of data being transferred. I’ll explain after I explain what I did in my false assumption.) Microsoft’s default convention for implementing cache invalidation is to rely on “VaryBy..” semantics, such as varying the route parameters. That is great except that the route and parameters were currently not changing in our implementation. So, my initial proposal was to force the caching to be done on the server instead of on the client, and to invalidate when appropriate. public JsonResult DoSomething() { // // Do something here that has a side-effect // of making the cached data stale // Response.RemoveOutputCacheItem(Url.Action("GetData")); return Json("OK"); } [OutputCache(Duration = 300, Location = OutputCacheLocation.Server)] public JsonResult GetData() { return Json(new { LastModified = DateTime.Now.ToString() }, JsonRequestBehavior.AllowGet); } Invalidate $('#invalidate').on('click', function() { $.post($APPROOT + "Home/DoSomething", null, function(o) { window.location.reload(); }, 'json'); }); While Reload has no effect on the Last modified value, the Invalidate button causes the date to increment. When testing, this actually worked quite well. But concerns were raised about the payload of memory on the server. Personally I think the memory payload in practically any server-side caching is negligible, certainly if it is small enough that it would be transmitted over the wire to a client, so long as it is measured in kilobytes or tens of kilobytes and not megabytes. I think the real concern is that transmission; the point of caching is to make the user experience as smooth and seamless as possible with minimal waiting, so if the user is waiting for a (cached) payload, while it may be much faster than the time taken to recalculate or re-acquire the data, it is still measurably slower than relying on browser cache. The default implementation of OutputCacheAttribute is actually OutputCacheLocation.Any. This indicates that the cached item can be cached on the client, on a proxy server, or on the web server. From my tests, for tiny payloads, the behavior seemed to be caching on the server and no caching on the client; for a large payload from GET requests with querystring parameters seemed to be caching on the client but with an HTTP query with an “If-Modified-Since” header, resulting in a 304 Not Modified on the server (indicating it was also cached on the server but verified by the server that the client’s cache remains valid); and for a large payload from GET requests with all parameters in the path, the behavior seemed to be caching on the client without any validation checking from the client (no HTTP request for an If-Modified-Since check). Now, to be quite honest I am only guessing that these were the distinguishing factors of these behavior observations. Honestly, I saw variations of these behaviors happening all over the place as I tinkered with scenarios, and this was the initial pattern I felt I was observing. At any rate, for our purposes we were currently stuck with relying on “Any” as the location, which in theory would remove server-side caching if the server ran short on RAM (in theory, I don’t know, although the truth can probably be researched, which I don’t have time to get into). The point of all this is, we have client-side caching that we cannot get away from. So, how do you invalidate the client-side cache? Technically, you really can’t. The browser controls the cache bucket and no browsers provide hooks into the cache to invalidate them. But we can get smart about this, and work around the problem, by bypassing the cached data. Cached HTTP results are stored on the basis of varying by the full raw URL on HTTP GET methods, they are cached with an expiration (in the above sample’s case, 300 seconds, or 5 minutes), and are only cached if allowed to be cached in the first place as per the HTTP header directives in the HTTP response. So, to bypass the cache you don’t cache, or you need to know up front how long the cache should remain until it expires—neither of these being acceptable in a dynamic application—or you need to use POST instead of GET, or you need to vary up the URL. Microsoft originally got around the caching problem in ASP.NET 1.x by forcing the “normal” development cycle in the lifecycle of tags that always used the POST method over HTTP. Responses from POST requests are never cached. But POSTing is not clean as it does not follow the semantics of the verbiage if nothing is being sent up and data is only being retrieved. You can also use ETag in the HTTP headers, which isn’t particularly helpful in a dynamic application as it is no different from a URL + expiration policy. To summarize, to control cache: Disable caching from the server in the Response header (Pragma: no-cache) Predict the lifetime of the content and use an expiration policy Use POST not GET Etag Vary the URL (case-sensitive) Given our options, we need to vary up the URL. There a number of approaches to this, but almost all of the approaches involve relying on appending or modifying the querystring with parameters that are expected to be ignored by the server. $.getJSON($APPROOT + "Home/GetData?_="+Date.now(), function (o) { $('#results').text("Last modified: " + o.LastModified); }); In this sample, the URL is appended with “?_=”+Date.now(), resulting in this URL in the GET: /Home/GetData?_=1376170287015 This technique is often referred to as cache-busting. (And if you’re reading this blog article, you’re probably rolling your eyes. “Duh.”) jQuery inherently supports cache-busting, but it does not do it on its own from $.getJSON(), it only does it in $.ajax() when the options parameter includes {cache: false}, unless you invoke $.ajaxSetup({ cache: false }); first to disable all caching. Otherwise, for $.getJSON() you would have to do it manually by appending the URL. (Alright, you can stop rolling your eyes at me now, I’m just trying to be thorough here..) This is not our complete solution. We have a couple problems we still have to solve. First of all, in a complex client codebase, hacking at the URL from application logic might not be the most appropriate approach. Consider if you’re using Backbone.js with routes that synchronize objects to and from the server. It would be inappropriate to modify the routes themselves just for cache invalidation. A more generalized cache invalidation technique needs to be implemented in the XHR-invoking AJAX function itself. The approach in doing this will depend upon your Javascript libraries you are using, but, for example, if jQuery.getJSON() is being used in application code, then jQuery.getJSON itself could perhaps be replaced with an invalidation routine. var gj = $.getJSON; $.getJSON = function (url, data, callback) { url = invalidateCacheIfAppropriate(url); // todo: implement something like this return gj.call(this, url, data, callback); }; This is unconventional and probably a bad example since you’re hacking at a third party library, a better approach might be to wrap the invocation of $.getJSON() with an application function. var getJSONWrapper = function (url, data, callback) { url = invalidateCacheIfAppropriate(url); // todo: implement something like this return $.getJSON(url, data, callback); }; And from this point on, instead of invoking $.getJSON() in application code, you would invoke getJSONWrapper, in this example. The second problem we still need to solve is that the invalidation of cached data that derived from the server needs to be triggered by the server because it is the server, not the client, that knows that client cached data is no longer up-to-date. Depending on the application, the client logic might just know by keeping track of what server endpoints it is touching, but it might not! Besides, a server endpoint might have conditional invalidation triggers; the data might be stale given specific conditions that only the server may know and perhaps only upon some calculation. In other words, invalidation needs to be pushed by the server. One brute force, burdensome, and perhaps a little crazy approach to this might be to use actual “push technology”, formerly “Comet” or “long-polling”, now WebSockets, implemented perhaps with ASP.NET SignalR, where a connection is maintained between the client and the server and the server then has this open socket that can push invalidation flags to the client. We had no need for that level of integration and you probably don’t either, I just wanted to mention it because it might come back as food for thought for a related solution. One scenario I suppose where this might be useful is if another user of the web application has caused the invalidation, in which case the current user will not be in the request/response cycle to acquire the invalidation flag. Otherwise, it is perhaps a reasonable assumption that invalidation is only needed, and only triggered, in the context of a user’s own session. If not, perhaps it is a “good enough” assumption even if it is sometimes not true. The expiration policy can be set low enough that a reasonable compromise can be made between the current user’s changes and changes invoked by other systems or other users. While we may not know what server endpoint might introduce the invalidation of client cache data, we could assume that the invalidation will be triggered by any server endpoint(s), and build invalidation trigger logic on the response of server HTTP responses. To begin implementing some sort of invalidation trigger on the server I could flag invalidations to the client using HTTP header(s). public JsonResult DoSomething() { // // Do something here that has a side-effect // of making the cached data stale // InvalidateCacheItem(Url.Action("GetData")); return Json("OK"); } public void InvalidateCacheItem(string url) { Response.RemoveOutputCacheItem(url); // invalidate on server Response.AddHeader("X-Invalidate-Cache-Item", url); // invalidate on client } [OutputCache(Duration = 300)] public JsonResult GetData() { return Json(new { LastModified = DateTime.Now.ToString() }, JsonRequestBehavior.AllowGet); } At this point, the server is emitting a trigger to the HTTP client that says that “as a result of a recent operation, that other URL, the one for GetData, is no longer valid for your current cache, if you have one”. The header alone can be handled by different client implementations (or proxies) in different ways. I didn’t come across any “standard” HTTP response that does this “officially”, so I’ll come up with a convention here. Now we need to handle this on the client. Before I do anything first of all I need to refactor the existing AJAX functionality on the client so that instead of using $.getJSON, I might use $.ajax or some other flexible XHR handler, and wrap it all in custom functions such as httpGET()/httpPOST() and handleResponse(). var httpGET = function(url, data, callback) { return httpAction(url, data, callback, "GET"); }; var httpPOST = function (url, data, callback) { return httpAction(url, data, callback, "POST"); }; var httpAction = function(url, data, callback, method) { url = cachebust(url); if (typeof(data) === "function") { callback = data; data = null; } $.ajax(url, { data: data, type: "GET", success: function(responsedata, status, xhr) { handleResponse(responsedata, status, xhr, callback); } }); }; var handleResponse = function (data, status, xhr, callback) { handleInvalidationFlags(xhr); callback.call(this, data, status, xhr); }; function handleInvalidationFlags(xhr) { // not yet implemented }; function cachebust(url) { // not yet implemented return url; }; // application logic httpGET($APPROOT + "Home/GetData", function(o) { $('#results').text("Last modified: " + o.LastModified); }); $('#reload').on('click', function() { window.location.reload(); }); $('#invalidate').on('click', function() { httpPOST($APPROOT + "Home/Invalidate", function (o) { window.location.reload(); }); }); At this point we’re not doing anything yet, we’ve just broken up the HTTP/XHR functionality into wrapper functions that we can now modify to manipulate the request and to deal with the invalidation flag in the response. Now all our work will be in handleInvalidationFlags() for capturing that new header we just emitted from the server, and cachebust() for hijacking the URLs of future requests. To deal with the invalidation flag in the response, we need to detect that the header is there, and add the cached item to a cached data set that can be stored locally in the browser with web storage. The best place to put this cached data set is in sessionStorage, which is supported by all current browsers. Putting it in a session cookie (a cookie with no expiration flag) works but is less ideal because it adds to the payload of all HTTP requests. Putting it in localStorage is less ideal because we do want the invalidation flag(s) to go away when the browser session ends, because that’s when the original browser cache will expire anyway. There is one caveat to sessionStorage: if a user opens a new tab or window, the browser will drop the sessionStorage in that new tab or window, but may reuse the browser cache. The only workaround I know of at the moment is to use localStorage (permanently retaining the invalidation flags) or a session cookie. In our case, we used a session cookie. Note also that IIS is case-insensitive on URI paths, but HTTP itself is not, and therefore browser caches will not be. We will need to ignore case when matching URLs with cache invalidation flags. Here is a more or less complete client-side implementation that seems to work in my initial test for this blog entry. function handleInvalidationFlags(xhr) { // capture HTTP header var invalidatedItemsHeader = xhr.getResponseHeader("X-Invalidate-Cache-Item"); if (!invalidatedItemsHeader) return; invalidatedItemsHeader = invalidatedItemsHeader.split(';'); // get invalidation flags from session storage var invalidatedItems = sessionStorage.getItem("invalidated-cache-items"); invalidatedItems = invalidatedItems ? JSON.parse(invalidatedItems) : {}; // update invalidation flags data set for (var i in invalidatedItemsHeader) { invalidatedItems[prepurl(invalidatedItemsHeader[i])] = Date.now(); } // store revised invalidation flags data set back into session storage sessionStorage.setItem("invalidated-cache-items", JSON.stringify(invalidatedItems)); } // since we're using IIS/ASP.NET which ignores case on the path, we need a function to force lower-case on the path function prepurl(u) { return u.split('?')[0].toLowerCase() + (u.indexOf("?") > -1 ? "?" + u.split('?')[1] : ""); } function cachebust(url) { // get invalidation flags from session storage var invalidatedItems = sessionStorage.getItem("invalidated-cache-items"); invalidatedItems = invalidatedItems ? JSON.parse(invalidatedItems) : {}; // if item match, return concatonated URL var invalidated = invalidatedItems[prepurl(url)]; if (invalidated) { return url + (url.indexOf("?") > -1 ? "&" : "?") + "_nocache=" + invalidated; } // no match; return unmodified return url; } Note that the date/time value of when the invalidation occurred is permanently stored as the concatenation value. This allows the data to remain cached, just updated to that point in time. If invalidation occurs again, that concatenation value is revised to the new date/time. Running this now, after invalidation is triggered by the server, the subsequent request of data is appended with a cache-buster querystring field. In Summary, .. .. a consistent approach to client-side cache invalidation triggered by the server might be by following these steps. Use X-Invalidate-Cache-Item as an HTTP response header to flag potentially cached URLs as expired. You might consider using a semicolon-delimited response to list multiple items. (Do not URI-encode the semicolon when using it as a URI list delimiter.) Semicolon is a reserved/invalid character in URI and is a valid delimiter in HTTP headers, so this is valid. Someday, browsers might support this HTTP response header by automatically invalidating browser cache items declared in this header, which would be awesome. In the mean time ... Capture these flags on the client into a data set, and store the data set into session storage in the format: { "http://url.com/route/action": (date_value_of_invalidation_flag), "http://url.com/route/action/2": (date_value_of_invalidation_flag) } { "http://url.com/route/action": (date_value_of_invalidation_flag), "http://url.com/route/action/2": (date_value_of_invalidation_flag) } Hijack all XHR requests so that the URL is appropriately appended with cachebusting querystring parameter if the URL was found in the invalidation flags data set, i.e. http://url.com/route/action becomes something like http://url.com/route/action?_nocache=(date_value_of_invalidation_flag), being sure to hijack only the XHR request and not any logic that generated the URL in the first place. Remember that IIS and ASP.NET by default convention ignore case (“/Route/Action” == “/route/action”) on the path, but the HTTP specification does not and therefore the browser cache bucket will not ignore case. Force all URL checks for invalidation flags to be case-insensitive to the left of the querystring (if there is a querystring, otherwise for the entire URL). Make sure the AJAX requests’ querystring parameters are in consistent order. Changing the sequential order of parameters may be handled the same on the server but will be cached differently on the client. These steps are for “pull”-based XHR-driven invalidation flags being pulled from the server via XHR. For “push”-based invalidation triggered by the server, consider using something like a SignalR channel or hub to maintain an open channel of communication using WebSockets or long polling. Server application logic can then invoke this channel or hub to send an invalidation flag to the client or to all clients. On the client side, an invalidation flag “push” triggered in #7 above, for which #1 and #2 above would no longer apply, can still utilize #3 through #6. You can download the project I used for this blog entry here: ClientSideCacheInvalidation.zip

I was facing a problem where while a person logs out his session is invalidated but the JSESSIONID still remained in the browser. As a result while logging in the Java API used to get the request from the browser along with a JSESSIONID(Just the ID since the session was invalidated) and would create the new session with the same ID. To fix this problem I used the above code so that whenever a user logs out the entire JSESSIONID becomes empty and thus cookie wont exist for that site.Anyone using JAVA can utilize this in their code. @RequestMapping(value = "/logout", method = RequestMethod.POST) public void logout(HttpServletRequest request, HttpServletResponse response) { /* Getting session and then invalidating it */ HttpSession session = request.getSession(false); if (request.isRequestedSessionIdValid() && session != null) { session.invalidate(); } handleLogOutResponse(response); } /** * This method would edit the cookie information and make JSESSIONID empty * while responding to logout. This would further help in order to. This would help * to avoid same cookie ID each time a person logs in * @param response */ private void handleLogOutResponse(HttpServletResponse response) { Cookie[] cookies = request.getCookies(); for (Cookie cookie : cookies) { cookie.setMaxAge(0); cookie.setValue(null); cookie.setPath("/"); response.addCookie(cookie); } }

Since the 70′s, vi and vim are very popular text editors among programmers. 5 years ago, I wrote an article named “100 vim commands every programmer should know” and here is a reworked, updated version. Enjoy! Basics :e filename Open filename for edition :w Save file :q Exit Vim :q! Quit without saving :x Write file (if changes has been made) and exit :sav filename Saves file as filename . Repeats the last change made in normal mode 5. Repeats 5 times the last change made in normal mode Moving in the file k or Up Arrow move the cursor up one line j or Down Arrow move the cursor down one line e move the cursor to the end of the word b move the cursor to the begining of the word 0 move the cursor to the begining of the line G move the cursor to the end of the file gg move the cursor to the begining of the file L move the cursor to the bottom of the screen :59 move cursor to line 59. Replace 59 by the desired line number. 20| move cursor to column 20. % Move cursor to matching parenthesis [[ Jump to function start [{ Jump to block start Cut, copy & paste y Copy the selected text to clipboard p Paste clipboard contents dd Cut current line yy Copy current line y$ Copy to end of line D Cut to end of line Search /word Search word from top to bottom ?word Search word from bottom to top * Search the word under cursor /\cstring Search STRING or string, case insensitive /jo[ha]n Search john or joan /\< the Search the, theatre or then /the\> Search the or breathe /\< the\> Search the /\< ¦.\> Search all words of 4 letters /\/ Search fred but not alfred or frederick /fred\|joe Search fred or joe /\<\d\d\d\d\> Search exactly 4 digits /^\n\{3} Find 3 empty lines :bufdo /searchstr/ Search in all open files bufdo %s/something/somethingelse/g Search something in all the open buffers and replace it with somethingelse Replace :%s/old/new/g Replace all occurences of old by new in file :%s/onward/forward/gi Replace onward by forward, case unsensitive :%s/old/new/gc Replace all occurences with confirmation :2,35s/old/new/g Replace all occurences between lines 2 and 35 :5,$s/old/new/g Replace all occurences from line 5 to EOF :%s/^/hello/g Replace the begining of each line by hello :%s/$/Harry/g Replace the end of each line by Harry :%s/onward/forward/gi Replace onward by forward, case unsensitive :%s/ *$//g Delete all white spaces :g/string/d Delete all lines containing string :v/string/d Delete all lines containing which didn’t contain string :s/Bill/Steve/ Replace the first occurence of Bill by Steve in current line :s/Bill/Steve/g Replace Bill by Steve in current line :%s/Bill/Steve/g Replace Bill by Steve in all the file :%s/^M//g Delete DOS carriage returns (^M) :%s/\r/\r/g Transform DOS carriage returns in returns :%s#<[^>]\+>##g Delete HTML tags but keeps text :%s/^\(.*\)\n\1$/\1/ Delete lines which appears twice Ctrl+a Increment number under the cursor Ctrl+x Decrement number under cursor ggVGg? Change text to Rot13 Case Vu Lowercase line VU Uppercase line g~~ Invert case vEU Switch word to uppercase vE~ Modify word case ggguG Set all text to lowercase gggUG Set all text to uppercase :set ignorecase Ignore case in searches :set smartcase Ignore case in searches excepted if an uppercase letter is used :%s/\<./\u&/g Sets first letter of each word to uppercase :%s/\<./\l&/g Sets first letter of each word to lowercase :%s/.*/\u& Sets first letter of each line to uppercase :%s/.*/\l& Sets first letter of each line to lowercase Read/Write files :1,10 w outfile Saves lines 1 to 10 in outfile :1,10 w >> outfile Appends lines 1 to 10 to outfile :r infile Insert the content of infile :23r infile Insert the content of infile under line 23 File explorer :e . Open integrated file explorer :Sex Split window and open integrated file explorer :Sex! Same as :Sex but split window vertically :browse e Graphical file explorer :ls List buffers :cd .. Move to parent directory :args List files :args *.php Open file list :grep expression *.php Returns a list of .php files contening expression gf Open file name under cursor Interact with Unix :!pwd Execute the pwd unix command, then returns to Vi !!pwd Execute the pwd unix command and insert output in file :sh Temporary returns to Unix $exit Retourns to Vi Alignment :%!fmt Align all lines !}fmt Align all lines at the current position 5!!fmt Align the next 5 lines Tabs/Windows :tabnew Creates a new tab gt Show next tab :tabfirst Show first tab :tablast Show last tab :tabm n(position) Rearrange tabs :tabdo %s/foo/bar/g Execute a command in all tabs :tab ball Puts all open files in tabs :new abc.txt Edit abc.txt in new window Window spliting :e filename Edit filename in current window :split filename Split the window and open filename ctrl-w up arrow Puts cursor in top window ctrl-w ctrl-w Puts cursor in next window ctrl-w_ Maximize current window vertically ctrl-w| Maximize current window horizontally ctrl-w= Gives the same size to all windows 10 ctrl-w+ Add 10 lines to current window :vsplit file Split window vertically :sview file Same as :split in readonly mode :hide Close current window :nly Close all windows, excepted current :b 2 Open #2 in this window Auto-completion Ctrl+n Ctrl+p (in insert mode) Complete word Ctrl+x Ctrl+l Complete line :set dictionary=dict Define dict as a dictionnary Ctrl+x Ctrl+k Complete with dictionnary Marks m {a-z} Marks current position as {a-z} ' {a-z} Move to position {a-z} '' Move to previous position Abbreviations :ab mail [email protected] Define mail as abbreviation of [email protected] Text indent :set autoindent Turn on auto-indent :set smartindent Turn on intelligent auto-indent :set shiftwidth=4 Defines 4 spaces as indent size ctrl-t, ctrl-d Indent/un-indent in insert mode >> Indent << Un-indent =% Indent the code between parenthesis 1GVG= Indent the whole file Syntax highlighting :syntax on Turn on syntax highlighting :syntax off Turn off syntax highlighting :set syntax=perl Force syntax highlighting

In my last post, we saw how to use ASP.NET SignalR and AngularJS together using a "Hello World" type of example. Now that we have some idea of how to use both technologies together, we will take a look at a more advanced scenario in order to make the frameworks work better together. I assume that you have already seen the SignalR Stock ticker sample. If not, download it from GitHub or add the NuGet Package to an existing ASP.NET web application. Make sure that you take some time to run the sample at least once on multiple browsers and have a glance at the code before you proceed any further. I hope you had a look at the code on both the server and the client side of the stock ticker sample. We will not make any modification to the server code and the layout of the page, but we will rewrite the JavaScript part using features of AngularJS. Since there is a lot of client-side code to convert, let’s do it in a two-part series: Creating a custom service to communicate with the hub on the server and using the service in a controller (this post) Performing UI changes on the page, like enabling/disabling buttons, scrolling stock values in a list, and adding animation effect to values in a table and list (next post) Make a copy of the StockTicker.html file and give it a name of your choice. Add two JavaScript files, controller.js and factory.js, to the project. We will add script to these files soon. Modify the script reference section of the page to include the following script files: Let’s start implementing the SignalR part inside a custom service. To keep the controller free from doing anything other than providing data to the view and handling view events, we are creating a custom service to handle the hub communication logic. The service is responsible for: Creating objects needed for communication Configuring client functions to proxy and to respond when a market is opened, closed, or reset, or when a stock value is updated Starting a hub connection Getting the current values of stocks and the current market statuses of the stocks once the connection is started Opening, closing or resetting markets on demand Following is the module and the factory that handles the functionality mentioned above: var app = angular.module('app', []); app.value('$', $); app.factory('stockTickerData', ['$', '$rootScope', function ($, $rootScope) { function stockTickerOperations() { //Objects needed for SignalR var connection; var proxy; //To set values to fields in the controller var setMarketState; var setValues; var updateStocks; //This function will be called by controller to set callback functions var setCallbacks = function (setMarketStateCallback, setValuesCallback, updateStocksCallback) { setMarketState = setMarketStateCallback; setValues = setValuesCallback; updateStocks = updateStocksCallback; }; var initializeClient = function () { //Creating connection and proxy objects connection = $.hubConnection(); proxy = connection.createHubProxy('stockTicker'); configureProxyClientFunctions(); start(); }; var configureProxyClientFunctions = function () { proxy.on('marketOpened', function () { //set market state as open $rootScope.$apply(setMarketState(true)); }); proxy.on('marketClosed', function () { //set market state as closed $rootScope.$apply(setMarketState(false)); }); proxy.on('marketReset', function () { //Reset stock values initializeStockMarket(); }); proxy.on('updateStockPrice', function (stock) { $rootScope.$apply(updateStocks(stock)); }); }; var initializeStockMarket = function () { //Getting values of stocks from the hub and setting it to controllers field proxy.invoke('getAllStocks').done(function (data) { $rootScope.$apply(setValues(data)); }).pipe(function () { //Setting market state to field in controller based on the current state proxy.invoke('getMarketState').done(function (state) { if (state == 'Open') $rootScope.$apply(setMarketState(true)); else $rootScope.$apply(setMarketState(false)); }); }); }; var start = function () { //Starting the connection and initializing market connection.start().pipe(function () { initializeStockMarket(); }); }; var openMarket = function () { proxy.invoke('openMarket'); }; var closeMarket = function () { proxy.invoke('closeMarket'); }; var reset = function () { proxy.invoke('reset'); }; return { initializeClient: initializeClient, openMarket: openMarket, closeMarket: closeMarket, reset: reset, setCallbacks: setCallbacks } }; return stockTickerOperations; } ]); We need a controller to start the work. The controller will have the following components: An array to store the current stock values and a Boolean value to store the current market state Setters to assign values to the fields A function to modify the value of an entry in the stocks array Functions to handle open, close and reset operations when the corresponding button is clicked Set callbacks to the service and ask the service to kick off the communication Following is the code in the controller: var StockTickerCtrl = function ($scope, stockTickerData) { $scope.stocks = []; $scope.marketIsOpen = false; $scope.openMarket = function () { ops.openMarket(); } $scope.closeMarket = function () { ops.closeMarket(); } $scope.reset = function () { ops.reset(); } function assignStocks(stocks) { $scope.stocks = stocks; } function replaceStock(stock) { for (var count = 0; count < $scope.stocks.length; count++) { if ($scope.stocks[count].Symbol == stock.Symbol) { $scope.stocks[count] = stock; } } } function setMarketState(isOpen) { $scope.marketIsOpen = isOpen; } var ops = stockTickerData(); ops.setCallbacks(setMarketState, assignStocks, replaceStock); ops.initializeClient(); } The layout of the HTML page will remain unchanged. But we need to change the way data is rendered on the screen. The stock ticker sample uses a poor man’s template technique to render content in the table and in the scrolling list. Since we are using AngularJS, let’s replace it with expressions. Following is the mark-up on the page in Angular’s style: ASP.NET SignalR Stock Ticker Sample Live Stock Table SymbolPriceOpenHighLowChange% loading... {{stock.Symbol} {{stock.Price | number:2} {{stock.DayOpen | number:2} {{stock.DayHigh | number:2} {{stock.DayLow | number:2} {{stock.Change} {{stock.PercentChange} Live Stock Ticker loading... {{stock.Symbol} {{stock.Price | number:2}{{stock.Change} ({{stock.PercentChange}) Open this page on a browser and the original stock ticker page on another browser window and play with the buttons. You will see that both screens have the same data at any given point in time. The only difference would be the state of the buttons, their color and a scrolling list. We will fix them in the next post. Happy coding!

It’s no news that quicksort is considered one of the most important algorithms of the century and that it is the de facto system sort for many languages, including the Arrays.sort in Java. So, what’s new about quicksort? Well, nothing except that I just now figured out (two damn years after the release of Java 7) that the quicksort implementation of Arrays.sort has been replaced with a variant called dual-pivot quicksort. This thread is not only awesome for this reason but also how humble Jon Bentley and Joshua Bloch really are. What did I do then? Just like everybody else, I wanted to implement it and do some benchmarking against some 10 million integers (random and duplicate). Oddly enough, I found the following results: Random Data Basic quicksort: 1222 ms Three-way quicksort: 1295 ms (seriously!) Dual-pivot quicksort: 1066 ms Duplicate Data Basic quicksort: 378 ms Three-way quicksort: 15 ms Dual-pivot quicksort: six ms Stupid Question One I am afraid that I am missing something in the implementation of the three-way partition. Across several runs against random inputs (of 10 million numbers), I could see that the single pivot always performs better (although the difference is less than 100 milliseconds for 10 million numbers). I understand that the whole purpose of making the three-way quicksort the default quicksort until now is that it does not give 0(n2) performance on duplicate keys, which is very evident when I run it against duplicate inputs. But is it true that, for the sake of handling duplicate data, a small penalty is taken by the three-way quicksort? Or is my implementation bad? Stupid Question Two My dual-pivot implementation (link below) does not handle duplicates well. It takes forever (0(n2)) to execute. Is there a good way to avoid this? Referring to the Arrays.sort implementation, I figured out that ascending sequences and duplicates are eliminated well before the actual sorting is done. So, as a dirty fix, if the pivots are equal I fast-forward the lowerIndex until it is different than pivot2. Is this a fair implementation? else if (pivot1==pivot2){ while (pivot1==pivot2 && lowIndex=j) break; exchange(input, i, j); } exchange(input, pivotIndex, j); return j; } } Three-way package basics.sorting.quick; import static basics.shuffle.KnuthShuffle.shuffle; import static basics.sorting.utils.SortUtils.exchange; import static basics.sorting.utils.SortUtils.less; public class QuickSort3Way { public void sort (int[] input){ //input=shuffle(input); sort (input, 0, input.length-1); } public void sort(int[] input, int lowIndex, int highIndex) { if (highIndex<=lowIndex) return; int lt=lowIndex; int gt=highIndex; int i=lowIndex+1; int pivotIndex=lowIndex; int pivotValue=input[pivotIndex]; while (i<=gt){ if (less(input[i],pivotValue)){ exchange(input, i++, lt++); } else if (less (pivotValue, input[i])){ exchange(input, i, gt--); } else{ i++; } } sort (input, lowIndex, lt-1); sort (input, gt+1, highIndex); } } Dual-pivot package basics.sorting.quick; import static basics.shuffle.KnuthShuffle.shuffle; import static basics.sorting.utils.SortUtils.exchange; import static basics.sorting.utils.SortUtils.less; public class QuickSortDualPivot { public void sort (int[] input){ //input=shuffle(input); sort (input, 0, input.length-1); } private void sort(int[] input, int lowIndex, int highIndex) { if (highIndex<=lowIndex) return; int pivot1=input[lowIndex]; int pivot2=input[highIndex]; if (pivot1>pivot2){ exchange(input, lowIndex, highIndex); pivot1=input[lowIndex]; pivot2=input[highIndex]; //sort(input, lowIndex, highIndex); } else if (pivot1==pivot2){ while (pivot1==pivot2 && lowIndex

i went to the local capital kanban meetup yesterday evening. it was a bunch of project managers discussing kanban and waste in it. seemed completely out of my comfort zone and a way to meet new people in tech here in town so i attended. it turned out to be really cool and way more interesting than my expectations were. i wanted to mention some of those here, specifically some of the it wastes that were mentioned i see all the time, the insights i got from the paper airplane factory game, and some after meeting talk that changed my perspective on what i perceive as problems in our industry with good software east of california (hah, trick question, there is no good software done east of california…). it wastes andrea ross did a presentation about waste in it. kanban, the production process used by toyota then turned into a project management and software development, has 7 or 8 forms of what it calls waste. these are primarely in the factory line production process, so you have to draw your own metaphors and similes, and that’s what andrea’s presentation extrapolated on. from a high level, these are: defects / rework overproduction waiting non-standard over processing transportation / logistics intellect motion excess inventory her slides that have bullet point examples for each one are pretty self-explanatory. what was interesting to me was the sheer volume of bullet points i see all the time, together, in the same projects i work on. some can’t be avoided, nature of the business and all that. still, it was pretty eye opening to see that a traditional factory production process has identified these items as the core waste items, and software development has plenty of them with just about the same meanings. i won’t cover them in detail here as her slides do. kanban, bottlenecks, and waste the concept of kanban is to quickly identify bottlenecks in the existing production process, and iterate to improve the process to fix them. notice i said “existing” and “process”. the existing part is where kanban has been easier to market than say six sigma which is bought into wholesale, hence why it’s easier to be a six sigma consultant than a kanban one. kanban you basically overlay on top of what you have and it surfaces the problems your existing process has pretty clearly. meaning, if you see a bunch of cards on a kanban board that are in the “analysis” column, and very few in the rest, it’s pretty clear where the bottle neck is located. now the “process” part is analogous to the production line; in this case all that goes into making software from the traditional waterfall perspective: design, development, deployment. however, the key here is you aren’t fixing the “bottleneck”, but rather the process itself. that is what i learned through our paper airplane factory exercise quite clearly. there are a series of games like this that can be modified, but the point is they help teach the bottleneck vs. process modification process extremely clearly. the key takeaway for me was fixing the bottleneck, like the 5 developers + 1 manager in a war room during a troubling moment during a software project, is actually a form of waste. yes, it’s great teams rise up to tackle these problems in the moment. however, it’s important to note that it’s the project manager’s job to both recognize this as waste and fix the actual process problem. i’ll explain this below. note: if you’re concerned about spoilers, please be aware of 2 things. first, there are more than just the airplane factory game that you can find online. second, if you do read the following section and later participate in the exercise, please either let the teacher/presenter know, or try not to modify the process too much to allow others to learn. airplane factory the game is like so (abridged version, you can find the full instructions here ): divide your people into 4 groups, each sitting adjacent to each other. circle or semi-circular seating arrangements works best to encourage intentional bottleneck adjustment engagement. cut up the airplane folding instructions alone the designated lines. give the first part of the instructions to the first group in the line. give the second part of the instructions to the second group, and repeat on down the line. some people may not necessarily have designated jobs beyond passing papers, etc. this is intentional to illustrate the intellect waste of not using human ip… and also to note how they’ll often become efficient passers, helpers, or even qa. ensure the group/person who’s last in line is aware of how far the plane must fly as a metric of defining a successful plane. setup a 5 minute timer, start it, and yell “go!” after 5 minutes, identify how many successfully flown airplanes were made as well as how much waste (crumpled papers, non-flying planes, etc) were created. that’s the round 1 score. iterate. the iterate step is where you reflect on what just happened and attempt to modify the process. i’ll go over how ours went down so you get an idea. round 1 line setup : we had 8 people in our line. round 1, we had 1 lady do the half fold, the 2nd guy do the additional 2 folds + paper sides cut, andrea and i pass the paper to our left, another lady handle the 1st wing folds, and a gentleman at the end to make the wings and throw it. our last person was a lady who handled qa and scoring. process : very quickly we had a bottleneck with the lady at my stations left. the instructions weren’t very clear and she struggled to learn how to do the first one. both andrea and i quickly went to help; andrea attempting to do it with her, me taking a picture of the instructions with my iphone, and attempting to duplicate at my desk while ensuring i kept passing the planes to my left into an ever growing pile. once i figured it out (i had actually built the exact same plane last week for my daughters birthday present which has an electronic propeller you attach), i told the ladies to ignore the “requirements” as they were crap and i walked them through how to successfully complete their step. i then quickly returned to my desk which had a pile of unmoved inventory. the only person who didn’t struggle with their assembly was the 1st in the line who had to fold paper in half. i believe we ended up with 2 planes and 1 waste. takeaway : our teacher quickly pointed how we went “downstream” to fix the production line process. this is a reactionary, and completely normal mode, to fix production line problems. it’s also wrong. you’re supposed to identify the upstream problem causing it and fix that. additionally, we didn’t stop the line to ensure we fixed this problem first before continuing, also wrong. car companies like toyota do this via a chain that’s pulled to stop the line so they can ensure the problem is resolved. sometimes they even take part of their line off the main line to ensure things keep moving. as a side note, apparently gm used to keep going. door doesn’t fit right? keep going; jam the mofo on there. they’d end up with a lot full of cars pretty quickly… even if they were low quality. ford was similar, but they’d ensure the cars were actually sold first before they sold them, thus not resulting in lots full of inventory they couldn’t sell like gm… even if the quality of pre-sold cars was still low. we also noted various other problems such as no training in each plane’s building instructions, no one stopping if the station after them go overwhelmed with their ever growing stack, and sometimes idle resources (people with not much to do). round 2 improvements : first, the teacher actually implemented designated stack areas with a piece of paper on each station, and then wrote a number on the paper; this was the max amount of completed planes you could place for the next station to build so you didn’t exacerbate a bottleneck. second, i become a designated passer to my left while my partner moved to the left station to have a 2 women team doing the complicated folding. process : my job was pretty easy; pass, and ensure i don’t bottleneck it. every single station was faster since they had practiced their portion. the guy to my right had actually done his first 3 paper cuts wrong in round 1 which caused confusion to my left station, but had it down pat in round 2. the last station still experimented with various angles of folding to see how far the plane could actually fly. we actually had the last women in the line, qa, send a messed up plane back through the line as an unfolded piece of paper because it didn’t fly right; w00t, less waste! a bottleneck, again, formed to my left, but the girls found a way to divide up the 2 step process between them to be more efficient. as our 5 minutes progressed, they got faster and eventually started making progress on the backlog. we made 5 planes, 1 waste. takeaway : we built 4 planes with 1 waste. the first person, as usual, was too fast. the guy to my right had an inefficient process because he’d have to fold, pick up the scissors, cut, then put ‘em down again. we had all abandoned our airplane instructions by this point. round 3 improvements : everything else was fine, so we decided to give me the cutting job and the guy to my right would just fold. the girls to my left on-the-fly modification was good and we kept it. process : my first 3 were slow, but once i practiced, i was uber fast and we were humming. the girls to my left were killing it. i managed to keep my right stack always below or at 2 in the pile. very quickly it became apparent the 1st person was too fast; she was constantly folding and then waiting before she was allowed to make another. we made 8 airplanes, no waste. takeaway : those of with spouses were already getting texted like mad to leave, but we all wanted to see round 3 succeed better than 2, and see if we made the process perfect. we didn’t; it went the complete opposite direction to the front of the line needing minor modifications. overall, though, our output increased a lot, our waste went down, and it was very clear that the adjustments we made + the teachers maximum stack amounts were working well. my overall takeaways i went to this meeting to both meet new people in town to network with as well as to get out of my comfort zone. i find when i do the latter, i learn a lot and sometimes get a new perspective. it gave me a new appreciation for project managers who have not just 1, but 5 projects they have to manage to make an attempt to do this on. this also assumes they get enough time to really learn about each teams issues, where those bottlenecks are, and what the best ways are to address them. not by just fixing the bottlenecks, but by fixing the process itself, ensuring stop guards are in place not as many items/cards in a column, etc. it also made me intimately aware of how i, as a consultant, immediately want to fix the bottleneck, and have learned ways (such as the war room) to solve them… when in reality, it’s a pm issue for a greater process problem. the other thing that makes it more complex is the whole “all things being equal”. for example, a kanban board a pm would use on the whole process vs. just the kanban board my software team would use. if my team fails to do tdd and ends up with a variety of bugs in the system because we’re forced to develop quickly and produce bad work, this show up on hers as us being the bottleneck. without time to talk to us and really empower us to change our process, nothing will change. i see this time and time again. the excuses, which are sometimes valid, range from “the software’s good enough even with the bugs”, or “tdd is too much work for not enough value” or “we can’t write a test suite for a huge mess that isn’t even testable”. …and that’s just a small portion of what i’ve seen gone wrong. if you’ve ever worked for a design agency, or even a large firm that has a huge new client, it’s very apparent many teams have a hard time getting sign off from clients which causes a bottleneck in the analysis column on the kanban board because the items either pile up, or priority constantly shifts… yet they never actually make it out of their column. a pm there who works with the government offered his strategies for dealing with the strange qa cycles government agencies will have where it goes into a black hole for 6 weeks thus really screwing up his kanban metrics. overall, it was neat to be in a room with people who were geeking out on improving process. you see a lot of software developers get bored with programming or frustrated with how their lack of process is going, so they read up on xp and agile. when you look at what these pm’s deal with, it makes you feel like just a small part in a larger overall process. more importantly, my preconceptions about leadership being the problem 99% of my problem projects really had a wrench thrown in. i was bitching about it to one of the pm’s, and quickly explained, in great detail, why some big companies which don’t have a hard line metric such as money to predict performance will often use lean methodologies since “ensuring customer satisfaction” is hard to measure depending on your business, and requires a more exploratory way of doing business. that said, it was great to hear that the common problems i experience in software dev with solutions were the same, just 1 of many that pm’s have to deal with. i highly encourage software developers to partake in one of these exercises, even if you do scrum vs. kanban. really eye opening stuff.

One of the five essential attributes of cloud computing (see The 5-3-2 Principle of Cloud Computing) is resource pooling, which is an important differentiator separating the thought process of traditional IT from that of a service-based, cloud computing approach. Resource pooling in the context of cloud computing and from a service provider’s viewpoint denotes a set of strategies and a methodical way of managing resources. For a user, resource pooling institutes an abstraction for presenting and consuming resources in a consistent and transparent fashion. This article presents these key concepts derived from resource pooling: Resource Pools Virtualization in the Context of Cloud Computing Standardization, Automation, and Optimization Fabric Cloud Closing Thoughts Resource Pools Ultimately, data center resources can be logically placed into three categories. They are: compute, networks, and storage. For many, this grouping may appear trivial. It is, however, a foundation upon which some cloud computing methodologies are developed, products designed, and solutions formulated. Compute This is a collection of all CPU capabilities. Essentially all data center servers, either for supporting or actually running a workload, are all part of this compute group. Compute pool represents the total capacity for executing code and running instances. The process to construct a compute pool is to first inventory all servers and identify virtualization candidates followed by implementing server virtualization. It is never too early to introduce a system management solution to facilitate the processes, which in my view is a strategic investment and a critical component for all cloud initiatives. Networks The physical and logical artifacts putting in place to connect resources, segment, and isolate resources from layer three and below, etc., are gathered in the network pool. Networking enables resources becoming visible and hence possibly manageable. In the age of instant gratification, networks and mobility are redefining the security and system administration boundaries, and play a direct and impactful role in user productivity and customer satisfaction. Networking in cloud computing is more than just remote access, but empowerment for a user to self-serve and consume resources anytime, anywhere, with any device. BYOD and consumerization of IT are various expressions of these concepts. Storage This has long been a very specialized and sometimes mysterious part of IT. An enterprise storage solution is frequently characterized as a high-cost item with a significant financial and contractual commitment, specialized hardware, proprietary API and software, a dependency on direct vendor support, etc. In cloud computing, storage has become even more noticeable since the ability to grow and shrink based on demands, i.e. elasticity, demands an enterprise-level, massive, reliable, and resilient storage solution at a global scale. While enterprise IT is consolidating resources and transforming the existing establishment into a cloud computing environment, how to leverage existing storage devices from various vendors and integrate them with the next generation storage solutions is among the highest priorities for modernizing a data center. Virtualization in the Context of Cloud Computing In the last decade, virtualization has proved its value and accelerated the realization of cloud computing. Then, virtualization was mainly server virtualization, which in an over-simplified statement means hosting multiple server instances with the same hardware while each instance runs transparently and in insolation, as if each consumes the entire hardware and is the only instance running. Much of the customer expectations, business needs, and methodologies has since evolved. Now, we should validate virtualization in the context of cloud computing to fully address the innovations rapidly changing how IT conducts business and delivers services. As discussed below, in the context of cloud computing, consumable resources are delivered in some virtualized form. Various virtualization layers collectively construct and form the so-called fabric. Server Virtualization The concept of server virtualization remains: running multiple server instances with the same hardware while each instance runs transparently and in isolation, as if each instance is the only instance running and consuming the entire server hardware. In addition to virtualizing and consolidating servers, server virtualization also signifies the practices of standardizing server deployment switching away from physical boxes to VMs. Server virtualization is for packaging, delivering, and consuming a compute pool. There are a few important considerations of virtualizing servers. IT needs the ability to identify and manage bare metal such that the entire resource life-cycle management from commencing to decommissioning can be standardized and automated. To fundamentally reduce the support and training cost while increasing productivity, a consistent platform with tools applicable across physical, virtual, on-premises, and off-premises deployments is essential. The last thing IT wants is one set of tools for physical resources and another for those virtualized, one set of tools for on-premises deployment and another for those deployed to a service provider, and one set of tools for development and another for deploying applications. The requirement is one methodology for all, one skill set for all, and one set of tools for all. This advantage is obvious when developing applications and deploying Windows Server 2012 R2 on premises or off premises to Windows Azure. The Active Directory security model can work across sites, System Center can manage resources deployed off premises to Windows Azure, and Visual Studio can publish applications across platforms. Windows infrastructure architecture, security, and deployment models are all directly applicable. Network Virtualization The similar idea of server virtualization applies here. Network virtualization is the ability to run multiple networks on the same network device while each network runs transparently and in isolation, as if each network is the only network running and consuming the entire network hardware. Conceptually, since each network instance is running in isolation, one tenant’s 192.168.x network is not aware of another tenant’s identical192.168.x network running with the same network device. Network virtualization provides the translation between physical network characteristics and the representation of and a resource identity in a virtualized network. Consequently, above the network virtualization layer, various tenants while running in isolation can have identical network configurations. A great example of network virtualization is Windows Azure virtual networking. At any given time, there can be multiple Windows Azure subscribers all allocating the same 192.168.x address space with an identical subnet scheme (192.168.1.x/16) for deploying VMs. Those VMs belonging to one subscriber will however not be aware of or visible to those deployed by others, despite the fact that the network configuration, IP scheme, and IP address assignments may all be identical. Network virtualization in Windows Azure isolates on subscriber from the others such that each subscriber operates as if the subscription is the only one employing a 192.168.x address space. Storage Virtualization I believe this is where the next wave of drastic cost reduction of IT post-server virtualization happens. Historically, storage has been a high cost item in any IT budget in each and every aspects including hardware, software, staffing, maintenance, SLA, etc. Since the introduction of Windows Server 2012, there is a clear direction where storage virtualization is built into OS and becoming a commodity. New capabilities like Storage Pool, Hyper-V over SMB, Scale-Out Fire Share, etc., are now part of Windows Server OS and are making storage virtualization part of server administration routines and easily manageable with tools and utilities like PowerShell, which is familiar to many IT professionals. The concept of storage virtualization remains consistent with the idea of logically separating a computing object from its hardware, in this case the storage capacity. Storage virtualization is the ability to integrate multiple and heterogeneous storage devices, aggregate the storage capacities, and present/manage as one logical storage device with a continuous storage space. JBOD is a technology to realize this concept. Standardization, Automation and Optimization Each of the three resource pools has an abstraction to logically present itself with characteristics and work patterns. A compute pool is a collection of physical (virtualization and infrastructure) hosts and VMs. A virtualization host hosts VMs that run workloads deployed by service owners and consumed by authorized users. A network pool encompasses network resources including physical devices, logical switches, address spaces, and site configurations. Network virtualization as enabled/defined in configurations can identify and translate a logical/virtual IP address into a physical one, such that tenants with the same network hardware can implement an identical network scheme without a concern. A storage pool is based on storage virtualization which is a concept of presenting an aggregated storage capacity as one continuous storage space as if provided from one logical storage device. In other words, the three resource pools are wrapped with server virtualization, network virtualization, and storage virtualization, respectively. Each virtualization presents a set of methodologies on which work patterns are derived and common practices are developed. These virtualization layers provides opportunities to standardize, automate, and optimize deployments and considerably facilitates the adoption of cloud computing. Standardization Virtualizing resources decouples the dependency between instances and the underlying hardware. This offers an opportunity to simplify and standardize the logical representation of a resource. For instance, a VM is defined and deployed with a VM template that provides a level of consistency with a standardized configuration. Automation Once VM characteristics are identified and standardized, we can now generate an instance by providing only instance-based information or information that depends on run-time, such as the VM machine name, which must be validated at run-time to prevent duplicated names. This requirement for providing only minimal information at deployment can be significantly simplify and streamline operations for automation. And with automation, resources can then be deployed, instantiated, relocated, taken off-line, brought back online, or removed rapidly and automatically based on set criteria. Standardization and automation are essential mechanisms so that workload can be scaled on demand, i.e., become elastic. Optimization Standardization provides a set of common criteria. Automation executes operations based on set criteria with volumes, consistency, and expediency. With standardization and automation, instances can be instantiated with consistency, efficiency, and predictability. In other words, resources can be operated in bulk with consistency and predictability. The next logical step is then to optimize the usage based on SLA. The presented progression is what resource pooling and virtualizations can provide and facilitate. These methodologies are now built into products and solutions. Windows Server 2012 R2 and System Center 2012 and later integrate server virtualization, network virtualization, and storage virtualization into one consistent solution platform with standardization, automation, and optimization for building and managing clouds. Fabric This is a significant abstraction in cloud computing. Fabric implies accessibility and discoverability, and denotes the ability to discover, identify, and manage a resource. Conceptually, fabric is an umbrella term encompassing all the underlying infrastructure supporting a cloud computing environment. At the same time, a fabric controller represents the system management solution which manages, i.e. owns, fabric. In cloud architecture, fabric consists of the three resource pools: compute, networks, and storage. Compute provides the computing capabilities, executes code, and runs instances. Networks glues the resources based on requirements. Storage is where VMs, configurations, data, and resources are kept. Fabric shields the physical complexities of the three resource pools presented with server virtualization, network virtualization, and storage virtualization. All operations are eventually directed by the fabric controller of a data center. Above fabric, there are logical views of consumable resources including VMs, virtual networks, and logical storage drives. By deploying VMs, configuring virtual networks, or acquiring storage, a user consumes resources. Under fabric, there are virtualization and infrastructure hosts, Active Directory, DNS, clusters, load balancers, address pools, network sites, library shares, storage arrays, topology, racks, cables, etc., all under the fabric controller’s command to collectively present and support fabric. For a service provider, building a cloud computing environment is essentially establishing a fabric controller and constructing fabric. Namely, instituting a comprehensive management solution, building the three resource pools, and integrating server virtualization, network virtualization, and storage virtualization to form fabric. From a user’s point of view, how and where a resource is physically provided is not a concern, but the accessibility, readiness, scalability, and fulfillment of SLA are. Cloud This is a well-defined term and we should not be confused with it. (see NIST SP 800-145 and the 5-3-2 Principle of Cloud Computing) We need to be very clear on: what a cloud must exhibit (the five essential attributes), how to consume it (with SaaS, PaaS, or IaaS), and the model a service is deployed in (like private cloud, public cloud, and hybrid cloud). Cloud is a concept, a state, a set of capabilities such that a business can be delivered as a service, i.e. available on demand. The architecture of a cloud computing environment is presented with three resource pools: compute, networks, and storage. Each is an abstraction provided by a virtualization layer. Server virtualization presents a compute pool with VMs that supply the computing, i.e. CPUs, and power to execute code and run instances. Network virtualization offers a network pool and is the mechanism that allows multiple tenants with identical network configurations on the same virtualization host while connecting, segmenting, isolating network traffic with virtual NICs, logical switches, address space, network sites, IP pools, etc. Storage virtualization provides a logical storage device with the capacity to appear continuous and aggregated with a pool of storage devices behind the scene. The three resource pools together constitute the fabric (of a cloud) while the three virtualization layers collectively form the abstraction, such that while the underlying physical infrastructure may be intricate, the user experience above fabric remains logical and consistent. Deploying a VM, configuring a virtual network, or acquiring storage is transparent with virtualization regardless of where the VM actually resides, how the virtual network is physically wired, or what devices in the aggregate the requested storage is provided with. Closing Thoughts Cloud is a very consumer-focused approach. It is about a customer’s ability and control based on SLA in getting resources when needed and with scale, and equally important releasing resources when no longer required. It is not about products and technologies. It is about servicing, consuming, and strengthening the bottom line.

Earlier in the week I wrote a blog post showing how to calculate the eigenvector centrality of an adjacency matrix using JBLAS and the next step was to work out the eigenvector centrality of a neo4j sub graph. There were 3 steps involved in doing this: Export the neo4j sub graph as an adjacency matrix Run JBLAS over it to get eigenvector centrality scores for each node Write those scores back into neo4j I decided to make use of the Paul Revere data set from Kieran Healy’s blog post which consists of people and groups that they had membership of. The script to import the data is on my fork of the revere repository. Having imported the data the next step was to write a cypher query which would give me the people in anadjacency matrix with the number in each column/row intersection showing how many common groups that pair of people had. I thought it’d be easier to build this query incrementally so I started out writing a query which would return one row of the adjacency matrix: MATCH p1:Person, p2:Person WHERE p1.name = "Paul Revere" WITH p1, p2 MATCH p = p1-[?:MEMBER_OF]->()<-[?:MEMBER_OF]-p2 WITH p1.name AS p1, p2.name AS p2, COUNT(p) AS links ORDER BY p2 RETURN p1, COLLECT(links) AS row Here we start with Paul Revere and then find the relationships between him and every other person by way of a common group membership. We use an optional relationship since we need to include a value in each column/row of our adjacency matrix we need to return a 0 value for anyone he doesn’t intersect with. If we run that query we get back the following: +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | p1 | row | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | "Paul Revere" | [2,1,1,1,1,1,1,1,1,1,1,1,1,1,2,3,1,1,1,1,1,1,3,3,1,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,3,2,1,1,2,1,2,1,1,1,1,1,0,1,1,1,1,3,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,1,2,1,1,1,1,1,1,2,1,3,1,3,2,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,1,0,1,0,1,1,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,4,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,3,1,1,1,1,1,1,2,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,3,1,1,2,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,3,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,3,1,1,2,1,1,1,1,1,1,1,1,3,1,1,1,1,3,1,1,1,1,0,1,2,1,1,1,1,1,1,1] | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ As it turns outs we’ve only got to remove the WHERE clause and order everybody and we’ve get the adjacency matrix for everyone: MATCH p1:Person, p2:Person WITH p1, p2 MATCH p = p1-[?:MEMBER_OF]->()<-[?:MEMBER_OF]-p2 WITH p1.name AS p1, p2.name AS p2, COUNT(p) AS links ORDER BY p2 RETURN p1, COLLECT(links) AS row ORDER BY p1 +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | p1 | row | +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | "Abiel Ruddock" | [0,1,1,1,0,1,0,1,0,0,1,1,1,0,1,2,0,1,0,1,1,1,2,2,1,0,0,1,1,0,1,1,1,1,1,0,0,0,0,1,1,0,0,2,2,0,0,1,1,2,1,1,1,0,1,0,1,1,0,0,2,1,0,0,0,0,1,0,0,1,1,0,0,0,0,0,0,0,1,1,0,1,1,1,1,1,1,1,1,1,0,2,1,2,1,0,0,0,0,1,1,0,1,0,0,1,0,2,0,0,1,0,0,0,1,0,0,2,0,1,0,1,1,1,0,0,1,1,0,0,0,0,0,0,2,0,0,0,0,0,0,0,1,0,1,1,0,1,1,1,2,0,0,1,1,0,0,2,0,1,2,1,1,0,0,0,0,0,0,0,1,1,1,0,0,0,0,0,1,2,1,0,1,1,1,1,1,0,0,1,1,0,0,0,0,1,0,1,1,0,0,1,0,0,2,1,0,0,1,1,1,1,0,1,0,0,0,1,0,1,0,1,1,0,0,1,0,1,0,1,0,0,1,0,2,1,1,0,0,2,0,1,0,0,0,0,1,0,1,0,1,0,1,0] | | "Abraham Hunt" | [1,0,1,1,0,1,0,0,0,0,0,1,0,0,0,1,0,1,0,1,1,0,1,1,0,0,0,1,1,0,1,0,0,1,0,0,0,0,0,1,0,0,0,1,1,0,0,0,1,1,1,1,1,0,0,0,1,0,0,0,1,1,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,1,0,0,1,1,0,1,0,1,1,1,1,0,0,0,0,1,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,1,0,1,0,0,1,1,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,1,0,0,0,1,0,1,0,0,0,1,0,0,1,0,1,1,0,1,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,1,1,1,0,1,0,1,0,1,0,0,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,1,0,0,0,1,1,1,1,0,1,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,0,1,0,0,0,0,0,0,0,0,1,0,1,0,1,0] | ... +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 254 rows 9897 ms The next step was to wire up the query results with the JBLAS code that I wrote in the previous post. I ended up with the following: public class Neo4jAdjacencyMatrixSpike { public static void main(String[] args) throws SQLException { ClientResponse response = client() .resource("http://localhost:7474/db/data/cypher") .entity(queryAsJson(), MediaType.APPLICATION_JSON) .accept(MediaType.APPLICATION_JSON) .post(ClientResponse.class); JsonNode result = response.getEntity(JsonNode.class); ArrayNode rows = (ArrayNode) result.get("data"); List principalEigenvector = JBLASSpike.getPrincipalEigenvector(new DoubleMatrix(asMatrix(rows))); List people = asPeople(rows); updatePeopleWithEigenvector(people, principalEigenvector); System.out.println(sort(people).take(10)); } private static double[][] asMatrix(ArrayNode rows) { double[][] matrix = new double[rows.size()][254]; int rowCount = 0; for (JsonNode row : rows) { ArrayNode matrixRow = (ArrayNode) row.get(2); double[] rowInMatrix = new double[254]; matrix[rowCount] = rowInMatrix; int columnCount = 0; for (JsonNode jsonNode : matrixRow) { matrix[rowCount][columnCount] = jsonNode.asInt(); columnCount++; } rowCount++; } return matrix; } // rest cut for brevity } Here we are taking the query and then converting it into an array of arrays before passing it to our JBLAS code to calculate the principal eigenvector. We then return the top 10 people: Person{name='William Cooper', eigenvector=0.172604992239612, nodeId=68}, Person{name='Nathaniel Barber', eigenvector=0.17260499223961198, nodeId=18}, Person{name='John Hoffins', eigenvector=0.17260499223961195, nodeId=118}, Person{name='Paul Revere', eigenvector=0.17171142003936804, nodeId=207}, Person{name='Caleb Davis', eigenvector=0.16383970722169897, nodeId=71}, Person{name='Caleb Hopkins', eigenvector=0.16383970722169897, nodeId=121}, Person{name='Henry Bass', eigenvector=0.16383970722169897, nodeId=21}, Person{name='Thomas Chase', eigenvector=0.16383970722169897, nodeId=54}, Person{name='William Greenleaf', eigenvector=0.16383970722169897, nodeId=104}, Person{name='Edward Proctor', eigenvector=0.15600043886738055, nodeId=201} I get back the same 10 people as Kieran Healy although they have different eigenvector values. As far as I understand the absolute value doesn’t matter, what’s more important is the relative score to other people so I think we’re ok. The final step was to write these eigenvector values back into neo4j which we can do with the following code: private static void updateNeo4jWithEigenvectors(List people) { for (Person person : people) { ObjectNode request = JsonNodeFactory.instance.objectNode(); request.put("query", "START p = node({nodeId}) SET p.eigenvectorCentrality={value}"); ObjectNode params = JsonNodeFactory.instance.objectNode(); params.put("nodeId", person.nodeId); params.put("value", person.eigenvector); request.put("params", params); client() .resource("http://localhost:7474/db/data/cypher") .entity(request, MediaType.APPLICATION_JSON) .accept(MediaType.APPLICATION_JSON) .post(ClientResponse.class); } } Now we might use that eigenvector centrality value in other queries, such as one to show who the most central/potentially influential people are in each group: MATCH g:Group<-[:MEMBER_OF]-p WITH g.name AS group, p.name AS personName, p.eigenvectorCentrality as eigen ORDER BY eigen DESC WITH group, COLLECT(personName) AS people RETURN group, HEAD(people) + [HEAD(TAIL(people))] + [HEAD(TAIL(TAIL(people)))] AS mostCentral +--------------------------------------------------------------------------+ | group | mostCentral | +--------------------------------------------------------------------------+ | "StAndrewsLodge" | ["Paul Revere","Joseph Warren","Thomas Urann"] | | "BostonCommittee" | ["William Cooper","Nathaniel Barber","John Hoffins"] | | "LoyalNine" | ["Caleb Hopkins","William Greenleaf","Caleb Davis"] | | "LondonEnemies" | ["William Cooper","Nathaniel Barber","John Hoffins"] | | "LongRoomClub" | ["Paul Revere","John Hancock","Benjamin Clarke"] | | "NorthCaucus" | ["William Cooper","Nathaniel Barber","John Hoffins"] | | "TeaParty" | ["William Cooper","Nathaniel Barber","John Hoffins"] | +--------------------------------------------------------------------------+ 7 rows 280 ms Our top ten feature frequently although it’s interesting that only one of them is in the ‘LongRoomClub’ group which perhaps indicates that people in that group are less likely to be members of the other ones. I’d be interested if anyone can think of other potential uses for eigenvector centrality once we’ve got it back in the graph. All the code described in this post is on github if you want to take it for a spin.

This simple function allows you to getect a country code and city of your site visitors, using the www.geoplugin.net service. You may use this data to add default values in registration form or use it for any other purposes, like blocking access to your site from special regions. Source: http://www.apphp.com/index.php?snippet=php-get-country-by-ip function getLocationInfoByIp(){ $client = @$_SERVER['HTTP_CLIENT_IP']; $forward = @$_SERVER['HTTP_X_FORWARDED_FOR']; $remote = @$_SERVER['REMOTE_ADDR']; $result = array('country'=>'', 'city'=>''); if(filter_var($client, FILTER_VALIDATE_IP)){ $ip = $client; }elseif(filter_var($forward, FILTER_VALIDATE_IP)){ $ip = $forward; }else{ $ip = $remote; } $ip_data = @json_decode(file_get_contents("http://www.geoplugin.net/json.gp?ip=".$ip)); if($ip_data && $ip_data->geoplugin_countryName != null){ $result['country'] = $ip_data->geoplugin_countryCode; $result['city'] = $ip_data->geoplugin_city; } return $result; }

The above skyline is from "City," a simple flight-simulator by Ricardo "Mr. Doob" Cabello. "City" is a demo of the capabilities of WebGL, and is written in an impressive 100 lines of JavaScript using Three.js. In his blog post "How to Do a Procedural City in 100 Lines," Jerome Etienne walks you through the process of recreating Cabello's "City." The secret lies in creating 20,000 cubes that are given random sizes and positions and merging them together to create a city. Let's hope that this algorithm is never used for actual city-planning, though, because the buildings can randomly intersect each other and there are no logical spaces for streets! Check out Etienne's blog post here and watch the screencast introduction here:

This post is an extension to the post I wrote some time ago about displaying code snippets on a WordPress site. That post showed how you can create a WordPress plugin that makes a shortcode for a number of different languages that will encode the content so that you can display code snippets on your WordPress site. Using this plugin, you will have access to four shortcodes: HTML, CSS, JavaScript and PHP. HTML .css_class { } var js = document.getElementById('element'); But because we were encoding the content using htmlentities it caused some problems with the way WordPress renders content. WordPress will automatically change double line-breaks to paragraph tags. In this article, we will see what those problems are and come up with a solution to solve them. WordPress Automatic Paragraphs When WordPress displays your content on the page it will run it through two different filters: wptexturize and wpautop. wptexturize is a filter that will convert all quotes into smart quotes. It will also convert apostrophes, dashes, ellipses, the trademark symbol and the multiplication symbol. Because code normally uses straight quotes instead of curly quotes, text inside pre and code tags are ignored. wpautop is a filter that will change double line-breaks into paragragh tags. When you add a new line while using the tinyMCE editor, it will create a double line-break, so this filter solves that problem. This is a problem inside shortcodes that are running the content through the htmlentities() function because it will convert the HTML so it is shown in the shortcode content area. Therefore, if you create a shortcode to display HTML like this: [ html ] [ /html ] Because we have line breaks in this shortcode by going to a new line, wpautop() will try to convert this text. Since we are using htmlentities on the content of the shortcode, the output will display these paragraph tags and line break tags. What will actually be returned from the shortcode is this: You can see the closing paragraph tag and the line break at the end of the line with the div element. If you are trying to display more information, such as a CSS class with a line break on each property, it will be displayed this way: .alert-danger,<;br /> .alert-error { color: #b94a48; background-color: #f2dede; border-color: #eed3d7; } Turning Off wpautop() Removing these line breaks and paragraph tags from the content is very easily done in WordPress. All we have to do is make sure the content doesn't go through the wpautop filter by removing it. To remove a filter in WordPress, all you have to do is use the function remove_filter(), pass in the tag of the_content and remove the function wpautop. remove_filter('the_content', 'wpautop'); But if we use this function to remove the wpautop() then it will be removed for all content, which will remove all paragraphs in your content, which makes your content unreadable. In order to display code snippets correctly, we need to find a way of solving this problem without turning off wpautop() on all content. We will only want to turn this off on content inside shortcodes. Removing wpautop() from Content Only in Shortcodes To remove the wpautop() function only in shortcodes we need to create a brand-new filter through which to run all of our content. First, we turn off wpautop filters and wptexturize filters on all content. remove_filter('the_content', 'wpautop'); remove_filter('the_content', 'wptexturize'); This new filter will need to check to see how the content is being displayed. If the content is inside a shortcode, then it won't run through the wptexturize and wpautop functions. If the content is not inside the shortcodes, then it will apply the wptexturize and wpautop functions. function remove_auto_p_in_shortcode_formatter($content) { $new_content = ''; $pattern_full = '{(\[raw\].*?\[/raw\])}is'; $pattern_contents = '{\[raw\](.*?)\[/raw\]}is'; $pieces = preg_split($pattern_full, $content, -1, PREG_SPLIT_DELIM_CAPTURE); foreach ($pieces as $piece) { if (preg_match($pattern_contents, $piece, $matches)) { $new_content .= $matches[1]; } else { $new_content .= wptexturize(wpautop($piece)); } } return $new_content; } add_filter('the_content', 'remove_auto_p_in_shortcode_formatter', 99); Now, all the code inside shortcodes will not have a trailing paragraph tag and you will no longer have line breaks at the end of your code snippets. If you'd like a copy of the syntax highlighter used on Paulund, it is available on GitHub as a WordPress plugin. Paulund Syntax Highlighter

Imagine that you need to find out how your actors are performing, but without using the Typesafe Console. (You don’t actually want to use this code in production, but it is an interesting project to learn.) What we are interested in is intercepting the receiveMessage call of the ActorCell class. A Short Introduction to AOP We could roll our own build of Akka, but that is not a good idea. Instead, we would like to somehow modify the bytecode of the existing Akka JARs to include our instrumentation code. It turns out that we can use aspect oriented programming to implement cross-cutting concerns in our code. Cross-cutting concerns are pieces of code that cut across the object hierarchy; in other words, we inject the same functionality to different levels of our class structure. In OOP, the only way to share functionality is with inheritance. I will explain the simplest case here, but the same rules apply to mix-in inheritance. If I have: class A { def foo: Int = 42 } class B { def bar: Int = 42 } class AA extends A class BB extends B Then the only way to share the implementation of foo is to extend A; the only way to share the implementation of bar is to extend B. Suppose we now want to measure how many times we call foo on all subtypes of A and bar, but only in subtypes of BB. This now turns out to be rather clumsy. Even if all we do is println("Executing foo") or println("Executing bar"), we have a lot of duplication on our hands. Instead, we would like to write something like this: before executing A.foo { println("Executing foo") } before executing BB.bar { println("Executing bar") } And have some machinery apply this to the classes that make up our system. Importantly, we would like this logic to be applied to every subtype of A and every subtype of BB, even if we only have their compiled forms in a JAR. Enter AspectJ And this is exactly what AspectJ does. It allows us to define these cross-cutting concerns and to weave them into our classes. The weaving can be done at compile time, or even at class load time. In other words, the AspectJ weaver can modify the classes with our cross-cutting concerns as the JVM loads them. The technical details now involve translating our before executing pseudo-syntax into the syntax that AspectJ can use. We are going to be particularly lazy and use AspectJ’s Java syntax. package monitor; import org.aspectj.lang.JoinPoint; import org.aspectj.lang.annotation.Aspect; import org.aspectj.lang.annotation.Before; import org.aspectj.lang.annotation.Pointcut; import javax.management.InstanceAlreadyExistsException; import javax.management.MBeanRegistrationException; import javax.management.MalformedObjectNameException; import javax.management.NotCompliantMBeanException; @Aspect public class MonitorAspect { @Pointcut( value = "execution (* akka.actor.ActorCell.receiveMessage(..))" + "&& args(msg)", argNames = "msg") public void receiveMessagePointcut(Object msg) {} @Before(value = "receiveMessagePointcut(msg)", argNames = "jp,msg") public void message(JoinPoint jp, Object msg) { // log the message System.out.println("Message " + msg); } } The annotations come from the AspectJ dependencies, which, in SBT syntax are just: "org.aspectj" % "aspectjweaver" % "1.7.2", "org.aspectj" % "aspectjrt" % "1.7.2" Excellent. We have defined a pointcut, which you can imagine as a target in the structure of our system. In this particular case, the pointcut says on execution of the receiveMessage method in akka.actor...ActorCell class, with any parameters of any type (..), returning any type * as long as the argument is called msg and inferred to be of the type Object. Without an advice though, a pointcut would be useless–just like having a method that is never called. An advice applies our logic at the point identified by the pointcut. Here, we have the message advice, which runs on execution of the methods matched by the receiveMssagePointcut. At the moment, it does nothing of importance. To see this in action, we need to tell the JVM to use the AspectJ weaver. We do so by specifying the Java agent, which registers a Java Instrumentation implementation, which performs the class file transformation. However, this transformation is costly. Imagine if we had to re-compile every class as it is loaded. To restrict the scope of the transformations, the AspectJ load-time weaver loads an XML file (boo, hiss, I know) from META-INF/aop.xml, which includes its settings. Now, to get it all running all you need to do is include javaagent:~/path-to/aspectjweaver.jar when we start the JVM. Measuring Now that we have our monitor.MonitoringAspect and META-INF/aop.xml ready, all we need to do is implement the logic that records the messages and prints out their per-second averages. I will leave it to the curious reader to come up with a much better approach, but here is one that works, albeit in a very naive way: package monitor; import java.util.HashMap; import java.util.Map; class ActorSystemMessages { private final Map messages = new HashMap<>(); void recordMessage() { long second = System.currentTimeMillis() / 1000; int count = 0; if (messages.containsKey(second)) { count = messages.get(second); } messages.put(second, ++count); } float average() { if (messages.isEmpty()) return 0; int total = 0; for (Integer i : messages.values()) { total += i; } return total / messages.size(); } } We can now use it in our MonitorAspect: @Aspect public class MonitorAspect { private ActorSystemMessages asm = new ActorSystemMessages(); @Pointcut( value = "execution (* akka.actor.ActorCell.receiveMessage(..)) " + "&& args(msg)", argNames = "msg") public void receiveMessagePointcut(Object msg) {} @Before(value = "receiveMessagePointcut(msg)", argNames = "jp,msg") public void message(JoinPoint jp, Object msg) { asm.recordMessage(); System.out.println("Average throughput " + asm.average()); } } Bringing in an Actor Let's see it all run. We will make a simple actor. This time, we will use the Actor DSL. We are not interested that much in its behavior, but we want to see messages being sent around. So, we construct a simple App subclass with two actors. One that sends the messages around and one that prints any message it receives. import akka.actor.{ActorRef, ActorSystem} object Main extends App { import akka.actor.ActorDSL._ import Commands._ implicit val system = ActorSystem() val chatter = actor(new Act { become { case i: Int => self ! (sender, i) case (sender: ActorRef, i: Int) => if (i > 0) self ! (sender, i - 1) else sender ! "zero" } }) implicit val _ = actor(new Act { become { case x => println(">>> " + x) } }) def commandLoop(): Unit = { readLine() match { case CountdownCommand(count) => chatter ! count.toInt case QuitCommand => return } commandLoop() } commandLoop() system.shutdown() } object Commands { val CountdownCommand = """(\d+)""".r val QuitCommand = "quit" } The chatter actor, as you can see, receives the number of messages to be crunched. It then sends a message to itself as a tuple containing the original sender and the number of messages, which will continue to decrease until we hit 0, when we send the "zero"String back to the original sender. As an interesting aside, we have the tail-recursive commandLoop() function that deals with the input that the users type in. Running the Example If you run the example without specifying the -javaagent JVM parameter, the aspect will not be weaved in; consequently, no bytecode will be modified and our logging will not work. Because your IDEs are different, the only reliable way is to run it in SBT. And so, execute sbt run, enter the number of messages and see them displayed. Note that I’m setting the javaOptions, fork and connectInput. The javaOptions is obvious since that’s how I specify the -javaagent parameter and fork makes SBT fork the java process so that the javaOptions takes effect. Finally, the connectInput parameter connects the System.in to the console’s STDIN. (We must do this because we use readLine() in our app.) JMX Now, I don’t like println in the best of the times, and System.out.println is even worse. So, the last modification is to add JMX exporter and to expose the ActorSystemPerformance MBean. The rather baroque JMX code is: public interface ActorSystemPerformanceMXBean { float getMessagesPerSecond(); } public class ActorSystemPerformanceMXBeanImpl implements ActorSystemPerformanceMXBean{ private ActorSystemMessages messages; ActorSystemPerformanceMXBeanImpl(ActorSystemMessages messages) { this.messages = messages; } @Override public float getMessagesPerSecond() { return this.messages.average(); } } public class JMXEndpoint { public static void start(ActorSystemMessages messages) throws ... { MBeanServer mbs = ManagementFactory.getPlatformMBeanServer(); ObjectName name = new ObjectName("monitor:type=Performance"); ActorSystemPerformanceMXBeanImpl mbean = new ActorSystemPerformanceMXBeanImpl(messages); mbs.registerMBean(mbean, name); } } With this in place, we can remove the System.out.println and call the JMXEndpoint.start) method in the aspect’s constructor, giving us: @Aspect public class MonitorAspect { final ActorSystemMessages messages; public MonitorAspect() throws ... { this.messages = new ActorSystemMessages(); JMXEndpoint.start(messages); } @Pointcut(...) public void receiveMessagePointcut(Object msg) {} @Before(...) public void message(JoinPoint jp, Object msg) { messages.recordMessage(); } } Run the application using sbt run again, connect to the JMX MBean using jconsole and see the wonders: Summary This article is a simple exploration of AOP (as implemented in AspectJ) and its use in Scala and Akka. The implementation is very simplistic. If you use it in production, I will endorse you for Enterprise PHP on LinkedIn. However, it is an interesting exercise and really shows how Scala fits well into even the slightly more esoteric Java libraries. The source code for your compiling pleasure is at https//github.com/eigengo/activator-akka-aspectj.

In this post I will compare the pros and cons of using an in-process cache versus a distributed cache and when you should use either one. First, let's see what each one is. As the name suggests, an in-process cache is an object cache built within the same address space as your application. The Google Guava Library provides a simple in-process cache API that is a good example. On the other hand, a distributed cache is external to your application and quite possibly deployed on multiple nodes forming a large logical cache. Memcached is a popular distributed cache. Ehcache from Terracotta is a product that can be configured to function either way. Following are some considerations that should be kept in mind when making a decision. Considerations In-Process Cache Distributed Cache Comments Consistency While using an in-process cache, your cache elements are local to a single instance of your application. Many medium-to-large applications, however, will not have a single application instance as they will most likely be load-balanced. In such a setting, you will end up with as many caches as your application instances, each having a different state resulting in inconsistency. State may however be eventually consistent as cached items time-out or are evicted from all cache instances. Distributed caches, although deployed on a cluster of multiple nodes, offer a single logical view (and state) of the cache. In most cases, an object stored in a distributed cache cluster will reside on a single node in a distributed cache cluster. By means of a hashing algorithm, the cache engine can always determine on which node a particular key-value resides. Since there is always a single state of the cache cluster, it is never inconsistent. If you are caching immutable objects, consistency ceases to be an issue. In such a case, an in-process cache is a better choice as many overheads typically associated with external distributed caches are simply not there. If your application is deployed on multiple nodes, you cache mutable objects and you want your reads to always be consistent rather than eventually consistent, a distributed cache is the way to go. Overheads This dated but very descriptive article describes how an in-process cache can negatively effect performance of an application with an embedded cache primarily due to garbage collection overheads. Your results however are heavily dependent on factors such as the size of the cache and how quickly objects are being evicted and timed-out. A distributed cache will have two major overheads that will make it slower than an in-process cache (but better than not caching at all): network latency and object serialization As described earlier, if you are looking for an always-consistent global cache state in a multi-node deployment, a distributed cache is what you are looking for (at the cost of performance that you may get from a local in-process cache). Reliability An in-process cache makes use of the same heap space as your program so one has to be careful when determining the upper limits of memory usage for the cache. If your program runs out of memory there is no easy way to recover from it. A distributed cache runs as an independent processes across multiple nodes and therefore failure of a single node does not result in a complete failure of the cache. As a result of a node failure, items that are no longer cached will make their way into surviving nodes on the next cache miss. Also in the case of distributed caches, the worst consequence of a complete cache failure should be degraded performance of the application as opposed to complete system failure. An in-process cache seems like a better option for a small and predictable number of frequently accessed, preferably immutable objects. For large, unpredictable volumes, you are better off with a distributed cache. Recommendation For a small, predictable number of preferably immutable objects that have to be read multiple times, an in-process cache is a good solution because it will perform better than a distributed cache. However, for cases in which the number of objects that can be or should be cached is unpredictable and large, and consistency of reads is a must-have, a distributed cache is perhaps a better solution even though it may not bring the same performance benefits as an in-process cache. It goes without saying that your application can use both schemes for different types of objects depending on what suits the scenario best.

Hi! Some time ago I have written an article about Mockito and using RETURNS_DEEP_STUBS when working with JAXB. Quite recently we have faced a similliar issue with deeply nesetd JAXB and the awesome testing framework written in Groovy called Spock. Natively Spock does not support creating deep stubs or spies so we needed to create a workaround for it and this article will show you how to do it. Project structure We will be working on the same data structure as in the RETURNS_DEEP_STUBS when working with JAXB article so the project structure will be quite simillar: As you can see the main difference is such that the tests are present in the /test/groovy/ folder instead of /test/java/ folder. Extended Spock Specification In order to use Spock as a testing framework you have to create Groovy test scripts that extend the Spock Specification class. The details of how to use Spock are available here. In this project I have created an abstract class that extends Specification and adds two additional methods for creating nested Test Doubles (I don't remember if I haven't seen a prototype of this approach somewhere on the internet). ExtendedSpockSpecification.groovy package com.blogspot.toomuchcoding.spock; import spock.lang.Specification /** * Created with IntelliJ IDEA. * User: MGrzejszczak * Date: 14.06.13 * Time: 15:26 */ abstract class ExtendedSpockSpecification extends Specification { /** * The method creates nested structure of spies for all the elements present in the property parameter. Those spies are set on the input object. * * @param object - object on which you want to create nested spies * @param property - field accessors delimited by a dot - JavaBean convention * @return Spy of the last object from the property path */ protected def createNestedSpies(object, String property) { def lastObject = object property.tokenize('.').inject object, { obj, prop -> if (obj[prop] == null) { def foundProp = obj.metaClass.properties.find { it.name == prop } obj[prop] = Spy(foundProp.type) } lastObject = obj[prop] } lastObject } /** * The method creates nested structure of mocks for all the elements present in the property parameter. Those mocks are set on the input object. * * @param object - object on which you want to create nested mocks * @param property - field accessors delimited by a dot - JavaBean convention * @return Mock of the last object from the property path */ protected def createNestedMocks(object, String property) { def lastObject = object property.tokenize('.').inject object, { obj, prop -> def foundProp = obj.metaClass.properties.find { it.name == prop } def mockedProp = Mock(foundProp.type) lastObject."${prop}" >> mockedProp lastObject = mockedProp } lastObject } } These two methods work in a very simillar manner. Assuming that the method's argument property looks as follows: "a.b.c.d" then the methods tokenize the string by "." and iterate over the array -["a","b","c","d"]. We then iterate over the properties of the Meta Class to find the one whose name is equal to prop (for example "a"). If that is the case we then use Spock's Mock/Spy method to create a Test Double of a given class (type). Finally we have to bind the mocked nested element to its parent. For the Spy it's easy since we set the value on the parent (lastObject = obj[prop]). For the mocks however we need to use the overloaded >> operator to record the behavior for our mock - that's why dynamically call the property that is present in the prop variable (lastObject."${prop}" >> mockedProp). Then we return from the closure the mocked/spied instance and we repeat the process for it Class to be tested Let's take a look at the class to be tested: PlayerServiceImpl.java package com.blogspot.toomuchcoding.service; import com.blogspot.toomuchcoding.model.PlayerDetails; /** * User: mgrzejszczak * Date: 08.06.13 * Time: 19:02 */ public class PlayerServiceImpl implements PlayerService { @Override public boolean isPlayerOfGivenCountry(PlayerDetails playerDetails, String country) { String countryValue = playerDetails.getClubDetails().getCountry().getCountryCode().getCountryCode().value(); return countryValue.equalsIgnoreCase(country); } } The test class And now the test class: PlayerServiceImplWrittenUsingSpockTest.groovy package com.blogspot.toomuchcoding.service import com.blogspot.toomuchcoding.model.* import com.blogspot.toomuchcoding.spock.ExtendedSpockSpecification /** * User: mgrzejszczak * Date: 14.06.13 * Time: 16:06 */ class PlayerServiceImplWrittenUsingSpockTest extends ExtendedSpockSpecification { public static final String COUNTRY_CODE_ENG = "ENG"; PlayerServiceImpl objectUnderTest def setup(){ objectUnderTest = new PlayerServiceImpl() } def "should return true if country code is the same when creating nested structures using groovy"() { given: PlayerDetails playerDetails = new PlayerDetails( clubDetails: new ClubDetails( country: new CountryDetails( countryCode: new CountryCodeDetails( countryCode: CountryCodeType.ENG ) ) ) ) when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return true if country code is the same when creating nested structures using spock mocks - requires CGLIB for non interface types"() { given: PlayerDetails playerDetails = Mock() ClubDetails clubDetails = Mock() CountryDetails countryDetails = Mock() CountryCodeDetails countryCodeDetails = Mock() countryCodeDetails.countryCode >> CountryCodeType.ENG countryDetails.countryCode >> countryCodeDetails clubDetails.country >> countryDetails playerDetails.clubDetails >> clubDetails when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return true if country code is the same using ExtendedSpockSpecification's createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.ENG when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return false if country code is not the same using ExtendedSpockSpecification createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.PL when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } def "should return true if country code is the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.ENG when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } def "should return false if country code is not the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.PL when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } } Let's move through the test methods one by one. First I present the code and then have a quick description of the presented snippet. def "should return true if country code is the same when creating nested structures using groovy"() { given: PlayerDetails playerDetails = new PlayerDetails( clubDetails: new ClubDetails( country: new CountryDetails( countryCode: new CountryCodeDetails( countryCode: CountryCodeType.ENG ) ) ) ) when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you could find the approach of creating nested structures by using the Groovy feature of passing properties to be set in the constructor. def "should return true if country code is the same when creating nested structures using spock mocks - requires CGLIB for non interface types"() { given: PlayerDetails playerDetails = Mock() ClubDetails clubDetails = Mock() CountryDetails countryDetails = Mock() CountryCodeDetails countryCodeDetails = Mock() countryCodeDetails.countryCode >> CountryCodeType.ENG countryDetails.countryCode >> countryCodeDetails clubDetails.country >> countryDetails playerDetails.clubDetails >> clubDetails when: boolean playerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you can find a test that creates mocks using Spock - mind you that you need CGLIB as a dependency when you are mocking non interface types. def "should return true if country code is the same using ExtendedSpockSpecification's createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.ENG when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you have an example of creating nested mocks using the createNestedMocks method. def "should return false if country code is not the same using ExtendedSpockSpecification createNestedMocks"() { given: PlayerDetails playerDetails = Mock() CountryCodeDetails countryCodeDetails = createNestedMocks(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode >> CountryCodeType.PL when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } An example showing that creating nested mocks using the createNestedMocks method really works - should return false for improper country code. def "should return true if country code is the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.ENG when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: playerIsOfGivenCountry } Here you have an example of creating nested spies using the createNestedSpies method. def "should return false if country code is not the same using ExtendedSpockSpecification's createNestedSpies"() { given: PlayerDetails playerDetails = Spy() CountryCodeDetails countryCodeDetails = createNestedSpies(playerDetails, "clubDetails.country.countryCode") countryCodeDetails.countryCode = CountryCodeType.PL when: booleanplayerIsOfGivenCountry = objectUnderTest.isPlayerOfGivenCountry(playerDetails, COUNTRY_CODE_ENG); then: !playerIsOfGivenCountry } An example showing that creating nested spies using the createNestedSpies method really works - should return false for improper country code. Summary In this post I have shown you how you can create nested mocks and spies using Spock. It can be useful especially when you are working with nested structures such as JAXB. Still you have to bear in mind that those structures to some extend violate the Law of Demeter. If you check my previous article about Mockito you would see that: We are getting the nested elements from the JAXB generated classes. Although it violates the Law of Demeter it is quite common to call methods of structures because JAXB generated classes are in fact structures so in fact I fully agree with Martin Fowler that it should be called the Suggestion of Demeter. And in case of this example the idea is the same - we are talking about structures so we don't violate the Law of Demeter. Advantages With a single method you can mock/spy nested elements Code cleaner than creating tons of objects that you then have to manually set Disadvantages Your IDE won't help you with providing the property names since the properties are presented as Strings You have to set Test Doubles only in the Specification context (and sometimes you want to go outside this scope) Sources As usual the sources are available at BitBucket and GitHub.

The Java API for JSON Processing (JSR-353) is the Java standard for producing and consuming JSON which was introduced as part of Java EE 7. JSR-353 includes object (DOM like) and stream (StAX like) APIs. In this post I will demonstrate the initial JSR-353 support we have added to MOXy's JSON binding in EclipseLink 2.6. You can now use MOXy to marshal to: javax.json.JsonArrayBuilder javax.json.JsonObjectBuilder And unmarshal from: javax.json.JsonStructure javax.json.JsonObject javax.json.JsonArray You can try this out today using a nightly build of EclipseLink 2.6.0: http://www.eclipse.org/eclipselink/downloads/nightly.php The JSR-353 reference implementation is available here: https://java.net/projects/jsonp/downloads/download/ri/javax.json-ri-1.0.zip Java Model Below is the simple customer model that we will use for this post. Note for this example we are only using the standard JAXB (JSR-222) annotations. Customer package blog.jsonp.moxy; import java.util.*; import javax.xml.bind.annotation.*; @XmlType(propOrder={"id", "firstName", "lastName", "phoneNumbers"}) public class Customer { private int id; private String firstName; private String lastName; private List phoneNumbers = new ArrayList(); public int getId() { return id; } public void setId(int id) { this.id = id; } public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } @XmlElement(nillable=true) public String getLastName() { return lastName; } public void setLastName(String lastName) { this.lastName = lastName; } @XmlElement public List getPhoneNumbers() { return phoneNumbers; } } PhoneNumber package blog.jsonp.moxy; import javax.xml.bind.annotation.*; @XmlAccessorType(XmlAccessType.FIELD) public class PhoneNumber { private String type; private String number; public String getType() { return type; } public void setType(String type) { this.type = type; } public String getNumber() { return number; } public void setNumber(String number) { this.number = number; } } jaxb.properties To specify MOXy as your JAXB provider you need to include a file called jaxb.properties in the same package as your domain model with the following entry (see: Specifying EclipseLink MOXy as your JAXB Provider) javax.xml.bind.context.factory=org.eclipse.persistence.jaxb.JAXBContextFactory Marshal Demo In the demo code below we will use a combination of JSR-353 and MOXy APIs to produce JSON. JSR-353's JsonObjectBuilder and JsonArrayBuilder are used to produces instances of JsonObject and JsonArray. We can use MOXy to marshal to these builders by wrapping them in instances of MOXy's JsonObjectBuilderResult and JsonArrayBuilderResult. package blog.jsonp.moxy; import java.util.*; import javax.json.*; import javax.json.stream.JsonGenerator; import javax.xml.bind.*; import org.eclipse.persistence.jaxb.JAXBContextProperties; import org.eclipse.persistence.oxm.json.*; public class MarshalDemo { public static void main(String[] args) throws Exception { // Create the EclipseLink JAXB (MOXy) Marshaller Map jaxbProperties = new HashMap(2); jaxbProperties.put(JAXBContextProperties.MEDIA_TYPE, "application/json"); jaxbProperties.put(JAXBContextProperties.JSON_INCLUDE_ROOT, false); JAXBContext jc = JAXBContext.newInstance(new Class[] {Customer.class}, jaxbProperties); Marshaller marshaller = jc.createMarshaller(); // Create the JsonArrayBuilder JsonArrayBuilder customersArrayBuilder = Json.createArrayBuilder(); // Build the First Customer Customer customer = new Customer(); customer.setId(1); customer.setFirstName("Jane"); customer.setLastName(null); PhoneNumber phoneNumber = new PhoneNumber(); phoneNumber.setType("cell"); phoneNumber.setNumber("555-1111"); customer.getPhoneNumbers().add(phoneNumber); // Marshal the First Customer Object into the JsonArray JsonArrayBuilderResult result = new JsonArrayBuilderResult(customersArrayBuilder); marshaller.marshal(customer, result); // Build List of PhoneNumer Objects for Second Customer List phoneNumbers = new ArrayList(2); PhoneNumber workPhone = new PhoneNumber(); workPhone.setType("work"); workPhone.setNumber("555-2222"); phoneNumbers.add(workPhone); PhoneNumber homePhone = new PhoneNumber(); homePhone.setType("home"); homePhone.setNumber("555-3333"); phoneNumbers.add(homePhone); // Marshal the List of PhoneNumber Objects JsonArrayBuilderResult arrayBuilderResult = new JsonArrayBuilderResult(); marshaller.marshal(phoneNumbers, arrayBuilderResult); customersArrayBuilder // Use JSR-353 APIs for Second Customer's Data .add(Json.createObjectBuilder() .add("id", 2) .add("firstName", "Bob") .addNull("lastName") // Included Marshalled PhoneNumber Objects .add("phoneNumbers", arrayBuilderResult.getJsonArrayBuilder()) ) .build(); // Write JSON to System.out Map jsonProperties = new HashMap(1); jsonProperties.put(JsonGenerator.PRETTY_PRINTING, true); JsonWriterFactory writerFactory = Json.createWriterFactory(jsonProperties); JsonWriter writer = writerFactory.createWriter(System.out); writer.writeArray(customersArrayBuilder.build()); writer.close(); } } Highlighted lines: 36, 37, 38, 54, 55, 64 Output Below is the output from running the marshal demo (MarshalDemo). The highlighted portions (lines 2-12 and 18-25) correspond to the portions that were populated from our Java model. [ { "id":1, "firstName":"Jane", "lastName":null, "phoneNumbers":[ { "type":"cell", "number":"555-1111" } ] }, { "id":2, "firstName":"Bob", "lastName":null, "phoneNumbers":[ { "type":"work", "number":"555-2222" }, { "type":"home", "number":"555-3333" } ] } ] Highlighted lines: 2-12, 18-25 Unmarshal Demo MOXy enables you to unmarshal from a JSR-353 JsonStructure (JsonObject or JsonArray). To do this simply wrap the JsonStructure in an instance of MOXy's JsonStructureSource and use one of the unmarshal operations that takes an instance of Source. package blog.jsonp.moxy; import java.io.FileInputStream; import java.util.*; import javax.json.*; import javax.xml.bind.*; import org.eclipse.persistence.jaxb.JAXBContextProperties; import org.eclipse.persistence.oxm.json.JsonStructureSource; public class UnmarshalDemo { public static void main(String[] args) throws Exception { try (FileInputStream is = new FileInputStream("src/blog/jsonp/moxy/input.json")) { // Create the EclipseLink JAXB (MOXy) Unmarshaller Map jaxbProperties = new HashMap(2); jaxbProperties.put(JAXBContextProperties.MEDIA_TYPE, "application/json"); jaxbProperties.put(JAXBContextProperties.JSON_INCLUDE_ROOT, false); JAXBContext jc = JAXBContext.newInstance(new Class[] {Customer.class}, jaxbProperties); Unmarshaller unmarshaller = jc.createUnmarshaller(); // Parse the JSON JsonReader jsonReader = Json.createReader(is); // Unmarshal Root Level JsonArray JsonArray customersArray = jsonReader.readArray(); JsonStructureSource arraySource = new JsonStructureSource(customersArray); List customers = (List) unmarshaller.unmarshal(arraySource, Customer.class) .getValue(); for(Customer customer : customers) { System.out.println(customer.getFirstName()); } // Unmarshal Nested JsonObject JsonObject customerObject = customersArray.getJsonObject(1); JsonStructureSource objectSource = new JsonStructureSource(customerObject); Customer customer = unmarshaller.unmarshal(objectSource, Customer.class) .getValue(); for(PhoneNumber phoneNumber : customer.getPhoneNumbers()) { System.out.println(phoneNumber.getNumber()); } } } } Highlighted lines: 27-30, 37-39 Input (input.json) The following JSON input will be converted to a JsonArray using a JsonReader. [ { "id":1, "firstName":"Jane", "lastName":null, "phoneNumbers":[ { "type":"cell", "number":"555-1111" } ] }, { "id":2, "firstName":"Bob", "lastName":null, "phoneNumbers":[ { "type":"work", "number":"555-2222" }, { "type":"home", "number":"555-3333" } ] } ] Highlighted lines: 4, 15, 20, 24 Output Below is the output from running the unmarshal demo (UnmarshalDemo). Jane Bob 555-2222 555-3333

We’re all works in progress, honey. And believe me when I tell you that I’ve had to work harder than most. ― Susan Elizabeth Phillips, "Ain't She Sweet" It's pretty well understood that limiting Work In Progress - or WIP as it is often abbreviated - is a good thing. Ideally, WIP should be limited to one item in progress at a time. Having multiple pieces of inventory on-hand is a form of waste, since each incurs a handling cost, and any work done on one of them will depreciate while another is being worked on. In theory at least, restricting WIP to one item at a time will reduce this waste and get value out of the door as quickly as possible. This principle of Single Piece Flow (SPF) is central to Lean-Kanban ways of working, especially in a manufacturing context. In a software context the accepted WIP limits tend to be rather higher. It is often limited to one item per developer, such as by allowing each developer only one avatar to place on an item, and it can be reduced further if pair-programming is in use. As such, software teams might not often achieve SPF but the value of limiting WIP as far as possible is still understood. There are however problems in interpreting limited WIP in Scrum. This is because a Scrum board will often take the form of a task board ... not a Kanban board. In other words, the work being limited by Scrum teams is not always a user story or similar requirement. It is often a task. Task-limited WIP allows developers to progress tasks from any user story in any order. They could potentially limit themselves to one or two tasks from a story, complete them, then move on to a task from a different story and maybe a task from a third. In effect multiple stories - perhaps even the entire Sprint Backlog of stories - can be in progress before so much as one story gets completed. None of this breaks Scrum rules. There's nothing to stop a team, in Sprint Planning, from organizing the Sprint Backlog into any number of tasks which can be progressed in any order they choose, and from delivering all of the user stories in one go at the end of the Sprint. The Sprint Goal can of course be met by this approach, and there should still be a nice task burn-down to show the associated technical risks being managed. The problem is that it defers approval of each user story to the end of the Sprint (i.e. the Sprint Review), when it is best-practice to get continual sign-off by a Product Owner throughout the iteration. On-going inspection allows the business risks of delivery to be managed well, and not just the technical risks. This is an issue that all Scrum teams must consider when they formulate a Sprint Plan. Is it important to limit WIP in terms of user stories rather than tasks, and thereby facilitate early approval of those stories by a Product Owner? Or would this compromise the team's principle of incremental delivery ... and amount to Lean-Kanban by the back door?

We need to make multiple web-based projects with a lot of shared functionality. For that, some sort of a plugin system would be a good option (as an alternative to copy-pasting stuff). Some frameworks (like grails) have the option to make web plugins, but most don’t, so something custom-made is to be implemented. First, let’s define what is the required functionality. The “plugin”: should be included simply by importing via maven/ivy should register all classes (either automatically, or via a one-line configuration) in a dependency injection container, if one is used should be vertical – i.e. contain all files, from javascript, css and templates, through controllers, to service layer classes should not require complex configuration that needs to be copy-pasted from project to project should allow easy development and debugging without redeployment The java classes are put into a jar file and added to the lib directory, therefore to the classpath, so that’s the easy part. But we need to get the web resources extracted to the respective locations, where they can be used by the rest of the code. There are three general approaches to that: build-time extraction, runtime extraction and runtime loading from the classpath. The last approach would require a controller (or servlet) that loads the resources from the classpath (the respective jar), cache them, and serve them. That has a couple of significant drawbacks, one of which is that being in a jar, they can’t be easily replaced during development. Working with classpath resources is also tricky, as you don’t know the names of the files in advance. The other two approaches are very similar. Grails, for example, uses the build-time extraction – the plugin is a zip file, containing all the needed resources, and they are extracted to the respective locations while the project is built. This is fine, but it would require a little more configuration (maven, in our case), which would also probably have to be copied over from project to project. So we picked the runtime extraction approach. It happens on startup – when the application is loaded, a startup listener of some sort (a spring components with @PostConstruct in our case) iterates through all jar files in the lib folder, and extracts the files from a specific folder (e.g. “web”). So, the structure of the jar file looks like this: com company pkg Foo.class Bar.class web plugin-name css main.css js foo.js bar.js images logo.png views foo.jsp bar.jsp The end-result is that on after the application is started, you get all the needed web resources accessible from the application, so you can include them in the pages (views) of your main application. And the code that does the extraction is rather simple (using zip4j for the zip part). This can be a servlet context listener, rather than a spring bean – it doesn’t make any difference. /** * Component that locates modules (in the form of jar files) and extracts their web elements, if any, on startup * * @author Bozhidar */ @Component public class ModuleExtractor { private static final Logger logger = LoggerFactory.getLogger(ModuleExtractor.class); @Inject private ServletContext ctx; @SuppressWarnings("unchecked") @PostConstruct public void init() { File lib = new File(ctx.getRealPath("/WEB-INF/lib")); File[] jars = lib.listFiles(); String targetPath = ctx.getRealPath("/"); String viewPath = "/WEB-INF/views"; //that can be made configurable for (File jar : jars) { try { ZipFile file = new ZipFile(jar); for (FileHeader header : (List) file.getFileHeaders()) { if (header.getFileName().startsWith("web/") && !fileExists(header)) { // extract views in WEB-INF (inaccessible to the outside world) // all other files are extracted in the root of the application if (header.getFileName().contains("/views/")) { file.extractFile(header, targetPath + viewPath); } else { file.extractFile(header, targetPath); } } } } catch (ZipException ex) { logger.warn("Error opening jar file and looking for a web-module in: " + jar, ex); } } } private boolean fileExists(FileHeader header) { return new File(ctx.getRealPath(header.getFileName())).exists(); } } So, in order to make a plugin, you just make a maven project with jar packaging, and add it as dependency to your main project, everything else is taken care of. You might need to register the ModuleExtractor if classpath scanning for beans is not enabled (or you choose to make it a listener), but that’s it. Note: this solution doesn’t aim to be a full-featured plugin system that solves all problems. It doesn’t support versioning, submodules, etc. That’s why the title is “simple”. But you can do many things with it, and it’s has a very low complexity. Note 2: Servlet 3.0 has a native way of doing almost the same thing, but it doesn’t allow dynamically changing the assets. If you don’t need to change them and don’t need save-and-refresh, then it’s probably the better option.

EclipseLink, reference implementation of JPA, has JPA support for NoSQL databases (MongoDB and Oracle NoSQL) as of the version 2.4. In this tutorial we will discuss the use of MongoDB database with the JPA support of EclipseLink. The transaction previously done using the console and native java driver will be done in a web application with the help of EclipseLink. Tools and technologies used in the sample application are as follows: MongoDB version 2.4.1 MongoDB Java Driver version 2.11.1 JSF version 2.2 PrimeFaces version 3.5 EclipseLink version 2.4 Jetty 7.x Maven Plugin JDK version 1.7 Maven 3.0.4 Project Dependencies org.glassfish javax.faces 2.2.0-SNAPSHOT org.primefaces primefaces 3.5 org.primefaces.themes bootstrap 1.0.10 org.eclipse.persistence org.eclipse.persistence.jpa 2.4.0-SNAPSHOT org.eclipse.persistence org.eclipse.persistence.nosql 2.4.0-SNAPSHOT jboss jboss-j2ee 4.2.2.GA org.mongodb mongo-java-driver 2.11.1 commons-fileupload commons-fileupload 1.3 Entity Class @Entity @NoSql(dataFormat=DataFormatType.MAPPED) public class Article implements Serializable { public Article() { } @Id @GeneratedValue @Field(name="_id") private String id; @ElementCollection private List categoryLists = new ArrayList(); @Basic private String title; @Basic private String content; @Basic @Temporal(javax.persistence.TemporalType.DATE) private Date date; @Basic private String author; @ElementCollection private List tagLists = new ArrayList(); @NoSQL notation sets the data format and type and maps the NoSQL data. Because of using MongoDB in our sample application and documents in MongoDB stored in BSON format, MAP is used as data type. @ElementCollection notation maps the embedded collection into the parent document. Because more than one category and tag associated with an article would be a matter in our sample application, we map them as an element collection. Embedded Objects @Embeddable @NoSql(dataFormat=DataFormatType.MAPPED) public class Categories implements Serializable { @Basic private String category; @Embeddable @NoSql(dataFormat=DataFormatType.MAPPED) public class Tags implements Serializable { @Basic private String tag; We see @Embeddable notation at the top of the Categories and Tags’ class unlike Article entity class. The documents stored in the parent document are mapped with this notation. Please note that embedded objects do not need unique field. persistence.xml com.kodcu.entity.Article com.com.kodcu.entity.Categories com.kodcu.entity.Tags CRUD Operations index.xhtml MyBean.java public void saveArticle() { em.getTransaction().begin(); if(null == article.getId()) em.persist(article); else em.merge(article); em.getTransaction().commit(); } public void removeArticle() { em.getTransaction().begin(); em.remove(selectArticle); em.getTransaction().commit(); } 6. Demo Application Real content above and the demo application, can be accessed at NoSQL with JPA

If you use log4j with your project that also uses Spring framework and/or hibernate or any other framework, and you would feel the need to set different Log levels for your own code and for third party components, you may also want different log levels for your own components. The answer is simple. Configure different loggers for different package names and provide different log levels for each. For example, lets take a look at the following log4j XML In the logger tag, specify the package name/prefix/regex and then specify the log level. The following logger will output only logs of level ERROR and above for packages starting with org.springframework

CQEngine or collection query engine is a library that allows you to build indices over java collections and query them for objects using exposed properties. It offers similar capability to LINQ in .net but is thought to be faster because it builds indices over collections before querying them and uses set theory instead of iterations. In this post we will see how to query a simple collection of objects, in our example, a collection of users of a hypothetical system, using CQEngine. In a subsequent post, we will also see how iteratively searching a collection compares to querying via CQEngine. The first step is to get the CQEengine jar file. Download the jar from the CQEngine website or if you are using Maven, add the following dependency. com.googlecode.cqengine cqengine 1.0.3 Next, lets create the Class whose object we will be searching for: package co.syntx.examples.cqengine; import com.googlecode.cqengine.attribute.Attribute; import com.googlecode.cqengine.attribute.SimpleAttribute; public class User { private String username; private String password; private String fullname; private Role role; public User(String username, String password, String fullname, Role role) { super(); this.username = username; this.password = password; this.fullname = fullname; this.role = role; } public static final Attribute FULL_NAME = new SimpleAttribute("fullname") { public String getValue(User user) { return user.fullname; } }; public static final Attribute USERNAME = new SimpleAttribute("username") { public String getValue(User user) { return user.username; } }; public String getUsername() { return username; } public void setUsername(String username) { this.username = username; } public String getPassword() { return password; } public void setPassword(String password) { this.password = password; } public String getFullname() { return fullname; } public void setFullname(String fullname) { this.fullname = fullname; } public Role getRole() { return role; } public void setRole(Role role) { this.role = role; } } Next, we write a class, to perform our searches. I will go function by function. 1. Function to Build a Test Indexed Collection: In the following function, we build an indexed collection, define indices on attributes, and populate this collection with a certain number of objects. In actual usage, your collection will probably be filled with objects being returned from the DB, read from a file or other similar scenarios. public void buildIndexedCollection(int size) throws Exception { indexedUsers = CQEngine.newInstance(); indexedUsers.addIndex(HashIndex.onAttribute(User.FULL_NAME)); indexedUsers.addIndex(SuffixTreeIndex.onAttribute(User.FULL_NAME)); for (int i = 0; i < size; i++) { String username = RandomStringGenerator.generateRandomString(8,RandomStringGenerator.Mode.ALPHANUMERIC); String password = RandomStringGenerator.generateRandomString(8,RandomStringGenerator.Mode.ALPHANUMERIC); String fullname = RandomStringGenerator.generateRandomString(5,RandomStringGenerator.Mode.ALPHA) + " " + RandomStringGenerator.generateRandomString(5,RandomStringGenerator.Mode.ALPHA); Role role = new Role(); role.setName("admin"); indexedUsers.add(new User(username, password, fullname, role)); } } In line 3 we are initializing a new Indexed Collection, a reference of which is stored in the class variable indexedUsers. The reference is of type IndexedCollection In lines 4 and 5, we define two indices i) a Hash Index suitable for equal style queries. ii) a Suffix Index suitable for ends with style queried. For the purpose of this example, we are building indices only on the the Full name field. In line 8, we use a random string generator to populate dummy objects. In line 14 we add our object to our indexed collection. 2. Function to Perform Indexed Search for Exact Matches: In this function, we are querying for names that exactly match a given name. The equal function takes in the attribute upon which to perform the query and the value to search. The method equal is statically imported via a import static com.googlecode.cqengine.query.QueryFactory.*; In the example below, we are looping the results returned by the retrieve method and not doing anything with it. In your case, you may choose to return the Iterator returned by retrieve. public void indexedSearchForEquals(String fullname) throws Exception { Query query = equal(User.FULL_NAME, fullname); for (User user : indexedUsers.retrieve(query)) { // System.out.println(user.getFullname()); } } 3. Function to Perform Indexed Search for Ends With Matches: In this function, we are querying for names that end with a certain suffix. public void indexedSearchForEndsWith(String endswith) throws Exception { Query query1 = endsWith(User.FULL_NAME, endswith); for (User user : indexedUsers.retrieve(query1)) { // System.out.println(user.getFullname()); } } 4. Function to Perform Indexed Search for Equals or Ends With Matches: This function is a combination of both queries mentioned below and has an or relation between them. public void indexedSearchForEqualOrEndsWith(String equals, String ends) throws Exception { Query query = or(equal(User.FULL_NAME, equals),endsWith(User.FULL_NAME, ends)); for (User user : indexedUsers.retrieve(query)) { // System.out.println(user.getFullname()); } } 5. Putting it together: All the functions above belong to a class called CQEngineTest. We create a new object, build a test collection, then search for either exact matches, strings that end with a certain suffix or either. CQEngineTest test = new CQEngineTest(); test.buildIndexedCollection(size); test.indexedSearchForEqualOrEndsWith("test", "test"); In this example, we have used a Hash Index and a Suffix Tree Index. There are many other types of indices that you can choose depending on the type of query that you want to perform. A list of these indices and when to use them can be found on the cqengine project page. Also, apart from equal or endswith operations, there are others that you would typically expect to find. In a subsequent post we will also see how an indexed search compares with a typical iterative search in terms of times.