Some interesting insider info was released this week about the Facebook deployment process. I've compiled a TL;DR version here in case you don't want to read the whole piece, but I recommend you do. The information comes straight from Chuck Rossi, the head of Facebook's release engineering team, who was interviewed by Ars Technica's Ryan Paul: An important aspect of Facebook's development culture is the idea that developers are fully responsible for how their code behaves in production. This philosophy mirrors the "DevOps" movement, which encourages lowering the wall between software development and IT operations. If any of the code in a Facebook update causes problems in production, the developer who wrote it is on the hook for making sure that the issue gets resolved as quickly as possible. --Ryan Paul, Ars Technica "Exclusive: a behind-the-scenes look at Facebook release engineering" I think we've got more evidence here that "DevOps" is just a name for what many super-successful companies have already been doing. On to the TL;DR! For deployment Facebook uses Uses custom BitTorrent P2P system Site updates take 15-30 minutes JS, CSS, and Graphics are hosted on distributed CDNs Deployment steps Update goes to an "a2" tier which rolls update out to a small, random collection of users. Check-in procedure on company IRC where all developers who submitted code for the update have confirm that they are ready to respond if there is a problem when the update goes out. Head of release engineering issues command in terminal to begin deployment Watch web-based monitoring dashboards as the update rolls out. A small amount of servers will fail during most deployments, but it usually doesn't cause any issues. That's because any of Facebook's servers can handle any sort of page request, so they don't have to worry about serialization and migration of user session states. The servers can keep handling incoming page requests during their software updates. No-downtime, as we all know. Frequency of deployments: One minor update on most business days One major update on a weekly basis, usually Tuesdays "Release early and often." The mantra rings true at Google, Facebook, and many other companies that we look up to as the best in the business. Make sure you check out the whole article for the full story on Facebook's release engineering.

Here is an A - Z list of some Javascript idioms and patterns. The idea is to convey in simple terms some features of the actual Javascript language (rather than how it can interact with DOM). Enjoy... Array Literals An array literal can be defined using a comma separated list in square brackets. var months = ['jan', 'feb', 'mar', 'apr', 'may', 'jun', 'jul', 'aug', 'sep', 'oct', 'nov', 'dec']; console.log(months[0]); // outputs jan console.log(months.length) // outputs 12 Arrays in javascript have a wide selection methods including push() and pop(). Suppose the world got taken over by a dictator who wanted to get rid of the last month of the year? The dictator would just do... months.pop(); And of course, the dictator will eventually want to add a month after himself when everyone will have to worship him: months.push("me"); Callbacks Since functions are objects, they can be passed as arguments to other functions. function peakOil(callback) { //... code callback(); // the parentheses mean the function is executed! } function changeCivilisationCallback(){ //... } // Now pass the changeCivilisationCallback to peakOil. // Note: no changeCivilisationCallback parentheses because it is not // executed at this point. // It will be excuted later inside peak oil. peakOil(changeCivilisationCallback); In the example above, the chanceCivilisationCallback callback function is invoked by peakOil. Logic could be added to check if the energy returns from solar panels and wind farms were sufficient in which case another callback, other than changeCivilisationCallback could be added. Configuration Object Instead of passing around a bunch of related properties... function addCar(colour, wheelsize, regplate) {...} Use a configuration object function addCar(carConf) {...} var myCarConf = { colour: "blue", wheelsize: "32", regplate: "00D98788" }; addCar(myCarConf); The use of a configuration object makes it makes it easier to write clean APIs that don't need to take a huge long list of parameters. They also means you are less likely to get silly errors if parameters are in the wrong order. Closures There are three ways to creats objects in Javascript: using literals, using the constuctor function and by using a closure. What closures offer that the other two approaches do not is encapsulation. Closures make it possible to hide away functions and variables. var counter = function(count) { console.log(">> setting count to " + this.count); return { getCount: function(){ return ++count; } } } mycounter = counter(0); console.log(mycounter.getCount()); // outputs 1 console.log(mycounter.getCount()); // outputs 2 console.log(mycounter.getCount()); // outputs 3 console.log(mycounter.getCount()); // outputs 4 // Same again with offset this time. mycounterWithOffset = counter(10); console.log(mycounterWithOffset .getCount()); // outputs 11 console.log(mycounterWithOffset .getCount()); // outputs 12 console.log(mycounterWithOffset .getCount()); // outputs 13 console.log(mycounterWithOffset .getCount()); // outputs 14 Note: The closure is the object literal returned from annoymous function. It "closes" over the count variable. No-one can access it except for the closure. It is encapsulated. The closure also has a sense of state. Note also how the it maintains the value of the counter. Constructor Functions (Built in) There are no classes in Javascript but there are construtor functions which use the new keyword syntax similar to the class based object creation in Java or other languages. Javascript has some built-in constructor functions. These include Object(), Date(), String() etc. var person = new Object(); // person variable is an Object person.name = "alex"; // properties can then be dynamically added Constructor Functions (Custom) When a function is invoked with the keyword new, it is referred to as a Constructor function. The new means that the new object will have a hidden link to value of the function's prototype member and the this keyword will be bound to the new object. function MyConstrutorFunction() { this.goodblog = "dublintech.blogspot.com"; } var newObject = new MyConstrutorFunction(); console.log(typeof newObject); // "object" console.log(newObject.goodblog); // "dublintech.blogspot.com" var noNewObject = MyConstrutorFunction(); console.log(typeof noNewObject); // "undefined" console.log(window.tastes); // "yummy" The convention is that constructor functions should begin with a capital letter. Note: if the new keyword is not used, then the 'this' variable inside the function will refer to the global object. Can you smell a potential mess? Hence why the capital letter convention for constructor functions is used. The capital letter means: "I am a constructor function, please use the new keyword". Currying Currying is the process of reducing the number of arguments passed to a function by setting some argument(s) to predefined values. Consider this function. function outputNumbers(begin, end) { var i; for (i = begin; i <= end; i++) { print(i); } } outputNumbers(0, 5); // outputs 0, 1, 2, 3, 4, 5 outputNumbers(1, 5); // outputs 1, 2, 3, 4, 5 Suppose, we want a similar function with a fixed "begin" value. Let's say the "begin" value was always 1. We could do: function outputNumbersFixedStart(start) { return function(end) { return outputNumbers(start, end); } } And then define a variable to be this new function... var outputFromOne = outputNumbersFixedStart(1); outputFromOne(3); 1, 2, 3 outputFromOne(5); 1, 2, 3, 4, 5 Delete Operator The delete operator can be used to remove properties from objects and arrays. var person = {name: 'Alex', age: 56}; // damn I don't want them to know my age remove it delete person.age; console.log("name" in person); // outputs true because it is still there console.log("age" in person); // outputs false var colours = ['red', 'green', 'blue'] // is red really in the array? console.log(colours.indexOf('red') > -1); // outputs true. // remove red, it's going out of fashion! delete colours[colours.indexOf('red')]; console.log(colours.indexOf('red') > -1); // outputs false console.log(colours.length) // length is still three, remember it's javascript! You cannot delete global variables or prototype attributes. console.log(delete Object.prototype) // can't be deleted, outputs false function MyFunction() { // ... } console.log(delete MyFunction.prototype) // can't be deleted, outputs false var myglobalVar = 1; console.log(delete this.myglobalVar) // can't be delete, outputs false Dynamic Arguments Arguments for a function do not have to be specifed in the function definition function myFunction(){ // ... Note myfunction has no arguments in signature for(var i=0; i < arguments.length; i++){ alert(arguments[i].value); } } myFunction("tony", "Magoo"); // any argument can be specified The arguments parameter is an array available to functions and gives access to all arguments that were specified in the invocation. for-in iterations for-in loops (also called enumeration) should be used to iterate over nonarray objects. var counties = { dublin: "good", kildare: "not bad", cork: "avoid" } for (var i in counties) { if (counties.hasOwnProperty(i)) { // filter out prototype properties console.log(i, ":", counties[i]); } } Function declaration In a function declaration, the function stands on its own and does not need to be assigned to anything. function multiple(a, b) { return a * b; } // Note, no semi colon is needed Function expressions When function is defined as part of something else's definition, it is considered a function expression. multiply = function multiplyFunction(a, b) { return a * b; }; // Note the semi colan must be placed after the function definition console.log(multiply(5, 10)); // outputs 50 In the above example, the function is named. It can also be anonymous, in which case the name property will be a blank string. multiply = function (a, b) { return a * b; } // Note the semi colan must be placed after the function definition console.log(multiply(5, 10)); // outputs 50 Functional Inheritance Functional inheritance is mechanism of inheritance that provides encapsulation by using closures. Before trying to understand the syntax, take an example first. Suppose we want to represent planets in the solar system. We decided to have a planet base object and then several planet child objects which inherit from the base object. Here is the base planet object: var planet = function(spec) { var that = {}; that.getName = function() { return spec.radius; }; that.getNumberOfMoons()= function() { return spec.numberOfMoons; }; return that; } Now for some planets. Let's start with Earth and Jupiter and to amuse ourselves let's add a function for Earth for people to leave and a function to Jupiter for people arriving. Sarah Palin has taken over and things have got pretty bad!!! var earth = function(spec) { var that = planet{spec}; // No need for new keyword! that.peopleLeave = function() { // ... people leave } return that; } var jupiter = function(spec) { var that = planet(spec); that.peopleArrive = function() { // .. people arrive } return that; } Now put the earth and jupiter in motion... var myEarth = earth({name:"earth",numberofmoons:1}); var myjupiter=jupiter({name:"jupiter",numberofmoons:66}); The three key points here: There is code reuse. There is encapsulation. The name and numberOfMoons properties are encapsulated. The child objects can add in their own specific functionality. Now an explanation of the syntax: The base object planet accepts some data in the spec object. The base object planet creates a closures called that which is returned. The that object has access to everything in the spec object. But, nothing else does. This provides a layer of encapsulation. The child objects, earth and jupiter, set up their own data and pass it to base planet object. The planet object returns a closure which contains base functionality. The child classes receive this closure and add further methods and variables to it. Hoisting No matter where var's are declared in a function, javascript will "hoist" them meaning that they behave as if they were declared at the top of the function. mylocation = "dublin"; // global variable function outputPosition() { console.log(mylocation); // outputs "undefined" not "dublin" var mylocation = "fingal" ; console.log(mylocation); // outputs "fingal" } outputPosition(); In the function above, the var declaration in the function means that the first log will "see" the mylocation in the function scope and not the one declared in the global scope. After declaration, the local mylocation var will have the value "undefined", hence why this is outputted first. Functions that are assigned to variables can also be hoisted. The only difference being that when functions are hoisted, their definitions also are - not just their declarations. Immediate Function Expressions Immediate function expression are executed as soon as they are defined. (function() { console.log("I ain't waiting around"); }()); There are two aspects of the syntax to note here. Firsty, there is a () immediately after the function definiton, this makes it execute. Secondly, the function can only execute if it is a function expression as opposed to a function declaration. The outer () make the function an expression. Another way to define a an immediate function expression is: var anotherWay = function() { console.log("I ain't waiting around"); }() JSON JavaScript Object Notation (JSON) is a notation used to represent objects. It is very similar to the format used for Javascript Object literals except the property names must be wrapped in quotes. The JSON format is not exclusive to javascript; it can be used by any language (Python, Ruby etc). JSON makes it very easy to see what's an array and what's an object. In XML this would be much harder. An external document - such as XSD - would have to be consulted. In this example, Mitt Romney has an array describing who might vore for him and an object which is his son. {"name": "Mitt Romney", "party": "republicans", "scary": "of course", "romneysMostLikelyVoters": ["oilguzzlers", "conservatives"], son : {name:'George Romney} Loose typing Javascript is loosely typed. This means that variables do not need to be typed. It also means there is no complex class hierarchies and there is no casting. var number1 = 50; var number2 = "51"; function output(varToOutput) { // function does not care about what type the parameter passed is. console.log(varToOutput); } output(number1); // outputs 50 output(number2); // outputs 51 Memoization Memoization is a mechanism whereby functions can cache data from previous executions. function myFunc(param){ if (!myFunc.cache) { myFunc.cache = {}; // If the cache doesn't exist, create it. } if (!myFunc.cache[param]) { //... Imagine the code to work out result below // is computationally intensive. var result = { //... }; myFunc.cache[param] = result; // now result is cached. } return myFunc.cache[param]; } Method When a function is stored as a property of an object, it is referred to as a method. var myObject { myProperty: function () { //... // the this keyword in here will refer to the myObject instance. // This means the "method" can read and change variables in the // object. } } Modules The goal of modules is to enable javascript code bases to more modular. Functions and variables are collated into a module and then the module can decide what functions and what variables the outside world can see - in the same way as encapsulations works in the object orientated paradigms. In javascript we create modules by combining characteristics of closures and immediate function expressions. var bankAccountModule = (function moduleScope() { var balance = 0; //private function doSomethingPrivate(){ // private method //... } return { //exposed to public addMoney: function(money) { //... }, withDrawMoney: function(money) { //... }, getBalance: function() { return balance; } }()); In the example above, we have a bank account module: The function expression moduleScope has its own scope. The private variable balance and the private function doSomethingPrivate, exist only within this scope and are only visible to functions within this scope. The moduleScope function returns an object literal. This is a closure which has access to the private variables and functions of moduleScope. The returned object's properties are "public" and accesible to the outside world. The returned object is automatically assigned to bankAccountModule The immediate function ()) syntax is used. This means that the module is initialised immediately. Because the returned object (the closure) is assigned to bankAccountModule, it means we can access the bankAccountModule as: bankAccountModule.addMoney(20); bankAccoumtModule.withdrawMoney(15); By convention, the filename of a module should match its namespace. So in this example, the filename should be bankAccountModule.js. Namespace Pattern Javascript doesn't have namespaces built into the language, meaning it is easy for variables to clash. Unless variables are defined in a function, they are considered global. However, it is possible to use "." in variables names. Meaning you can pretend you have name spaces. DUBLINTECH.myName = "Alex" DUBLINTECH.myAddress = "Dublin Object Literal Notation In javascript you can define an object as collection of name value pairs. The values can be property values or functions. var ireland = { capital: "Dublin", getCapital: function () { return this.capital; } }; Prototype properties (inheritance) Every object has a prototype object. It is useful when you want to add a property to all instances of a particular object. Suppose you have a constructor function, which representent Irish people who bought in the boom. function IrishPersonBoughtInTheBoom(){ } var mary = new IrishPersonBoughtInTheBoom (); var tony = new IrishPersonBoughtInTheBoom (); var peter = new IrishPersonBoughtInTheBoom (); ... Now, the Irish economy goes belly up, the property bubble explodes and you want to add a debt property to all instances of this function. To do this you would do: IrishPersonBoughtInTheBoom.prototype.debt = "ouch"; Then... console.log(mary.debt); // outputs "ouch" console.log(tony.debt); // outputs "ouch" console.log(peter.debt); // outputs "ouch" Now, when this approach is used, all instances of IrishPersonBoughtInTheBoom share the save copy of the debt property. This means, that they all have the same value as illustrated in this example. Returning functions A function always returns a value. If return is not specified for a function, the undefined value type will be returned. Javascript functions can also return some data or another function. var counter = function() { //... var count = 0; return function () { return count = count + 1; } } var nextValue = counter(); nextValue(); // outputs 1 nextValue(); // outputs 2 Note, in this case the inner function which is returned "closes" over the count variable - making it a closure - since it encapsulates its own count variable. This means it gets its own copy which is different to the variable return by nextValue.count. this keyword The this keyword in Java has different meanings, depending on the context it is used. In summary: In a method context, this refers to the object that contains the method. In a function context, this refers to the global object. Unless the function is a property of another object. In which case the this refers to that object. If this is used in a constructor, the this in the constructor function refers to the object which uses the constructor function. When the apply or call methods are used the value of this refers to what was explictly specified in the apply or call invocation. typeof typeof is a unary operator with one operand. It is used to determine the types of things (a bit like getClass() in Java). The values outputted by typeof are "number", "string", "boolean", "undefined", "function", "object". console.log(typeof "tony"); // outputs string console.log(typeof 6); // outputs number console.log(false); // outputs boolean console.log(this.doesNotExist); // outputs undefined if the global scope has no such var console.log(typeof function(){}); // outputs function console.log(typeof {name:"I am an object"}); //outputs object console.log(typeof ["I am an array"]) // typedef outputs object for arrays console.log(typeof null) // typedef outputs object for nulls Some implementations return "object" for typeof for regular expressions; others return "function". But the biggest problem with typeof is that it returns object for null. To test for null, use strict equality... if (myobject === null) { ... } Self-redefining functions This is a good performance technique. Suppose you have a function and the first time it is called you want it to perform some set up code that you never want to perfom again. You can execute the set up code and then make the function redefine itself after that so that the setup code is never re-excuted. var myFunction = function () { //set up code only to this once alert("set up, only called once"); // set up code now complete. // redefine function so that set up code is not re-executed myFunction = function() { alert("no set up code"); } } myFunction(); // outputs - Set up, only called once myFunction(); // outputs - no set up code this time myFunction(); // outputs - no set up code this time Note, any properties added to the set up part of this function will be lost when the function redefines itself. In addition, if this function is used with a different name (i.e. it is assigned to a variable), the re-definition will not happen and the set up code will re-execute. Scope In javascript there is a global scope and a function scope available for variables. The var keyword does not need to be used to define variable in the global scope but it must be used to define variable in the local function scope. When a variable is scoped to a local function shares the name with a global variable, the local scope takes precedence - unless var was not used to declare the local variable in which case any local references are pointing to the global reference. There is no block scope in javascript. By block we mean the code between {}, aka curly braces. var myFunction = function () { var noBlockScope = function ( ) { if (true) { // you'd think that d would only be visible to this if statement var d = 24; } if (true) { // this if statement can see the variable defined in the other if statement console.log(d); } } noBlockScope(); Single var pattern You can define all variables used by a function in one place. It is ensures tidy code and is considered best practise. function scrum() { var numberOfProps = 2, numberOfHookers = 1, numberOfSecondRows = 2, numberOfBackRow = 3 // function body... } If a variable is declared but not initialized with a value it will have the value undefined. Strict Equality In javascript it is possible to compare two objects using ==. However, in some cases this will perform type conversion which can yield unexpected equality matches. To ensure there is strict comparison (i.e. no type conversions) use the === syntax. console.log(1 == true) // outputs true console.log(1 === true) // outputs false console.log(45 == "45") // outputs true console.log(45 === "45") // outputs false Truthy and Falsey When javascript expects a boolean, you may specify a value of any type. Values that convert to true are said to be truthy and values that convert to false are said to be falsey. Example of truthy values are objects, arrays, functions, strings and numbers: // This will output 'Wow, they were all true' if ({} && {sillyproperty:"sillyvalue"} && [] && ['element'] && function() {} && "string" && 89) { console.log("wow, they were all true"); } Examples of falsey values are empty strings, undefined, null and the value 0. // This will out put: 'none of them were true' if (!("" || undefined || null || 0)) { console.log("none of them were true"); } Undefined and null In javascript, the undefined value means not initialised or unknown where null means an absence of a value. References JavaScript patterns Stoyan Stefanov JavaScript, The Definitive Guide David Flanagan JavaScript, The Good Parts Doug Crockford.

Brute force string matching is a very basic sub-string matching algorithm, but it’s good for some reasons. For example it doesn’t require preprocessing of the text or the pattern. The problem is that it’s very slow. That is why in many cases brute force matching can’t be very useful. For pattern matching we need something faster, but to understand other sub-string matching algorithms let’s take a look once again at brute force matching. In brute force sub-string matching we checked every single character from the text with the first character of the pattern. Once we have a match between them we shift the comparison between the second character of the pattern with the next character of the text, as shown on the picture below. This algorithm is slow for mainly two reasons. First, we have to check every single character from the text. On the other hand even if we find a match between a text character and the first character of the pattern we continue to check step by step (character by character) every single symbol of the pattern in order to find whether it is in the text. So is there any other approach to find whether the text contains the pattern? In fact there is a “faster” approach. In this case, in order to avoid the comparison between the pattern and the text character by character, we’ll try to compare them all at once, so we need a good hash function. With its help we can hash the pattern and check against hashed sub-strings of the text. We must be sure that the hash function is returning “small” hash codes for larger sub-strings. Another problem is that for larger patterns we can’t expect to have short hashes. But besides this the approach should be quite effective compared to the brute force string matching. This approach is known as Rabin-Karp algorithm. Overview Michael O. Rabin and Richard M. Karp came up with the idea of hashing the pattern and to check it against a hashed sub-string from the text in 1987. In general the idea seems quite simple, the only thing is that we need a hash function that gives different hashes for different sub-strings. Said hash function, for instance, may use the ASCII codes for every character, but we must be careful for multi-lingual support. The hash function may vary depending on many things, so it may consist of ASCII char to number converting, but it can also be anything else. The only thing we need is to convert a string (pattern) into some hash that is faster to compare. Let’s say we have the string “hello world”, and let’s assume that its hash is hash(‘hello world’) = 12345. So if hash(‘he’) = 1 we can say that the pattern “he” is contained in the text “hello world”. So in every step, we take from the text a sub-string with the length of m, where m is the pattern length. Thus we hash this sub-string and we can directly compare it to the hashed pattern, as in the picture above. Implementation So far we saw some diagrams explaining the Rabin-Karp algorithm, but let’s take a look at its implementation here, in this very basic example where a simple hash table is used in order to convert the characters into integers. The code is PHP and it’s used only to illustrate the principles of this algorithm. function hash_string($str, $len) { $hash = ''; $hash_table = array( 'h' => 1, 'e' => 2, 'l' => 3, 'o' => 4, 'w' => 5, 'r' => 6, 'd' => 7, ); for ($i = 0; $i < $len; $i++) { $hash .= $hash_table[$str{$i}]; } return (int)$hash; } function rabin_karp($text, $pattern) { $n = strlen($text); $m = strlen($pattern); $text_hash = hash_string(substr($text, 0, $m), $m); $pattern_hash = hash_string($pattern, $m); for ($i = 0; $i < $n-$m+1; $i++) { if ($text_hash == $pattern_hash) { return $i; } $text_hash = hash_string(substr($text, $i, $m), $m); } return -1; } // 2 echo rabin_karp('hello world', 'ello'); Multiple Pattern Match It’s great to say that the Rabin-Karp algorithm is great for multiple pattern match. Indeed its nature is supposed to support such functionality, which is its advantage in comparison to other string searching algorithms. Complexity The Rabin-Karp algorithm has the complexity of O(nm) where n, of course, is the length of the text, while m is the length of the pattern. So where is it compared to brute-force matching? Well, brute force matching complexity is O(nm), so as it seems there’s not much of a gain in performance. However, it’s considered that Rabin-Karp’s complexity is O(n+m) in practice, and that makes it a bit faster, as shown on the chart below. Note that the Rabin-Karp algorithm also needs O(m) preprocessing time. Application As we saw Rabin-Karp is not much faster than brute force matching. So where we should use it? 3 Reasons Why Rabin-Karp is Cool 1. Good for plagiarism, because it can deal with multiple pattern matching! 2. Not faster than brute force matching in theory, but in practice its complexity is O(n+m)! 3. With a good hashing function it can be quite effective and it’s easy to implement! 2 Reasons Why Rabin-Karp is Not Cool 1. There are lots of string matching algorithms that are faster than O(n+m) 2. It’s practically as slow as brute force matching and it requires additional space Final Words Rabin-Karp is a great algorithm for one simple reason – it can be used to match against multiple patterns. This makes it perfect to detect plagiarism even for larger phrases.

Microsoft offered programmers several different ways of dealing with the asynchronous programming since .NET 1.0. The first model was Asynchronous programming model or APM for short. The pattern is implemented with two methods named BeginOperation and EndOperation. .NET 4 introduced new pattern – Task Asynchronous Pattern and with the introduction of .NET 4.5, Microsoft added language support for language integrated asynchronous coding style. You can check the MSDN for more samples and information. I will assume that you are familiar with it and have written code using it. You can wrap existing APM pattern into TPL pattern using the Task.Factory.FromAsync methods. For example: public static Task> ExecuteAsync(this DataServiceQuery query, object state) { return Task.Factory.FromAsync>(query.BeginExecute, query.EndExecute, state); } It is easy to wrap most of the asynchronous functions this way, but some cannot be since the wrapper functions assume that the last two parameters to the BeginOperation are AsyncCallback and object, and there are some versions of asynchronous operations that have different specifications. Examples: Extra parameters after the object state parameter: IAsyncResult DataServiceContext.BeginExecuteBatch( AsyncCallback callback, object state, params DataServiceRequest[] queries); Missing the expected object state parameter and different return type: ICancelableAsyncResult BeginQuery(AsyncCallback callBack); WorkItemCollection EndQuery(ICancelableAsyncResult car); Short solution for the first example The short and elegant way for wrapping the first example is to provide the following wrapper: public static Task ExecuteBatchAsync(this DataServiceContext context, object state, params DataServiceRequest[] queries) { if (context == null) throw new ArgumentNullException("context"); return Task.Factory.FromAsync( context.BeginExecuteBatch(null, state, queries), context.EndExecuteBatch); } We simply call the Begin method ourselves and then wrap it using an another overload for FromAsync function. The longer way However, we can fully wrap it ourselves by simulating what the FromAsync wrapper does. The complete code is listed below. public static Task ExecuteBatchAsync(this DataServiceContext context, object state, params DataServiceRequest[] queries) { // this will be our sentry that will know when our async operation is completed var tcs = new TaskCompletionSource(); try { context.BeginExecuteBatch((iar) => { try { var result = context.EndExecuteBatch(iar as ICancelableAsyncResult); tcs.TrySetResult(result); } catch (OperationCanceledException ex) { // if the inner operation was canceled, this task is cancelled too tcs.TrySetCanceled(); } catch (Exception ex) { // general exception has been set bool flag = tcs.TrySetException(ex); if (flag && ex as ThreadAbortException != null) { tcs.Task.m_contingentProperties.m_exceptionsHolder.MarkAsHandled(false); } } }, state, queries); } catch { tcs.TrySetResult(default(DataServiceResponse)); // propagate exceptions to the outside throw; } return tcs.Task; } Besides educational benefits, writing the full wrapper code allows us to add cancellation, logging and diagnostic information. Once we understand how to wrap APM pattern, We can now tackle the second problem easily. Handling the BeginQuery/EndQuery We will first create our own wrapper function in the spirit of the above code with the notable difference that we use the ICancelableAsyncResult interface instead of the IAsyncResult. public static class TaskEx { public static Task FromAsync(Func beginMethod, Func endMethod) { if (beginMethod == null) throw new ArgumentNullException("beginMethod"); if (endMethod == null) throw new ArgumentNullException("endMethod"); var tcs = new TaskCompletionSource(); try { beginMethod((iar) => { try { var result = endMethod(iar as ICancelableAsyncResult); tcs.TrySetResult(result); } catch (OperationCanceledException ex) { tcs.TrySetCanceled(); } catch (Exception ex) { bool flag = tcs.TrySetException(ex); if (flag && ex as ThreadAbortException != null) { tcs.Task.m_contingentProperties.m_exceptionsHolder.MarkAsHandled(false); } } }); } catch { tcs.TrySetResult(default(TResult)); throw; } return tcs.Task; } } The code is pretty self-explanatory and we can go ahead with the wrapping. There are four different operations that are exposed both in synchronous and asynchronous version: Query, LinkQuery, CountOnlyQuery and RegularQuery. The extension methods are short since we have already created our generic wrapper above: public static Task RunQueryAsync(this Query query) { return TaskEx.FromAsync(query.BeginQuery, query.EndQuery); } public static Task RunLinkQueryAsync(this Query query) { return TaskEx.FromAsync(query.BeginLinkQuery, query.EndLinkQuery); } public static Task RunCountOnlyQueryAsync(this Query query) { return TaskEx.FromAsync(query.BeginCountOnlyQuery, query.EndCountOnlyQuery); } public static Task RunRegularQueryAsync(this Query query) { return TaskEx.FromAsync(query.BeginRegularQuery, query.EndRegularQuery); } That is it for today, you can write your own handy extensions easily for APM functions out there.

In my recent projects I've had to do a lot with certificates, java and HTTPS with client-side authentication. In most of these projects, either during testing, or setting up a new environment, I've run into various SSL configuration errors that often resulted in a rather uncomprehensive error such as: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) In most of the cases it was misconfiguration where keystores didn't containt the correct certificates, the certificate chain was incomplete or the client didn't supply a valid certificate. So in the last project I decided to document what was happening and what caused specific errors during the SSL handshake. In this article I'll show you why specific SSL errors occur, how you can detect them by analyzing the handshake information, and how to solve them. For this I use the following scenario: Server uses a certificate issued by a CA and requires client authentication. The server uses a simple truststore that lists this CA as trusted. Client connects using a certificate issued by this single trusted CA and has it's own trustore that also contains this certificate from the server. Not a very complicated situation, but one you often see. Note that the following information can also be used to identify problems when you don't work with client certificates or use self-signed certificates. The way to determine the problem in those cases, is pretty much the same. Happy Flow First we'll look at the happy flow, what happens in the handshake when we use client certificates. We won't look at the complete negotiation phase, but only until both the client and the server have exchanged their certificates and have validated the received certificate. If everything goes well until that point, the rest should work. The following is what you see when you run the client and the server using the java VM parameter: -Djavax.net.debug=ssl:handshake. The first thing that happens is that the client sends a ClientHello message using the TLS protocol version he supports, a random number and a list of suggested cipher suites and compression methods. From our client this looks like this: Client sends: *** ClientHello, TLSv1 RandomCookie: GMT: 1331663143 bytes = { 141, 219, 18, 140, 148, 60, 33, 241, 10, 21, 31, 90, 88, 145, 34, 153, 238, 105, 148, 72, 163, 210, 233, 49, 99, 224, 226, 64 } Session ID: {} Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] Compression Methods: { 0 } *** The server responds, very originally, with a ServerHello message, that contains the choices made based on the information provided by the client another random number and (optionally) a session id. Server sends: *** ServerHello, TLSv1 RandomCookie: GMT: 1331663143 bytes = { 172, 233, 79, 197, 14, 21, 187, 161, 114, 206, 7, 38, 188, 228, 120, 102, 115, 214, 155, 86, 211, 41, 156, 179, 138, 2, 230, 81 } Session ID: {79, 96, 145, 39, 203, 136, 206, 69, 170, 46, 194, 17, 154, 175, 13, 138, 143, 199, 162, 193, 110, 86, 113, 109, 248, 187, 220, 169, 47, 180, 44, 68} Cipher Suite: SSL_RSA_WITH_RC4_128_MD5 Compression Method: 0 Extension renegotiation_info, renegotiated_connection: *** So in this case we're going to use SSL_RSA_WITH_RC4_128_MD5 as Cipher Suite. The next step is also done by the server. The server next sends a Certificate message that contains its complete certificate chain: Server sends: *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=server, C=NL Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Key: Sun RSA public key, 1024 bits modulus: 143864428144045085986129639694300995179398936575198896494655652087658861594939489453166811774109137006267822033915476680673848164790815913192075840268069822357600376998775923266017630332239546722181180383155088413406178660120548292599278819762883993031950564327152510982887716901499177102158407884939613382007 public exponent: 65537 Validity: [From: Wed Mar 14 13:32:04 CET 2012, To: Thu Mar 14 13:32:04 CET 2013] Issuer: CN=Application CA, OU=GKD, O=Smartjava, L=Maasland, ST=ZH, C=NL SerialNumber: [ a881d144 5e631f21] ] Algorithm: [SHA1withRSA] Signature: 0000: C3 56 81 7F 33 91 8A FF 84 5E 0B BA 7A 01 D8 41 .V..3....^..z..A 0010: 6B 47 B2 F7 8F FB B5 77 23 D8 FB B2 35 19 6E C4 kG.....w#...5.n. 0020: A4 6A BC 23 BB 69 92 F6 85 5A 1E CB FE 23 C6 98 .j.#.i...Z...#.. 0030: A0 57 F8 FB E9 DB B0 40 BD 8E F8 35 F8 77 E1 09 [email protected].. 0040: 5A 2E 45 71 80 F6 89 E7 0B 93 E2 48 EB 40 92 13 Z.Eq.......H.@.. 0050: 14 AA 1F 59 AA 98 67 46 9B 52 33 49 9A 3C 91 9B ...Y..gF.R3I.<.. 0060: F1 CB 8A BD 7D D4 DD 76 C4 15 00 36 A3 B2 87 A7 .......v...6.... 0070: D5 FF 52 E3 68 D4 F0 E0 32 86 74 02 DD 92 EC 1D ..R.h...2.t..... ] chain [1] = [ [ Version: V3 Subject: CN=Application CA, OU=SL, O=SmartJava, L=Waalwijk, ST=ZH, C=NL Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Key: Sun RSA public key, 1024 bits modulus: 159927271510058538658170959055540487654246676457579822126433656091883150307639380685203152841988861440546492270915750324654620063428634486478674507234742748515614639629692189315918046446256610037776978028900716455223387878926383828815082154427031884246429239077082613371662803582187768145965112751392402313823 public exponent: 65537 Validity: [From: Mon Mar 12 13:35:16 CET 2012, To: Wed Apr 11 14:35:16 CEST 2012] Issuer: CN=Application CA, OU=CA, O=Blaat, L=Waalwijk, ST=ZH, C=NL SerialNumber: [ fe7636c5 6804e69c] Certificate Extensions: 3 [1]: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 6C CC 48 03 E4 BE 07 D6 9E F6 4C 78 53 54 A2 B8 l.H.......LxST.. 0010: 7B DA 40 09 ..@. ] ] [2]: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: 6C CC 48 03 E4 BE 07 D6 9E F6 4C 78 53 54 A2 B8 l.H.......LxST.. 0010: 7B DA 40 09 ..@. ] ] [3]: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:true PathLen:2147483647 ] ] Algorithm: [SHA1withRSA] Signature: 0000: 1A 30 08 15 01 8E A6 36 5F 38 22 C6 81 5E 69 B1 .0.....6_8"..^i. 0010: 42 9A 1E FF 0F C4 D7 40 5F 85 0E 42 35 E0 CC 00 B......@_..B5... 0020: 6E A5 2E 70 6B 79 64 C5 99 AE A4 29 CB 26 DE 60 n..pkyd....).&.` 0030: 0B A6 AB 19 06 6F 19 54 6C 1A 88 9E 3A 6A D4 BB .....o.Tl...:j.. 0040: CB 28 85 2F 72 4D DE 35 C0 9B F4 2F EF 8E 6D E8 .(./rM.5.../..m. 0050: 30 AC 12 7D B4 0D A3 08 DA D4 60 46 94 BD 12 AF 0.........`F.... 0060: 44 F7 C3 B8 9D 69 2D 6A 32 C8 4D AE 12 60 05 09 D....i-j2.M..`.. 0070: FE AE D0 1A 72 6D 91 CE DA 7C 8E D5 31 14 31 4C ....rm......1.1L ] In this message you can see that the issuer of this certificate is our example CA. Our client checks to see if this certificate is trusted, which it is in this case. Since we require the client to authenticate itself the server requests a certificate from the client and after that sends a helloDone. Server sends: *** CertificateRequest Cert Types: RSA, DSS Cert Authorities: *** ServerHelloDone In this message you can see that the server provides a list of Cert Authorities it trusts. The client will use this information to determine if it has a keypair that matches this CA. In our happy flow, it has one and responds with a Certificate message. Client sends: *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=Application 3, OU=Smartjava, O=Smartjava, L=NL, ST=ZH, C=NL Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Key: Sun RSA public key, 1024 bits modulus: 90655907749318585147523875906892969031300830816947226352221659107570169820452561428696751943383590982109524990627182456571533992582229229163232831159652561902456847954385746762477844009336466314872376131553489447601649924116778337873632641536164462534398137791450495316700015095054427027256393580022887087767 public exponent: 65537 Validity: [From: Mon Mar 12 15:13:24 CET 2012, To: Tue Mar 12 15:13:24 CET 2013] Issuer: CN=Application CA, OU=Smartjava, O=Smartjava, L=Maasland, ST=ZH, C=NL SerialNumber: [ b247ffb2 ce060768] ] Algorithm: [SHA1withRSA] Signature: 0000: 97 58 36 C5 28 87 B3 16 9B DD 31 0C E0 C6 23 76 .X6.(.....1...#v 0010: 72 82 5B 13 4D 23 B6 0E A9 2F 9F 0C 3F 97 15 6E r.[.M#.../..?..n 0020: 7B 38 EC DE E2 57 D7 AA 07 12 E3 98 B7 86 A7 CE .8...W.......... 0030: 57 8E A1 29 96 C9 F0 30 57 67 C7 F1 F2 98 90 64 W..)...0Wg.....d 0040: 6C B9 6C 05 24 8B 56 3F B1 FF 03 62 3D 81 DB 45 l.l.$.V?...b=..E 0050: D3 1F C1 B2 DD 77 CF 74 54 EB 9D 82 23 89 1A 70 .....w.tT...#..p 0060: F8 C4 68 6A B7 41 C7 DE 7B B6 3A 0C 17 E7 FA 98 ..hj.A....:..... 0070: 19 0C D8 91 FB 5E FE D2 B3 92 FD 2D 2A 6B 51 10 .....^.....-*kQ. ] chain [1] = [ [ Version: V3 Subject: CN=Application CA, OU=Smartjava, O=Smartjava, L=Maasland, ST=ZH, C=NL Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Key: Sun RSA public key, 1024 bits modulus: 159927271510058538658170959055540487654246676457579822126433656091883150307639380685203152841988861440546492270915750324654620063428634486478674507234742748515614639629692189315918046446256610037776978028900716455223387878926383828815082154427031884246429239077082613371662803582187768145965112751392402313823 public exponent: 65537 Validity: [From: Mon Mar 12 13:35:16 CET 2012, To: Wed Apr 11 14:35:16 CEST 2012] Issuer: CN=Application CA, OU=Smartjava, O=Smartjava, L=Maasland, ST=ZH, C=NL SerialNumber: [ fe7636c5 6804e69c] Certificate Extensions: 3 [1]: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 6C CC 48 03 E4 BE 07 D6 9E F6 4C 78 53 54 A2 B8 l.H.......LxST.. 0010: 7B DA 40 09 ..@. ] ] [2]: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: 6C CC 48 03 E4 BE 07 D6 9E F6 4C 78 53 54 A2 B8 l.H.......LxST.. 0010: 7B DA 40 09 ..@. ] ] [3]: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:true PathLen:2147483647 ] ] Algorithm: [SHA1withRSA] Signature: 0000: 1A 30 08 15 01 8E A6 36 5F 38 22 C6 81 5E 69 B1 .0.....6_8"..^i. 0010: 42 9A 1E FF 0F C4 D7 40 5F 85 0E 42 35 E0 CC 00 B......@_..B5... 0020: 6E A5 2E 70 6B 79 64 C5 99 AE A4 29 CB 26 DE 60 n..pkyd....).&.` 0030: 0B A6 AB 19 06 6F 19 54 6C 1A 88 9E 3A 6A D4 BB .....o.Tl...:j.. 0040: CB 28 85 2F 72 4D DE 35 C0 9B F4 2F EF 8E 6D E8 .(./rM.5.../..m. 0050: 30 AC 12 7D B4 0D A3 08 DA D4 60 46 94 BD 12 AF 0.........`F.... 0060: 44 F7 C3 B8 9D 69 2D 6A 32 C8 4D AE 12 60 05 09 D....i-j2.M..`.. 0070: FE AE D0 1A 72 6D 91 CE DA 7C 8E D5 31 14 31 4C ....rm......1.1L ] This certificate is checked on the server side and if all is well, the final steps in the handshake are executed to setup the secured connection. Note that there is a CertificateVerify step. In this step the client signs a message with its private key. This is done so the server can verify the client has access to its private key. This might seem a step where things can go wrong in an incorrectly configured environment. In the default java implementation this won't happen. In the phase where the client has to determine which certificate to present to the server, the java implementation already checks if the privatekey is available. What could possibly go wrong So what could possibly go wrong in this handshake? In the next couple of sections we'll look at some scenarios, and how to detect them. Passwords Now that we've seen what happens when things go right, lets look at a couple of scenarios where things go wrong. We'll start simple with the following exception, that we get at the moment we start up the client application: Exception in thread "main" java.security.UnrecoverableKeyException: Cannot recover key at sun.security.provider.KeyProtector.recover(KeyProtector.java:311) at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:121) at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:38) at java.security.KeyStore.getKey(KeyStore.java:763) at com.sun.net.ssl.internal.ssl.SunX509KeyManagerImpl.(SunX509KeyManagerImpl.java:113) at com.sun.net.ssl.internal.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:48) at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:239) at org.apache.http.conn.ssl.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:186) at org.apache.http.conn.ssl.SSLSocketFactory.(SSLSocketFactory.java:260) This very helpful message is thrown when (from the javadoc) " .. a key in the keystore cannot be recovered". There are a couple of reasons this can happen, but normally this occurs when the key in the keystore is accessed with the wrong password. Usually when you use the keytool to create and manage your keys, the keystore password is usually the same as the key password. However, if you import keys from a PKCS#12 type keystore, the password of the keystore can be easily set to a different value. Not all the SSL client allow you to specify a different password for the key and the keystore. If that is the case you can use the following command, to change the password of the key: keytool -keypasswd -alias -keystore It is also possible to set an incorrect password for the keystore. Luckily in that case the error message that is thrown is much more helpful: Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:771) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:38) at java.security.KeyStore.load(KeyStore.java:1185) ... Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:769) ... 3 more If this occurs at the server side, we can see the same message when the SSL listener is being set up. Incomplete CA Chains Now lets look at the first of the "peer not authenticated" exceptions. In the logging we see this exception at the client side: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) So enable SSL logging, run again, and we'll start with analyzing the handshake. We'll start by looking from the client side. If we look through the logging we find the following CertificateRequest message from the server and the ServerHelloDone. *** CertificateRequest Cert Types: RSA, DSS Cert Authorities: *** ServerHelloDone So thus far, everything went ok. The server has already sent its certificate, and since our client doesn't throw an error on that part, we can assume it is trusted by the client. So something seems to be wrong with the steps that come after this message from the server. If you look closer at this message, you can see that the server doesn't specify a set of Cert Authorities it trusts. This could be a misconfiguration at the server side, or it could just be that the server expects one of the trusted Root CAs. In any case, the client is free to send any certificate he wants. So the client sends the following certificate: *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=Application4, OU=Smartjava, O=Smartjava, L=NL, ST=NB, C=NL Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3 ... ] chain [1] = [ [ Version: V3 Subject: [email protected], CN=CA2, OU=Smartjava, O=Smartjava, L=Waalwijk, ST=NB, C=NL Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3 ... ] According to the specification the client now continues with the key exchange and generates secrets to exchange. Somewhere along the lines we can see the following: pool-1-thread-1, WRITE: TLSv1 Handshake, length = 32 pool-1-thread-1, READ: TLSv1 Alert, length = 2 pool-1-thread-1, RECV TLSv1 ALERT: fatal, internal_error pool-1-thread-1, called closeSocket() This means we've received an internal error. So something at the server side went wrong. Looking at the server we see the following in the SSL dump: *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=Application4, OU=Smartjava, O=Smartjava, L=NL, ST=NB, C=NL Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3 ... ] chain [1] = [ [ Version: V3 Subject: [email protected], CN=CA2, OU=Smartjava, O=Smartjava, L=Waalwijk, ST=NB, C=NL Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3 ... ] *** qtp1735121130-17, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty qtp1735121130-17, SEND TLSv1 ALERT: fatal, description = internal_error qtp1735121130-17, WRITE: TLSv1 Alert, length = 2 You can see that we received the certificate from the client, and directly after that we get this error. This error however doesn't really tell us anything. We do however have enough information to at least limit the possible errors. We know that the server didn't sent a list of CAs, we can see that the client sent a valid certificate, and that server somehow isn't able to process it. It looks like a problem with the server truststore. In this case the best approach is to look at the certificates the server trusts. Either in the cacerts file or in it's own truststore. Validate whether the CA certificate our client sends is in the server's truststore, and the server actually loads the stores we expect. It's of course also possible that the client has an incomplete chain of trust for the certificate received from the server. In that case we once again get the "peer not authenticated" error at the client side. If we look at the SSL debug logging, we see the following exception occuring at the client side: pool-1-thread-1, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty pool-1-thread-1, SEND TLSv1 ALERT: fatal, description = internal_error pool-1-thread-1, WRITE: TLSv1 Alert, length = 2 This exception occured directly after the server has sent its certificate using a "Certificate message": *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=server, C=NL Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Following the same reasoning as for the server we can conclude that there is something wrong with the client side truststore. For completeness sake, the server receives this error message when this situation occurs at the client: qtp1500389297-17, READ: TLSv1 Alert, length = 2 qtp1500389297-17, RECV TLSv1 ALERT: fatal, internal_error qtp1500389297-17, called closeSocket() qtp1500389297-17, handling exception: javax.net.ssl.SSLException: Received fatal alert: internal_error qtp1500389297-17, called close() qtp1500389297-17, called closeInternal(true) Invalid keys For the next exercise lets look at the following error that occurs during this handshake. In the logging at the client side we see the following error message in the SSL output: ool-1-thread-1, WRITE: TLSv1 Handshake, length = 32 pool-1-thread-1, READ: TLSv1 Alert, length = 2 pool-1-thread-1, RECV TLSv1 ALERT: fatal, internal_error pool-1-thread-1, called closeSocket() pool-1-thread-1, handling exception: javax.net.ssl.SSLException: Received fatal alert: internal_error pool-1-thread-1, IOException in getSession(): javax.net.ssl.SSLException: Received fatal alert: internal_error Which results in the very unhelpful: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) When you receive an internal error, there is usually something wrong at the server side. So looking at the serverside, lets see what caused this error. *** qtp2044601711-16, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty qtp2044601711-16, SEND TLSv1 ALERT: fatal, description = internal_error Hmm.. somewhat more useful. It seems that there is something wrong with the algorithm we used, the client seems to have provided an incorrect certificate. But what is wrong? If you look back at the happy flow, you can send that at a certain time the server asks the client for a certificate using a "Certificate" message. Lets look a bit closer at this message and the response: *** CertificateRequest Cert Types: RSA, DSS Cert Authorities: *** ServerHelloDone matching alias: application4 *** Certificate chain chain [0] = [ [ Version: V1 Subject: CN=Application4, OU=Smartjava, O=Smartjava, L=NL, ST=NB, C=NL Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3 Key: Sun DSA Public Key ... What you can see here is that the server specifies the cert types it accepts, and the authorities it accepts. The client responses in this case however with a DSA public key. Depending on the server implementation this can cause this strange message. Another possible scenario I've seen (especially with self-signed certificates) is that with a "CertificateRequest" message like this: *** CertificateRequest Cert Types: RSA, DSS Cert Authorities: *** ServerHelloDone This client won't respond with a certificate at all, if you only have DSA based keys in your keystore. It won't throw an error on the client side, but will cause a "null certificate chain" message as the server side. I haven't seen this scenario, though, when you don't use self-signed certificates. Certificate expiration So far we've seen how you can analyze the SSL handshake to determine where to look for configuration errors. In this last example we'll look at what happens when a certificate expires. In this case we once again see the very cryptic message at the client side: pool-1-thread-1, READ: TLSv1 Alert, length = 2 pool-1-thread-1, RECV TLSv1 ALERT: fatal, certificate_unknown pool-1-thread-1, called closeSocket() pool-1-thread-1, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown pool-1-thread-1, IOException in getSession(): javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown pool-1-thread-1, called close() pool-1-thread-1, called closeInternal(true) pool-1-thread-1, called close() pool-1-thread-1, called closeInternal(true) javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352) at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) If we look at the phase of the SSL handshake we're in, we can see that we've already sent our client certificate and finishing up the handshake when we receive this error. The error on the serverside is actually pretty helpful. After receiving the invalid certificate, in the debug logging, it shows us the following: *** qtp1735121130-17, SEND TLSv1 ALERT: fatal, description = certificate_unknown qtp1735121130-17, WRITE: TLSv1 Alert, length = 2 [Raw write]: length = 7 0000: 15 03 01 00 02 02 2E ....... qtp1735121130-17, called closeSocket() qtp1735121130-17, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed qtp1735121130-17, called close() qtp1735121130-17, called closeInternal(true) It tells us that during the validation of the certificate, a timestamp check failed. This tells us that we should look at the validity of the certificates in our certificate chain to see what is happening. Summary In this article you've seen a couple of common causes for SSL exceptions and ways to identify the exception. Their can be many causes for these kind of exceptions, the most common though are the following: Incorrect certificate chains in the client truststore Incorrect certificate chains in the server truststore Invalid key algorithm used for private keys Expired certificate or expired CA certificate Incorrect passwords used to access the keys Multiple private keys to choose from If you're presented with a such an exception a good general approach is this. You first check the keystores that are involved. Use the java keytool for this: keytool -list -v -keystore This will print out all the certificates and keys in the keystore. Check whether the keys are of a supported type, the required CA certificates are stored and that your application is using the correct one (spent hours figuring out an issue because I was looking into a truststore for my private key). If everything seems to be OK at first glance it's time to enable ssl debugging (-Djavax.net.debug=ssl:handshake) and check the handshake messages that are sent. Wikipedia has a nice overview of which message is sent at a specific time. For more information on the content of the messages look at the RFC 5246 (or the one of the SSL/TLS version you're using, but the handshake changes are minimal between versions). Using the messages and the handshake, determine at what place in the handshake things go wrong, taking into account that the client will continue with the handshake, while the server is processing it's certificate.

Here is a good and simple anti cross-site scripting (XSS) filter written for Java web applications. What it basically does is remove all suspicious strings from request parameters before returning them to the application. It’s an improvement over my previous post on the topic. You should configure it as the first filter in your chain (web.xml) and it’s generally a good idea to let it catch every request made to your site. The actual implementation consists of two classes, the actual filter is quite simple, it wraps the HTTP request object in a specialized HttpServletRequestWrapper that will perform our filtering. public class XSSFilter implements Filter { @Override public void init(FilterConfig filterConfig) throws ServletException { } @Override public void destroy() { } @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { chain.doFilter(new XSSRequestWrapper((HttpServletRequest) request), response); } } The wrapper overrides the getParameterValues(), getParameter() and getHeader() methods to execute the filtering before returning the desired field to the caller. The actual XSS checking and striping is performed in the stripXSS() private method. import java.util.regex.Pattern; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletRequestWrapper; public class XSSRequestWrapper extends HttpServletRequestWrapper { private static Pattern[] patterns = new Pattern[]{ // Script fragments Pattern.compile("", Pattern.CASE_INSENSITIVE), // src='...' Pattern.compile("src[\r\n]*=[\r\n]*\\\'(.*?)\\\'", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL), Pattern.compile("src[\r\n]*=[\r\n]*\\\"(.*?)\\\"", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL), // lonely script tags Pattern.compile("", Pattern.CASE_INSENSITIVE), Pattern.compile("", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL), // eval(...) Pattern.compile("eval\\((.*?)\\)", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL), // expression(...) Pattern.compile("expression\\((.*?)\\)", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL), // javascript:... Pattern.compile("javascript:", Pattern.CASE_INSENSITIVE), // vbscript:... Pattern.compile("vbscript:", Pattern.CASE_INSENSITIVE), // onload(...)=... Pattern.compile("onload(.*?)=", Pattern.CASE_INSENSITIVE | Pattern.MULTILINE | Pattern.DOTALL) }; public XSSRequestWrapper(HttpServletRequest servletRequest) { super(servletRequest); } @Override public String[] getParameterValues(String parameter) { String[] values = super.getParameterValues(parameter); if (values == null) { return null; } int count = values.length; String[] encodedValues = new String[count]; for (int i = 0; i < count; i++) { encodedValues[i] = stripXSS(values[i]); } return encodedValues; } @Override public String getParameter(String parameter) { String value = super.getParameter(parameter); return stripXSS(value); } @Override public String getHeader(String name) { String value = super.getHeader(name); return stripXSS(value); } private String stripXSS(String value) { if (value != null) { // NOTE: It's highly recommended to use the ESAPI library and uncomment the following line to // avoid encoded attacks. // value = ESAPI.encoder().canonicalize(value); // Avoid null characters value = value.replaceAll("\0", ""); // Remove all sections that match a pattern for (Pattern scriptPattern : patterns){ value = scriptPattern.matcher(value).replaceAll(""); } } return value; } } Notice the comment about the ESAPI library, I strongly recommend you check it out and try to include it in your projects. If you want to dig deeper on the topic I suggest you check out the OWASP page about XSS and RSnake’s XSS (Cross Site Scripting) Cheat Sheet.

In JavaScript, there are three main ways in which any value can be converted to a string. This blog post explains each way, along with its advantages and disadvantages. Three approaches for converting to string The three approaches for converting to string are: value.toString() "" + value String(value) The problem with approach #1 is that it doesn’t work if the value is null or undefined. That leaves us with approaches #2 and #3, which are basically equivalent. ""+value: The plus operator is fine for converting a value when it is surrounded by non-empty strings. As a way for converting a value to string, I find it less descriptive of one’s intentions. But that is a matter of taste, some people prefer this approach to String(value). String(value): This approach is nicely explicit: Apply the function String() to value. The only problem is that this function call will confuse some people, especially those coming from Java, because String is also a constructor. However, function and constructor produce completely different results: > String("abc") === new String("abc") false > typeof String("abc") 'string' > String("abc") instanceof String false > typeof new String("abc") 'object' > new String("abc") instanceof String true The function produces, as promised, a string (a primitive [1]). The constructor produces an instance of the type String (an object). The latter is hardly ever useful in JavaScript, which is why you can usually forget about String as a constructor and concentrate on its role as converting to string. A minor difference between ""+value and String(value) Until now you have heard that + and String() convert their “argument” to string. But how do they actually do that? It turns out that they do it in slightly different ways, but usually arrive at the same result. Converting primitives to string Both approaches use the internal ToString() operation to convert primitives to string. “Internal” means: a function specified by the ECMAScript 5.1 (§9.8) that isn’t accessible to the language itself. The following table explains how ToString() operates on primitives. Argument Result undefined "undefined" null "null" boolean value either "true" or "false" number value the number as a string, e.g. "1.765" string value no conversion necessary Converting objects to string Both approaches first convert an object to a primitive, before converting that primitive to string. However, + uses the internal ToNumber() operator (except for dates [2]), while String() uses ToString(). ToNumber(): To convert an object obj to a primitive, invoke obj.valueOf(). If the result is primitive, return that result. Otherwise, invoke obj.toString(). If the result is primitive, return that result. Otherwise, throw a TypeError. ToString(): Works the same, but invokes obj.toString() before obj.valueOf(). With the following object, you can observe the difference: var obj = { valueOf: function () { console.log("valueOf"); return {}; // not a primitive, keep going }, toString: function () { console.log("toString"); return {}; // not a primitive, keep going } }; Interaction: > "" + obj valueOf toString TypeError: Cannot convert object to primitive value > String(obj) toString valueOf TypeError: Cannot convert object to primitive value Most objects use the default implementation of valueOf() which returns this for objects. Hence, that method will always be skipped by ToNumber(). > var x = {} > x.valueOf() === x true Instances of Boolean, Number, and String wrap primitives and valueOf returns the wrapped primitive. But that still means that the final result will be the same as for toString(), even though it will have been produced in a different manner. > var n = new Number(756) > n.valueOf() === n false > n.valueOf() === 756 true Conclusion Which of the three approaches for converting to string should you choose? value.toString() can be OK, if you are sure that value will never be null or undefined. Otherwise, ""+value and String(value) are mostly equivalent. Which one people prefer is a matter of taste. I find String(value) more explicit. Related posts JavaScript values: not everything is an object [primitives versus objects] What is {} + {} in JavaScript? [explains how the + operator works] String concatenation in JavaScript [how to best concatenate many strings]

Like most node modules "natural" is packaged as an NPM and can be installed from the command line with node.js.

String matching is something crucial for database development and text processing software. Fortunately, every modern programming language and library is full of functions for string processing that help us in our everyday work. However it's important to understand their principles. String algorithms can typically be divided into several categories. One of these categories is string matching. When it comes to string matching, the most basic approach is what is known as brute force, which simply means to check every single character from the text to match against the pattern. In general we have a text and a pattern (most commonly shorter than the text). What we need to do is to answer the question whether this pattern appears in the text. Overview The principles of brute force string matching are quite simple. We must check for a match between the first characters of the pattern with the first character of the text as on the picture bellow. If they don’t match, we move forward to the second character of the text. Now we compare the first character of the pattern with the second character of the text. If they don’t match again, we move forward until we get a match or until we reach the end of the text. In case they match, we move forward to the second character of the pattern comparing it with the “next” character of the text, as shown in the picture bellow. Just because we have found a match between the first character from the pattern and some character of the text, doesn’t mean that the pattern appears in the text. We must move forward to see whether the full pattern is contained in the text. Implementation Implementation of brute force string matching is easy and here we can see a short PHP example. The bad news is that this algorithm is naturally quite slow. function sub_string($pattern, $subject) { $n = strlen($subject); $m = strlen($pattern); for ($i = 0; i < $n-$m; $i++) { $j = 0; while ($j < $m && $subject[$i+$j] == $pattern[$j]) { $j++; } if ($j == $m) return $i; } return -1; } echo sub_string('o wo', 'hello world!'); Complexity As I said this algorithm is slow. Actually every algorithm that contains “brute force” in its name is slow, but to show how slow string matching is, I can say that its complexity is O(n.m). Here n is the length of the text, while m is the length of the pattern. In case we fix the length of the text and test against variable length of the pattern, again we get a rapidly growing function. Application Brute force string matching can be very ineffective, but it can also be very handy in some cases. Just like the sequential search. It can be very useful… Doesn’t require pre-processing of the text – Indeed if we search the text only once we don’t need to pre-process it. Most of the algorithms for string matching need to build an index of the text in order to search quickly. This is great when you’ve to search more than once into a text, but if you do only once, perhaps (for short texts) brute force matching is great! Doesn’t require additional space – Because brute force matching doesn’t need pre-processing it also doesn’t require more space, which is one cool feature of this algorithm Can be quite effective for short texts and patterns It can be ineffective… If we search the text more than once – As I said in the previous section if you perform the search more than once it’s perhaps better to use another string matching algorithm that builds an index, and it’s faster. It’s slow – In general brute force algorithms are slow and brute force matching isn’t an exception. Final Words String matching is something very special in software development and it is used in various cases, so every developer must be familiar with this topic.

As with most new and useful technologies and techniques, it is often easy to overuse them in scenarios where they are not appropriate. If you have a shiny new hammer, the screws start to look like nails :). In this blog I want to discuss the requirement for high-frequency, low-latency event dispatch between different threads in a process and use of Disruptor. When The Disruptor framework is certainly a good fit where consumers of events need to receive all events that are published. However I think it is a poor fit if: 1) Subscribers of events do not need to receive all events that are published. For example, if they are receiving prices for a particular stock, they might only need the last value. Interestingly, in my domain, the sort of events that occur in high-frequency and that need to be dispatched with low-latency are often events that can be conflated in some form. Of course, that's not true for everybody and if all subscribers need to receive each and every event, then using Disruptor makes sense. 2) Subscribers can become slow and performance should remain good when some consumers are fast and others are slow. Why Lets take the scenario where a single publisher is generating stock prices for a single stock at a rate of 1000 prices per second. There are three subsribers in the process. If one of the subscribers blocks for 1 second, then 1000 events will backup in the RIngBuffer. Subscribers that are not keeping up with the publisher will cause the Java VM to work harder on minor garbage collections as newly created events live longer in the ringbuffer. In a more terminal scenario, say you size your ringbuffer at 1,000,000 events, if one subscriber blocks for more than 1000 seconds, then the publisher will have to block as well, as the ringbuffer becomes full, affecting the two fast subscribers that were keeping up. The impact on garbage collection will be even heavier, causing latency jitter. In this scenario, where events can be conflated, I would rather not use Disruptor, instead I would have the publisher dispatch to each subscriber, via some sort of conflating queue, here is a snippet that makes it clearer: public class SingleValueConflatingDispatcher { private final Subscriber subscriber; private volatile Event lastEvent; private SingleValueConflatingDispatcher(Subscriber subscriber) { this.subscriber = subscriber; } public void add(Event event) { this.lastEvent = event; } public void dispatch() { subscriber.dispatch(lastEvent); } } The publisher thread invokes add() and the subscriber thread invokes dispatch(). For more complex conflation strategies it's likely you'll need to use locks rather than memory barriers to prevent race conditions in such a ComplicatedConflatingDispatcher, however for me that's a small price to pay. The net effect is that there is no impact from any slow consumer, on other consumers. The conflation happens on the publisher's thread, rather than queuing up a large number of events. Mutability One suggestion to reduce the impact of garbage collection when using Disruptor with slower consumers is to use mutable events for the entries in the ringbuffer. Its true that mutable events are probably the only way to get to zero-GC, which helps give you the lowest levels of latency jitter possible. In the scenario, where conflation is possible, I would much rather stick with immutable events. Where you have lots of complex business logic operating on events downstream, with different developers/teams writing code operating on those events, its just too easy to introduce concurrency bugs with mutable events. I sleep much easier at night, knowing all events are immutable. Also while mutable events reduce the effects garbage collection, it does not prevent the publisher from blocking if one subscriber blocks for too long. However, if your usecase complexity is constrained and/or you have a small, skilled team building it all, then you do have the option of using mutable events. SpinLocks If Disruptor is right for your usecase, do watch out overuse of the BusySpinWaitStrategy. Stating the obvious, only use this when you can afford to burn away a CPU core when the application is doing nothing. Otherwise you have a range of WaitStrategies available..

A few posts ago, we saw how to use the function numpy.linalg.lstsq(...) to solve an over-determined system. This time, we'll use it to estimate the parameters of a regression line. A linear regression line is of the form w1x+w2=y and it is the line that minimizes the sum of the squares of the distance from each data point to the line. So, given n pairs of data (xi, yi), the parameters that we are looking for are w1 and w2 which minimize the error and we can compute the parameter vector w = (w1 , w2)T as the least-squares solution of the following over-determined system Let's use numpy to compute the regression line: from numpy import arange,array,ones,random,linalg from pylab import plot,show xi = arange(0,9) A = array([ xi, ones(9)]) # linearly generated sequence y = [19, 20, 20.5, 21.5, 22, 23, 23, 25.5, 24] w = linalg.lstsq(A.T,y)[0] # obtaining the parameters # plotting the line line = w[0]*xi+w[1] # regression line plot(xi,line,'r-',xi,y,'o') show() We can see the result in the plot below. You can find more about data fitting using numpy in the following posts: Polynomial curve fitting Curve fitting using fmin

One thing that I've wanted to implement for a while now is automatic vhosts on my dev box. The idea is that I want to drop a folder into a directory and have it automatically turned into a vhost for me accessible at http://foldername.dev. It turns out that this isn't nearly as hard as expected which is usually the case with things that I've been putting off! This is how to do it. Apache configuration The Apache magic is in an extension called mod_vhost_alias which you may need to enable in your httpd.conf file. You can then set up the VirtualHost wherever you keep such things. On a stock OS X, the extras/httpd-vhosts.conf file is used. Add the following to the bottom: VirtualDocumentRoot "/www/dev/%1/public" ServerName vhosts.dev ServerAlias *.dev UseCanonicalName Off LogFormat "%V %h %l %u %t \"%r\" %s %b" vcommon ErrorLog "/www/dev/vhosts-error_log" Options Indexes FollowSymLinks MultiViews AllowOverride All Order allow,deny Allow from all In the VirtualHost configuration, I have used the ServerAlias and VirtualDocumentRoot directives to map http://foldername.dev to the directory /www/dev/foldername/public. Hence, any folder that I place in /www/dev will have its own virtual host. Alter these appropriately for your set-up. Don't forget to restart Apache. Unfortunately, the computer hasn't a clue how to handle http://foldername.dev and the obvious solution is to run a local DNS server. Another solution is to use a PAC file. DNS server configuration This is easy enough with dnsmasq. On OS X, use Homebrew to install like this: brew install dnsmasq. On Linux, use your package manager; on Windows, you're own your own! Note that on OS X, you should set it to start up automatically using launchd as noted in the instructions after installation. You also need to copy the configuration file to /etc using: cp /usr/local/Cellar/dnsmasq/2.57/dnsmasq.conf.example /usr/local/etc/dnsmasq.conf (or whatever the latest version number is). on Linux, I would guess that your package manager provides a dnsmasq.conf file in /etc or /etc/dnsmasq. Next, edit dnsmasq.conf file and added the following lines to the bottom: listen-address=127.0.0.1 address=/.dev/127.0.0.1 Add the name server to your network configuration On OS X, Go to System Preferences -> Network -> {Wifi or Ethernet} -> Advanced… -> DNS and click on + button at the bottom of the left hand panel and add 127.0.0.1 to the list of DNS servers. Drag 127.0.0.1 at the top of the list. On Linux, you need to edit /etc/resolv.conf and add nameserver 127.0.0.1 to the file. Alternatively, you can use the appropriate GUI tools... Restart dnsmasq and you should now be able to execute host test.dev on the command line and see 127.0.0.1 as the resultant address. Alternative to DNS server: PAC file Since publishing this article, Chris Morell pointed out that you can also use PAC files rather than install a DNS server. Details are on his blog post. Check it works Create a directory called test in your dev directory. Within test, create public/index.php and within index.php add some code to prove it works. e.g. If you navigate to http://test.dev, you should see "Hello World" displayed. Caveats A couple of caveats: DOCUMENT_ROOT is not /www/dev/test as you'd expect. Instead it is the global document root. If you use mod_rewrite, then you'll need a RewriteBase / in your .htaccess file. Alternatively, you can change the section of your vhost to do the rewriting for you if all your projects are alike. Something like this should work: Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny Allow from all RewriteEngine On RewriteBase / RewriteCond %{REQUEST_FILENAME} -s [OR] RewriteCond %{REQUEST_FILENAME} -l [OR] RewriteCond %{REQUEST_FILENAME} -d RewriteRule ^.*$ - [NC,L] RewriteRule ^.*$ index.php [NC,L] All done That's it. You can now create as many projects as you like without having to worry about setting up new virtual hosts or modifying your host's file!

Since the very early days of Java (and C-like languages overall), the canonical way to start your program has been something like this: public class A { public static void main(String[] args) { new A().run(args); } public void run(String[] args) { // Your application starts here } } If you are still doing this, I’m here to tell you it’s time to stop. Letting go of ‘new’ First, install Guice in your project: com.google.inject guice 3.0 and then, modify your main method as follows: public class A { public static void main(String[] args) { Injector.getInstance(A.class).run(args); } } So, what does this buy you exactly? You will find a lot of articles explaining the various benefits of Guice, such as being able to substitute different environments on the fly, but I’m going to use a different angle in this article. Let’s start by assuming the existence of a Config class that contains various configuration parameters. I’ll just hardcode them for now and use fields to make the class smaller: public class Config { String host = "com.example.com"; int port = 1234; } This class is a singleton, it is instantiated somewhere in your main class and not used anywhere else at the moment. One day, you realize you need this instance in another class which happens to be deep in your runtime hierarchy, which we will call Deep. For example, if you put a break point in the method where you need this config object, your debugger would show you stack frames similar to this: com.example.A.main() com.example.B.f(int, String) com.example.C.g(String) com.example.Deep.h(Foo, int) The easy and wrong way to solve this problem is to make the Config instance static on some class (probably A) and access it directly from Deep. I’m hoping I don’t need to explain why this is a bad idea: not only do you want to avoid using statics, but you also want to make sure that each object is exposed only to objects that need them, and making the Config object static would make your instance visible to your entire code base. Not a good thing. The second thought is to pass the object down the stack, so you modify all the signatures as follows: com.example.A.main() com.example.B.f(int, String, Config) com.example.C.g(String, Config) com.example.Deep.h(Foo, int, Config) This is a bit better since you have severely restricted the exposure of the Config object, but note that you are still making it available to more methods than really need to: B#f and C#g have really nothing to do with this object, and a little sting of discomfort hits you when you start writing the Javadoc: public class C { ... /** * @param config This method doesn't really use this parameter, * it just passes it down so Deep#h can use it. */ public void g(String s, Config config) { Unnecessary exposure is actually not the worst part of this approach, the problem is that it changes all these signatures along the way, which is certainly undesirable in a private API and absolutely devastating in a public API. And of course, it’s absolutely not scalable: if you keep adding a parameter to your method whenever you need access to a certain object, you will soon be dealing with methods that take ten parameters, most of which they just pass down the chain. Here is how we solve this problem with dependency injection (performed by Guice in this example, but this is applicable to any library that implements JSR 330, obviously): public class Deep { @Inject private Config config; and we’re done. That’s it. You don’t need to modify the Config class in any way, nor do you need to make any change in any of the classes that separate Deep from your main class. With this, you have also minimized the exposure of the Config object to just the class that needs it. Injecting right There are various ways you can inject object into your class but I’ll just mention the two that, I think, are the most important. I just showed “field injection” in the previous paragraph, but be aware that you can also prefer to use “constructor injection”: public class Deep { private final Config config; @Inject public Deep(Config config) { this.config = config; } This time, you are adding a parameter to the constructor of your Deep class (which shouldn’t worry you too much since you will never invoke it directly, Guice will) and you assign the parameter to the field in the constructor. The benefit is that you can declare your field final. The downside, obviously, is that this approach is much more verbose. Personally, I see little point in final fields since I have hardly ever encountered a bug that was due to accidentally reassigning a field, so I tend to use field injection whenever I can. Taking it to the next level Obviously, the kind of configuration object I used as an example if not very realistic. Typically, a configuration will not hardcode values like I did and will, instead, read them from some external source. Similarly, you will want to inject objects that can’t necessarily be instantiated so early in the lifecycle of your application, such as servlet contexts, database connections, or implementations of your own interfaces. This topic itself would probably cover several chapters of a book dedicated to dependency injection, so I’ll just summarize it: not all objects can be injected this way, and one benefit of using a dependency injection framework in your code is that it will force you to think about what life cycle category your objects belong to. Having said that, if you want to find out how Guice can inject objects that get created at a later time in your application life cycle, look up the Javadoc for the Provider class. Wrapping up I hope this quick introduction to dependency injection piqued your interest and that you will consider using it in your project since it has so much more to offer than what I described in this post. If you want to learn more, I suggest starting with the excellent Guice documentation.

// code contains regular expression for contact number and email address in python str='[email protected]' match=re.search(r'\w+@\w+',str) #return [email protected] num=555-555-555 match_num=re.search(r'^(\d{3}--\d{3}--\d{4})$',num) #return 555-555-5555

in today’s lesson, we have made animated gears with css3. the result looks very nice. i have used css3 keyframes, animation and transforms (rotate) in order to achieve this result. please pay attention – the current demo works well only in firefox and chrome / safari (webkit). here are the samples and downloadable package: live demo download in package ok, download the example files and let's start coding ! step 1. html as usual, we start with the html. there are easy div elements. index.html step 2. css here are the css styles of our animated gears. css/layout.css /* css3 keyframes */ @-webkit-keyframes ckw { 0% { -moz-transform: rotate(0deg); -webkit-transform: rotate(0deg); } 100% { -moz-transform: rotate(360deg); -webkit-transform: rotate(360deg); } } @-moz-keyframes ckw { 0% { -moz-transform: rotate(0deg); -webkit-transform: rotate(0deg); } 100% { -moz-transform: rotate(360deg); -webkit-transform: rotate(360deg); } } @-webkit-keyframes cckw { 0% { -moz-transform: rotate(360deg); -webkit-transform: rotate(360deg); } 100% { -moz-transform: rotate(0deg); -webkit-transform: rotate(0deg); } } @-moz-keyframes cckw { 0% { -moz-transform: rotate(360deg); -webkit-transform: rotate(360deg); } 100% { -moz-transform: rotate(0deg); -webkit-transform: rotate(0deg); } } /* gears */ .gear { float: none; position: absolute; text-align: center; -moz-animation-timing-function: linear; -moz-animation-iteration-count: infinite; -moz-animation-direction: normal; -moz-animation-delay: 0; -moz-animation-play-state: running; -moz-animation-fill-mode: forwards; -webkit-animation-timing-function: linear; -webkit-animation-iteration-count: infinite; -webkit-animation-direction: normal; -webkit-animation-delay: 0; -webkit-animation-play-state: running; -webkit-animation-fill-mode: forwards; } #gear1 { background: url('../images/g1.png') no-repeat 0 0; height: 85px; left: 31px; top: 45px; width: 85px; -moz-animation-name: ckw; -moz-animation-duration: 10s; -webkit-animation-name: ckw; -webkit-animation-duration: 10s; } #gear2 { background: url('../images/g2.png') no-repeat 0 0; height: 125px; left: 105px; top: 10px; width: 125px; -moz-animation-name: cckw; -moz-animation-duration: 16.84s; -webkit-animation-name: cckw; -webkit-animation-duration: 16.84s; } #gear3 { background: url('../images/g3.png') no-repeat 0 0; height: 103px; left: 149px; top: 118px; width: 103px; -moz-animation-name: ckw; -moz-animation-duration: 13.5s; -webkit-animation-name: ckw; -webkit-animation-duration: 13.5s; } #gear4 { background: url('../images/g4.png') no-repeat 0 0; height: 144px; left: 46px; top: 173px; width: 144px; -moz-animation-name: cckw; -moz-animation-duration: 20.2s; -webkit-animation-name: cckw; -webkit-animation-duration: 20.2s; } #gear5 { background: url('../images/g1.png') no-repeat 0 0; height: 85px; left: 127px; top: 292px; width: 85px; -moz-animation-name: ckw; -moz-animation-duration: 10s; -webkit-animation-name: ckw; -webkit-animation-duration: 10s; } #gear6 { background: url('../images/g2.png') no-repeat 0 0; height: 125px; left: 200px; top: 283px; width: 125px; -moz-animation-name: cckw; -moz-animation-duration: 16.84s; -webkit-animation-name: cckw; -webkit-animation-duration: 16.84s; } #gear7 { background: url('../images/g3.png') no-repeat 0 0; height: 103px; left: 277px; top: 217px; width: 103px; -moz-animation-name: ckw; -moz-animation-duration: 13.5s; -webkit-animation-name: ckw; -webkit-animation-duration: 13.5s; } step 3. images i have used these images: live demo download in package conclusion hope you enjoyed the new tutorial, don’t forget to give thanks and leave a comment good luck!

A few days ago I spoke with my friend, Tomek, about suppressing FindBugs warnings. Here is brief summary of our conversation, which may be interesting for you. There is one simple method for suppressing the FindBugs warnings - usage of edu.umd.cs.findbugs.annotations.SuppressWarnings annotation. Just add it in the place where FindBugs reported problem, and use the appropriate bug code. You should start by adding com.google.code.findbugs:annotations:2.0.0 jar to the project dependencies. Then open the bug reported by FindBugs, and find its code, like in this example: Finally add something like this to the method holding the code marked by FindBugs: That's all, clear the FindBugs markers, and run it again, the problem will not be reported in this place again. Nice! Isn't it? - No, it isn't :( - Why you may ask? - Because there is another annotation in java.lang package with exactly the same name (!), used for suppressing different kind of warnings. Shouldn't it be used instead? - Well ... Another question is if we want to add another jar to the project dependencies just for suppressing FindBugs warnings - thank God the FindBugs authors marked this annotation with retention policy 'CLASS', which means the jar will not be required when running the project (ex. in web application container).

Stamping the version number and the build time of an application in a properties file so that it could be displayed by an application at runtime seemed like it should be a pretty straightforward task, although it took a bit of time to find a solution that didn’t require the timestamp, version, or ant-run plugins. I started with a version.txt file at the default package level in src/main/resources of my project, which looks as follows. version=${pom.version} build.date=${timestamp} By default the Maven resources plug-in will not do text substitution (filtering), so it will need to be enabled within the section of the pom.xml file. src/main/resources true Maven does actually define a ${maven.build.timestamp} property which in theory could have been used in the version.txt file, rather than the ${timestamp} property, but unfortunately a bug within Maven prevents the ${maven.build.timestamp} property from getting passed to the resource filtering mechanism. (issue:http://jira.codehaus.org/browse/MRESOURCES-99). The workaround is to create another property within the pom.xml file and set that new property to the timestamp value, in this case, the property name is “timestamp”, which is used above in the version.txt file. The maven.build.timestamp.format is an optional property for (obviously) defining the timestamp format. ${maven.build.timestamp} yyyy-MM-dd HH:mm Now, when the build is executed we end up with the version number and build time of the application in the version.txt file. Easy! version=1.0.2-SNAPSHOT build.date=2012-03-16 15:42

Within NoSQL, the operations of indexing, fetching and searching for information are intimately tied to the physical storage mechanisms. It is important to remember that rows are stored across hosts, but a single row is stored on a single host. (with replicas) Columns families are stored in sorted order, which makes querying a set of columns efficient (provided you are spanning rows). The Bad : Partitioning One of the tough things to get used to at first is that without any indexes queries that span rows can (very) be bad. Thinking back to our storage model however, that isn't surprising. The strategy that Cassandra uses to distribute the rows across hosts is called Partitioning. Partitioning is the act of carving up the range of rowkeys assigning them into the "token ring", which also assigns responsibility for a segment (i.e. partition) of the rowkey range to each host. You've probably seen this when you initialized your cluster with a "token". The token gives the host a location along the token ring, which assigns responsibility for a section of the token range. Partitioning is the act of mapping the rowkey into the token range. There are two primary partitioners: Random and Order Preserving. They are appropriately named. The RandomPartitioner hashes the rowkeys into tokens. With the RandomPartitioner, the token is a hash of the rowkey. This does a good job of evenly distributing your data across a set of nodes, but makes querying a range of the rowkey space incredibly difficult. From only a "start rowkey" value and an "end rowkey" value, Cassandra can't determine what range of the token space you need. It essentially needs to perform a "table scan" to answer the query, and a "table scan" in Cassandra is bad because it needs to go to each machine (most likely ALL machines if you have a good hash function) to answer the query. Now, at the great cost of even data distribution, you can employ the OrderPreservingPartitioner (OPP). I am *not* down with OPP. The OPP preserves order as it translates rowkeys into tokens. Now, given a start rowkey value and a end rowkey value, Cassandra *can* determine exactly which hosts have the data you are looking for. It computes the start value to a token the end value to a token, and simply selects and returns everything in between. BUT, by preserving order, unless your rowkeys are evenly distributed across the space, your tokens won't be either and you'll get a lopsided cluster, which greatly increases the cost of configuration and administration of the cluster. (not worth it) The Good : Secondary Indexes Cassandra does provide a native indexing mechanism in Secondary Indexes. Secondary Indexes work off of the columns values. You declare a secondary index on a Column Family. Datastax has good documentation on the usage. Under the hood, Cassandra maintains a "hidden column family" as the index. (See Ed Anuff's presentation for specifics) Since Cassandra doesn't maintain column value information in any one node, and secondary indexes are on columns value (rather than rowkeys), a query still needs to be sent to all nodes. Additionally, secondary indexes are not recommended for high-cardinality sets. I haven't looked yet, but I'm assuming this is because of the data model used within the "hidden column family". If the hidden column family stores a row per unique value (with rowkeys as columns), then it would mean scanning the rows to determine if they are within the range in the query. From Ed's presentation: Not recommended for high cardinality values(i.e.timestamps,birthdates,keywords,etc.) Requires at least one equality comparison in a query--not great for less-than/greater-than/range queries Unsorted - results are in token order, not query value order Limited to search on datatypes, Cassandra natively understands With all that said, secondary indexes work out of the box and we've had good success using them on simple values. The Ugly : Do-It-Yourself (DIY) / Wide-Rows Now, beauty is in the eye of the beholder. One of the beautiful things about NoSQL is the simplicity. The constructs are simple: Keyspaces, Column Families, Rows and Columns. Keeping it simple however means sometimes you need to take things into your own hands. This is the case with wide-row indexes. Utilizing Cassandra's storage model, its easy to build your own indexes where each row-key becomes a column in the index. This is sometimes hard to get your head around, but lets imagine we have a case whereby we want to select all users in a zip code. The main users column family is keyed on userid, zip code is a column on each user row. We could use secondary indexes, but there are quite a few zip codes. Instead we could maintain a column family with a single row called "idx_zipcode". We could then write columns into this row of the form "zipcode_userid". Since the columns are stored in sorted order, it is fast to query for all columns that start with "18964" (e.g. we could use 18964_ and 18964_ZZZZZZ as start and end values). One obvious downside of this approach is that rows are self-contained on a host. (again except for replicas) This means that all queries are going to hit a single node. I haven't yet found a good answer for this. Additionally, and IMHO, the ugliest part of DIY wide-row indexing is from a client perspective. In our implementation, we've done our best to be language agnostic on the client-side, allowing people to pick the best tool for the job to interact with the data in Cassandra. With that mentality, the DIY indexes present some trouble. Wide-rows often use composite keys (imagine if you had an idx_state_zip, which would allow you to query by state then zip). Although there is "native" support for composite keys, all of the client libraries implement their own version of them (Hector, Astyanax, and Thrift). This means that client needing to query data needs to have the added logic to first query the index, and additionally all clients need to construct the composite key in the same manner. Making It Better... For this very reason, we've decided to release two open source projects that help push this logic to the server-side. The first project is Cassandra-Triggers. This allows you to attached asynchronous activities to writes in Cassandra. (one such activity could be indexing) We've also released Cassandra-Indexing. This is hot off the presses and is still in its infancy (e.g. it only supports UT8Types in the index), but the intent is to provide a generic server-side mechanism that indexes data as its written to Cassandra. Employing the same server-side technique we used in Cassandra-Indexing, you simply configure the columns you want indexed, and the AOP code does the rest as you write to the target CF. As always, questions, comments and thoughts are welcome. (especially if I'm off-base somewhere)

i want to like javafx, really i do. the return of the applet reminds me of the 90s which is nice. i also like the idea of being able to drag an applet into windows, ubuntu , and mac to run it as a desktop application. it's a whole new take on their "write once, run anywhere" promise and breathing some life into a platform that needs it. java used to be so trendy and cool, it was the "ruby on rails of the 90s" now it's seemingly destined to be the "cobol of the 20s". if javafx lives up to its promise it could turn things around. well, i guess android is technically leading a java revival today unless oracle's lawsuit forces google to move to a different language. so far though i've been a little disappointed with javafx. it's like an el camino, a strange combination of awt and swing that doesn't quite feel natural. i'm going to keep trying it anyway and hope that one day it catches up to c# 1.0. look, i know this sounds terribly cynical so far but you have to believe me when i say i'm trying to like it. the reality is, even if javafx is a little clunky now it's still a considerable improvement over swing. over the next few months i'm going to upgrade all my ugly swing applications to javafx. the first one is something called debigulator . it's a batch archive program that i wrote for myself but has been downloaded more than i expected. it's also one of the ugliest programs ever created. just look at this monstrosity: besides being unattractive it also doesn't resize well. javafx addresses both of those so i'm porting it. the first thing to go is that awful file browser in the top left region, i'm embarrassed to look at it. i think i'll replace it with a simple treeview. to create a treeview we first have to create a treeitem subclass to store in the tree. the api documentation for the javafx treeitem class includes a partial implementation of a file browser. i looked at it but went a different direction because it recursively populates the entire tree up front and doesn't deal with things like folder & file icons. instead i wanted to dynamically populate a node when it's expanded. the treeitem also needs to store the path to the file represented by each item but only show the folder or file name. alright, let's get our treeitem implementation started. the constructor and class members look a little something like: public class filepathtreeitem extends treeitem{ public static image foldercollapseimage=new image(classloader.getsystemresourceasstream("com/huguesjohnson/javafxfilebrowsedemo/folder.png")); public static image folderexpandimage=new image(classloader.getsystemresourceasstream("com/huguesjohnson/javafxfilebrowsedemo/folder-open.png")); public static image fileimage=new image(classloader.getsystemresourceasstream("com/huguesjohnson/javafxfilebrowsedemo/text-x-generic.png")); //this stores the full path to the file or directory private string fullpath; public string getfullpath(){return(this.fullpath);} private boolean isdirectory; public boolean isdirectory(){return(this.isdirectory);} public filepathtreeitem(path file){ super(file.tostring()); this.fullpath=file.tostring(); next we want to set the icon, full path, and isdirectory members. this would be a good time to mention that all the icons in this demo come from the tango library . //test if this is a directory and set the icon if(files.isdirectory(file)){ this.isdirectory=true; this.setgraphic(new imageview(foldercollapseimage)); }else{ this.isdirectory=false; this.setgraphic(new imageview(fileimage)); //if you want different icons for different file types this is where you'd do it } //set the value if(!fullpath.endswith(file.separator)){ //set the value (which is what is displayed in the tree) string value=file.tostring(); int indexof=value.lastindexof(file.separator); if(indexof>0){ this.setvalue(value.substring(indexof+1)); }else{ this.setvalue(value); } } now let's add the event handler for the node expanded event. that check for source.isexpanded() sure seems unnecessary. man that was a fun piece of unexpected behavior to track down. this.addeventhandler(treeitem.branchexpandedevent(),new eventhandler(){ @override public void handle(event e){ filepathtreeitem source=(filepathtreeitem)e.getsource(); if(source.isdirectory()&&source.isexpanded()){ imageview iv=(imageview)source.getgraphic(); iv.setimage(folderexpandimage); } try{ if(source.getchildren().isempty()){ path path=paths.get(source.getfullpath()); basicfileattributes attribs=files.readattributes(path,basicfileattributes.class); if(attribs.isdirectory()){ directorystream dir=files.newdirectorystream(path); for(path file:dir){ filepathtreeitem treenode=new filepathtreeitem(file); source.getchildren().add(treenode); } } }else{ //if you want to implement rescanning a directory for changes this would be the place to do it } }catch(ioexception x){ x.printstacktrace(); } } }); we'll wrap up this treeitem implementation with an handler for the node collapsed event. again the source.isexpanded() check really shouldn't be needed but just go ahead and remove it to see the goofiness that follows. this.addeventhandler(treeitem.branchcollapsedevent(),new eventhandler(){ @override public void handle(event e){ filepathtreeitem source=(filepathtreeitem)e.getsource(); if(source.isdirectory()&&!source.isexpanded()){ imageview iv=(imageview)source.getgraphic(); iv.setimage(foldercollapseimage); } } }); now we can go to work on the main program. here's all the basic stuff. public class javafxfilebrowsedemoapp extends application{ private treeview treeview; public static void main(string[] args){ launch(args); } @override public void start(stage primarystage){ //create tree pane vbox treebox=new vbox(); treebox.setpadding(new insets(10,10,10,10)); treebox.setspacing(10); now it's time to start populating the tree. we'll use the computer name as the root node. although i might go back and hide the root node since it's kind of pointless for this application. it's really just showing off how to get the name from the inetaddress class which you either already knew or didn't care about. //setup the file browser root string hostname="computer"; try{hostname=inetaddress.getlocalhost().gethostname();}catch(unknownhostexception x){} treeitem rootnode=new treeitem<>(hostname,new imageview(new image(classloader.getsystemresourceasstream("com/huguesjohnson/javafxfilebrowsedemo/computer.png")))); one nifty addition to jdk7 is the ability to list all the drives on the system. that comes in handy for the next step where we need to add all the drives under the root node. iterable rootdirectories=filesystems.getdefault().getrootdirectories(); for(path name:rootdirectories){ filepathtreeitem treenode=new filepathtreeitem(name); rootnode.getchildren().add(treenode); } rootnode.setexpanded(true); all that's left is to add the treeview to the window and show it. //create the tree view treeview=new treeview<>(rootnode); //add everything to the tree pane treebox.getchildren().addall(new label("file browser"),treeview); vbox.setvgrow(treeview,priority.always); //setup and show the window primarystage.settitle("javafx file browse demo"); stackpane root=new stackpane(); root.getchildren().addall(treebox); primarystage.setscene(new scene(root,400,300)); primarystage.show(); here's what the final product looks like, much cleaner than the awful swing version and less than half the code:

The latest incarnation of the Spring Framework (3.1) brought an interesting feature called Flash Attributes. It is a remedy for the problem mentioned a long time ago, in one of my posts: Spring MVC - Session Attributes handling. This problem can be described in few words: if we want to pass the attributes via redirect between two controllers, we cannot use request attributes (they will not survive the redirect), and we cannot use Spring's @SessionAttributes (because of the way Spring handles it), only an ordinary HttpSession can be used, which is not very convenient. Below you will find an example of Flash Attributes usage, before you start reviewing it, read Using flash attributes section of Spring documentation. Suppose that we have two controllers: AController and BController, first one will prepare some data and pass to the second using Flash Attributes after the form submission. On the AController we will have something like this: @RequestMapping(method = RequestMethod.POST) public String handleFormSubmission(..., final RedirectAttributes redirectAttrs) { ... redirectAttrs.addFlashAttribute("AttributeName", value); return "redirect:to_some_url_handled_by_BController"; } When the form will be submitted, attribute value will be stored as Flash Attribute named "AttributeName", and thanks to the Spring, will be passed to BController, where it can be used for example in following way: @Controller ... @SessionAttributes("AttributeName") public class SearchCriteriaHandler { ... @RequestMapping(method = RequestMethod.GET) public void handleGetRequest(@ModelAttribute("AttributeName") final SomeType value) { ... } ... } Before your handler method will be called, Spring Framework will populate the Model with the available Flash Attributes - at this point value passed from AController will become a model attribute for the BController. Note, that because we also defined this attribute as the Session Attribute, it will be automatically stored for future usage within this controller, after the GET request handling. Let me say that I was waiting for this feature for the long time, ;)