In my two immediately previous posts, I looked at reducing the number of parameters required for a constructor or method invocation via custom types and parameter objects. In this post, I look at use of thebuilder pattern to reduce the number of parameters required for a constructor with some discussion on how this pattern can even help with non-constructor methods that take too many parameters. In the Second Edition of Effective Java, Josh Bloch introduces use of the builder pattern in Item #2 for dealing with constructors that require too many parameters. Bloch not only demonstrates how to use the Builder, but explains it advantages over constructors accepting a large number of parameters. I will get to those advantages at the end of this post, but think it's important to point out that Bloch has devoted an entire item in his book to this practice. To illustrate the advantages of this approach, I'll use the following example Person class. It doesn't have all the methods I would typically add to such a class because I want to focus on its construction. Person.java (without Builder Pattern) package dustin.examples; /** * Person class used as part of too many parameters demonstration. * * @author Dustin */ public class Person { private final String lastName; private final String firstName; private final String middleName; private final String salutation; private final String suffix; private final String streetAddress; private final String city; private final String state; private final boolean isFemale; private final boolean isEmployed; private final boolean isHomewOwner; public Person( final String newLastName, final String newFirstName, final String newMiddleName, final String newSalutation, final String newSuffix, final String newStreetAddress, final String newCity, final String newState, final boolean newIsFemale, final boolean newIsEmployed, final boolean newIsHomeOwner) { this.lastName = newLastName; this.firstName = newFirstName; this.middleName = newMiddleName; this.salutation = newSalutation; this.suffix = newSuffix; this.streetAddress = newStreetAddress; this.city = newCity; this.state = newState; this.isFemale = newIsFemale; this.isEmployed = newIsEmployed; this.isHomewOwner = newIsHomeOwner; } } This class's constructor works, but it is difficult for client code to use properly. The Builder pattern can be used to make the constructor easier to use. NetBeans will refactor this for me as I have written about previously. An example of the refactored code is shown next (NetBeans does this by creating all new Builder class). PersonBuilder.java package dustin.examples; public class PersonBuilder { private String newLastName; private String newFirstName; private String newMiddleName; private String newSalutation; private String newSuffix; private String newStreetAddress; private String newCity; private String newState; private boolean newIsFemale; private boolean newIsEmployed; private boolean newIsHomeOwner; public PersonBuilder() { } public PersonBuilder setNewLastName(String newLastName) { this.newLastName = newLastName; return this; } public PersonBuilder setNewFirstName(String newFirstName) { this.newFirstName = newFirstName; return this; } public PersonBuilder setNewMiddleName(String newMiddleName) { this.newMiddleName = newMiddleName; return this; } public PersonBuilder setNewSalutation(String newSalutation) { this.newSalutation = newSalutation; return this; } public PersonBuilder setNewSuffix(String newSuffix) { this.newSuffix = newSuffix; return this; } public PersonBuilder setNewStreetAddress(String newStreetAddress) { this.newStreetAddress = newStreetAddress; return this; } public PersonBuilder setNewCity(String newCity) { this.newCity = newCity; return this; } public PersonBuilder setNewState(String newState) { this.newState = newState; return this; } public PersonBuilder setNewIsFemale(boolean newIsFemale) { this.newIsFemale = newIsFemale; return this; } public PersonBuilder setNewIsEmployed(boolean newIsEmployed) { this.newIsEmployed = newIsEmployed; return this; } public PersonBuilder setNewIsHomeOwner(boolean newIsHomeOwner) { this.newIsHomeOwner = newIsHomeOwner; return this; } public Person createPerson() { return new Person(newLastName, newFirstName, newMiddleName, newSalutation, newSuffix, newStreetAddress, newCity, newState, newIsFemale, newIsEmployed, newIsHomeOwner); } } I prefer to have my Builder as a nested class inside the class whose object it builds, but the NetBeans automatic generation of a standalone Builder is very easy to use. Another difference between the NetBeans-generated Builder and the Builders I like to write is that my preferred Builder implementations have required fields provided in the Builder's constructor rather than provide a no-arguments constructor. The next code listing shows my Person class from above with a Builder added into it as a nested class. Person.java with Nested Person.Builder package dustin.examples; /** * Person class used as part of too many parameters demonstration. * * @author Dustin */ public class Person { private final String lastName; private final String firstName; private final String middleName; private final String salutation; private final String suffix; private final String streetAddress; private final String city; private final String state; private final boolean isFemale; private final boolean isEmployed; private final boolean isHomewOwner; public Person( final String newLastName, final String newFirstName, final String newMiddleName, final String newSalutation, final String newSuffix, final String newStreetAddress, final String newCity, final String newState, final boolean newIsFemale, final boolean newIsEmployed, final boolean newIsHomeOwner) { this.lastName = newLastName; this.firstName = newFirstName; this.middleName = newMiddleName; this.salutation = newSalutation; this.suffix = newSuffix; this.streetAddress = newStreetAddress; this.city = newCity; this.state = newState; this.isFemale = newIsFemale; this.isEmployed = newIsEmployed; this.isHomewOwner = newIsHomeOwner; } public static class PersonBuilder { private String nestedLastName; private String nestedFirstName; private String nestedMiddleName; private String nestedSalutation; private String nestedSuffix; private String nestedStreetAddress; private String nestedCity; private String nestedState; private boolean nestedIsFemale; private boolean nestedIsEmployed; private boolean nestedIsHomeOwner; public PersonBuilder( final String newFirstName, final String newCity, final String newState) { this.nestedFirstName = newFirstName; this.nestedCity = newCity; this.nestedState = newState; } public PersonBuilder lastName(String newLastName) { this.nestedLastName = newLastName; return this; } public PersonBuilder firstName(String newFirstName) { this.nestedFirstName = newFirstName; return this; } public PersonBuilder middleName(String newMiddleName) { this.nestedMiddleName = newMiddleName; return this; } public PersonBuilder salutation(String newSalutation) { this.nestedSalutation = newSalutation; return this; } public PersonBuilder suffix(String newSuffix) { this.nestedSuffix = newSuffix; return this; } public PersonBuilder streetAddress(String newStreetAddress) { this.nestedStreetAddress = newStreetAddress; return this; } public PersonBuilder city(String newCity) { this.nestedCity = newCity; return this; } public PersonBuilder state(String newState) { this.nestedState = newState; return this; } public PersonBuilder isFemale(boolean newIsFemale) { this.nestedIsFemale = newIsFemale; return this; } public PersonBuilder isEmployed(boolean newIsEmployed) { this.nestedIsEmployed = newIsEmployed; return this; } public PersonBuilder isHomeOwner(boolean newIsHomeOwner) { this.nestedIsHomeOwner = newIsHomeOwner; return this; } public Person createPerson() { return new Person( nestedLastName, nestedFirstName, nestedMiddleName, nestedSalutation, nestedSuffix, nestedStreetAddress, nestedCity, nestedState, nestedIsFemale, nestedIsEmployed, nestedIsHomeOwner); } } } The Builder can be even nicer when enhanced through use of custom types and parameters objects as outlined in my first two posts on the "too many parameters" problem. This is shown in the next code listing. Person.java with Nested Builder, Custom Types, and Parameters Object package dustin.examples; /** * Person class used as part of too many parameters demonstration. * * @author Dustin */ public class Person { private final FullName name; private final Address address; private final Gender gender; private final EmploymentStatus employment; private final HomeownerStatus homeOwnerStatus; /** * Parameterized constructor can be private because only my internal builder * needs to call me to provide an instance to clients. * * @param newName Name of this person. * @param newAddress Address of this person. * @param newGender Gender of this person. * @param newEmployment Employment status of this person. * @param newHomeOwner Home ownership status of this person. */ private Person( final FullName newName, final Address newAddress, final Gender newGender, final EmploymentStatus newEmployment, final HomeownerStatus newHomeOwner) { this.name = newName; this.address = newAddress; this.gender = newGender; this.employment = newEmployment; this.homeOwnerStatus = newHomeOwner; } public FullName getName() { return this.name; } public Address getAddress() { return this.address; } public Gender getGender() { return this.gender; } public EmploymentStatus getEmployment() { return this.employment; } public HomeownerStatus getHomeOwnerStatus() { return this.homeOwnerStatus; } /** * Builder class as outlined in the Second Edition of Joshua Bloch's * Effective Java that is used to build a {@link Person} instance. */ public static class PersonBuilder { private FullName nestedName; private Address nestedAddress; private Gender nestedGender; private EmploymentStatus nestedEmploymentStatus; private HomeownerStatus nestedHomeOwnerStatus; public PersonBuilder( final FullName newFullName, final Address newAddress) { this.nestedName = newFullName; this.nestedAddress = newAddress; } public PersonBuilder name(final FullName newName) { this.nestedName = newName; return this; } public PersonBuilder address(final Address newAddress) { this.nestedAddress = newAddress; return this; } public PersonBuilder gender(final Gender newGender) { this.nestedGender = newGender; return this; } public PersonBuilder employment(final EmploymentStatus newEmploymentStatus) { this.nestedEmploymentStatus = newEmploymentStatus; return this; } public PersonBuilder homeOwner(final HomeownerStatus newHomeOwnerStatus) { this.nestedHomeOwnerStatus = newHomeOwnerStatus; return this; } public Person createPerson() { return new Person( nestedName, nestedAddress, nestedGender, nestedEmploymentStatus, nestedHomeOwnerStatus); } } } The last couple of code listings show how a Builder is typically used - to construct an object. Indeed, the item on the builder (Item #2) in Joshua Bloch's Second Edition of Effective Java is in the chapter on creating (and destroying) object. However, the builder can help indirectly with non-constructor methods by allowing an easier way to build parameters objects that are passed to methods. For example, in the last code listing, the methods have some parameters objects (FullName and Address) passed to them. It can be tedious for clients to have to construct these parameters objects and the builder can be used to make that process less tedious. So, although the builder is used for construction in each case, it indirectly benefits non-constructor methods by allowing for easier use of the parameters objects that reduce a method's argument count. The new definitions of the FullName and Address classes to be used as parameters objects and using the Builder themselves are shown next. FullName.java with Builder package dustin.examples; /** * Full name of a person. * * @author Dustin */ public final class FullName { private final Name lastName; private final Name firstName; private final Name middleName; private final Salutation salutation; private final Suffix suffix; private FullName( final Name newLastName, final Name newFirstName, final Name newMiddleName, final Salutation newSalutation, final Suffix newSuffix) { this.lastName = newLastName; this.firstName = newFirstName; this.middleName = newMiddleName; this.salutation = newSalutation; this.suffix = newSuffix; } public Name getLastName() { return this.lastName; } public Name getFirstName() { return this.firstName; } public Name getMiddleName() { return this.middleName; } public Salutation getSalutation() { return this.salutation; } public Suffix getSuffix() { return this.suffix; } @Override public String toString() { return this.salutation + " " + this.firstName + " " + this.middleName + this.lastName + ", " + this.suffix; } public static class FullNameBuilder { private final Name nestedLastName; private final Name nestedFirstName; private Name nestedMiddleName; private Salutation nestedSalutation; private Suffix nestedSuffix; public FullNameBuilder( final Name newLastName, final Name newFirstName) { this.nestedLastName = newLastName; this.nestedFirstName = newFirstName; } public FullNameBuilder middleName(final Name newMiddleName) { this.nestedMiddleName = newMiddleName; return this; } public FullNameBuilder salutation(final Salutation newSalutation) { this.nestedSalutation = newSalutation; return this; } public FullNameBuilder suffix(final Suffix newSuffix) { this.nestedSuffix = newSuffix; return this; } public FullName createFullName() { return new FullName( nestedLastName, nestedFirstName, nestedMiddleName, nestedSalutation, nestedSuffix); } } } Address.java with Builder package dustin.examples; /** * Representation of a United States address. * * @author Dustin */ public final class Address { private final StreetAddress streetAddress; private final City city; private final State state; private Address(final StreetAddress newStreetAddress, final City newCity, final State newState) { this.streetAddress = newStreetAddress; this.city = newCity; this.state = newState; } public StreetAddress getStreetAddress() { return this.streetAddress; } public City getCity() { return this.city; } public State getState() { return this.state; } @Override public String toString() { return this.streetAddress + ", " + this.city + ", " + this.state; } public static class AddressBuilder { private StreetAddress nestedStreetAddress; private final City nestedCity; private final State nestedState; public AddressBuilder(final City newCity, final State newState) { this.nestedCity = newCity; this.nestedState = newState; } public AddressBuilder streetAddress(final StreetAddress newStreetAddress) { this.nestedStreetAddress = newStreetAddress; return this; } public Address createAddress() { return new Address(nestedStreetAddress, nestedCity, nestedState); } } } With the above builders included in the classes, a Person instance can be created as shown in the next code listing. A more traditional instantiation of a Person instance is shown after that for comparison. Two Examples of Client Code Instantiating a Person with Builders final Person person1 = new Person.PersonBuilder( new FullName.FullNameBuilder( new Name("Dynamite"), new Name("Napoleon")).createFullName(), new Address.AddressBuilder( new City("Preston"), State.ID).createAddress()).createPerson(); final Person person2 = new Person.PersonBuilder( new FullName.FullNameBuilder( new Name("Coltrane"), new Name("Rosco")).middleName(new Name("Purvis")).createFullName(), new Address.AddressBuilder( new City("Hazzard"), State.GA).createAddress()) .gender(Gender.MALE).employment(EmploymentStatus.EMPLOYED).createPerson(); Instantiating a Person Without a Builder final person = new Person("Coltrane", "Rosco", "Purvis", null, "Hazzard", "Georgia", false, true, true); As the previous code snippets show, the client code for calling a traditional Java constructor is far less readable and far easier to mess up than use of the builder classes. The variety of the same types (strings and booleans) and the necessity to place nulls in the constructor call for optional attributes make provide many ways for this approach to end badly. Benefits and Advantages There is a considerable cost to the Builder pattern in that one must essentially double the number of lines of code each attribute and for setting those attributes. This price pays off, however, when the client code benefits greatly in terms of usability and readability. The parameters to the constructor are reduced and are provided in highly readable method calls. Another advantage of the Builder approach is the ability to acquire an object in a single statement and state without the object in multiple states problem presented by using "set" methods. I am increasingly appreciating the value of immutability in a multi-core world and the Builder pattern is perfectly suited for an immutable class when that class features a large number of attributes. I also like that there is no need to pass in null for optional parameters to the constructor. The Builder pattern not only makes the code more readable, but makes it even easier to apply an IDE's code completion feature. Further benefits of the Builder pattern when used with constructors are outlined in Item #2 of the Second Edition of Effective Java. Costs and Disadvantages As shown and mentioned above, the number of lines of code of a given class must be essentially doubled for "set" methods with the builder approach. Furthermore, although client code is more readable, the client code is also more verbose. I consider the benefit of greater readability worth the cost as the number of arguments increase or as more arguments share the same type or as the number of optional arguments increase. More lines of code in the class with the builder sometimes mean that developers may forget to add support for a new attribute to the builder when they add that attribute to the main class. To try to help with this, I like to nest my builders inside the class that they build so that it's more obvious to the developer that there is a relevant builder that needs to be similarly updated. Although there is still risk of the developer forgetting to add support for a new attribute to the builder, this is really no different than the risk of forgetting to add a new attribute to a class's toString(), equals(Object), hashCode() or other methods often based on all attributes of a class. In my implementation of the Builder, I made the client pass required attributes into the builder's constructor rather than via "set" methods. The advantage of this is that the object is always instantiated in a "complete" state rather than sitting in an incomplete state until the developer calls (if ever calls) the appropriate "set" method to set additional fields. This is necessary to enjoy the benefits of immutability. However, a minor disadvantage of that approach is that I don't get the readability advantages of methods named for the field I am setting. The Builder, as its name suggests, is really only an alternative to constructors and not directly used to reduce the number of non-constructor method parameters. However, the builder can be used in conjunction with parameters objects to reduce the number of non-constructor method arguments. Further arguments against use of the Builder for object construction can be found in a comment on the A dive into the Builder pattern post. Conclusion I really like the Builder pattern for constructing objects when I have a lot of parameters, especially if many of these parameters are null and when many of them share the same data type. A developer might feel that the extra code to implement a Builder might not justify its benefits for a small number of parameters, especially if the few parameters are required and of different types. In such cases, it might be considered desirable to use traditional constructors or, if immutability is not desired, use a no-argument constructor and require the client to know to call the necessary "set" methods.

simple talk - How to get SQL Railroad Diagrams from MSDN BNF syntax notation. On SQL Server Books-On-Line, in the Transact-SQL Reference (database Engine), every SQL Statement has its syntax represented in ‘Backus–Naur Form’ notation (BNF) syntax. For a programmer in a hurry, this should be ideal because It is the only quick way to understand and appreciate all the permutations of the syntax. It is a great feature once you get your eye in. It isn’t the only way to get the information; You can, of course, reverse-engineer an understanding of the syntax from the examples, but your understanding won’t be complete, and you’ll have wasted time doing it. BNF is a good start in representing the syntax: Oracle and SQLite go one step further, and have proper railroad diagrams for their syntax, which is a far more accessible way of doing it. There are three problems with the BNF on MSDN. Firstly, it is isn’t a standard version of BNF, but an ancient fork from EBNF, inherited from Sybase. Secondly, it is excruciatingly difficult to understand, and thirdly it has a number of syntactic and semantic errors. The page describing DML triggers, for example, currently has the absurd BNF error that makes it state that all statements in the body of the trigger must be separated by commas. There are a few other detail problems too. Here is the offending syntax for a DML trigger, pasted from MSDN. ... I’ve been trying to create railroad diagrams for all the important SQL Server SQL statements, as good as you’d find for Oracle, and have so far published the CREATE TABLE and ALTER TABLE railroad diagrams based on the BNF. Although I’ve been aware of them, I’ve never realised until recently how many errors there are. Then, Colin Daley created a translator for the SQL Server dialect of BNF which outputs standard EBNF notation used by the W3C. The example MSDN BNF for the trigger would be rendered as … ... Colin’s intention was to allow anyone to paste SQL Server’s BNF notation into his website-based parser, and from this generate classic railroad diagrams via Gunther Rademacher's Railroad Diagram Generator. Colin's application does this for you: you're not aware that you are moving to a different site. Because Colin's 'translator' it is a parser, it will pick up syntax errors. Once you’ve fixed the syntax errors, you will get the syntax in the form of a human-readable railroad diagram and, in this form, the semantic mistakes become flamingly obvious. Gunter’s Railroad Diagram Generator is brilliant. To be able, after correcting the MSDN dialect of BNF, to generate a standard EBNF, and from thence to create railroad diagrams for SQL Server’s syntax that are as good as Oracle’s, is a great boon, and many thanks to Colin for the idea. Here is the result of the W3C EBNF from Colin’s application then being run through the Railroad diagram generator. Now that’s much better, you’ll agree. This is pretty easy to understand, and at this point any error is immediately obvious. This should be seriously useful, and it is to me. However there is that snag. The BNF is generally incorrect, and you can’t expect the average visitor to mess about with it. The answer is, of course, to correct the BNF on MSDN and maybe even add railroad diagrams for the syntax. Stop giggling! I agree it won’t happen. In the meantime, we need to collaboratively store and publish these corrected syntaxes ourselves as we do them. How? GitHub? SQL Server Central? Simple-Talk? What should those of us who use the system do with our corrected EBNF so that anyone can use them without hassle? Grammar Translator If you are familiar with the Grammar Translator, go ahead and create railroad diagrams from the Transact-SQL Reference. Otherwise, please see the FAQ. In particular, be sure to try thetutorial. Welcome to Railroad Diagram Generator! This is a tool for creating syntax diagrams, also known as railroad diagrams, from context-free grammars specified in EBNF. Syntax diagrams have been used for decades now, so the concept is well-known, and some tools for diagram generation are in existence. The features of this one are usage of the W3C's EBNF notation, web-scraping of grammars from W3C specifications, online editing of grammars, diagram presentation in SVG, and it was completely written in web languages (XQuery, XHTML, CSS, JavaScript). There's nothing like a diagram to help grok something (and the MSDN BNF SQL stuff really makes my brain hurt...)

I’ve been doing some work that involves reading in CSV files, for which I’ve been using OpenCSV, and my initial approach was to read through the file line by line, parse the contents, and save it into a list of maps. This works when the contents of the file fit into memory, but is problematic for larger files where I needed to stream the file and process each line individually, rather than all of them after the file was loaded. I initially wrote a variation on totallylazy’s Strings#lines to do this, and while I was able to stream the file, I made a mistake somewhere which meant the number of maps on the heap was always increasing. After spending a few hours trying to fix this, Michael suggested that it’d be easier to use an iterator instead, and I ended up with the following code: public class ParseCSVFile { public static void main(String[] args) throws IOException { final CSVReader csvReader = new CSVReader( new BufferedReader( new FileReader( "/path/to/file.csv" ) ), '\t' ); final String[] fields = csvReader.readNext(); Iterator>() lazilyLoadedFile = return new Iterator>() { String[] data = csvReader.readNext(); @Override public boolean hasNext() { return data != null; } @Override public Map next() { final Map properties = new HashMap(); for ( int i = 0; i < data.length; i++ ) { properties.put(fields[i], data[i]); } try { data = csvReader.readNext(); } catch ( IOException e ) { data = null; } return properties; } @Override public void remove() { throw new UnsupportedOperationException(); } }; } } Although this code works, it’s not the most readable function I’ve ever written, so any suggestions on how to do this in a cleaner way are welcome.

With these changes, we can access the REST API equally well fromhttp://:9999 and https://:9998.

I have recently started the Coursera Functional Programming with Scala course (taught by Martin Odersky - the creator of Scala) - which is actually serving as an introduction to both FP and Scala at the same time having done neither before. The course itself is great, however, trying to watch the videos and take in both the new Scala syntax and the FP concepts at the same time can take a bit of effort. I wanted to work through some of the core FP concepts in a more familiar context, so am going to apply some of the lessons/principles/exercises in Groovy. Functions: If you have done any Groovy programming then you will have come across Groovy functions/closures. As Groovy is dynamically typed (compared to Scala's static typing), you can play it fairly fast and loose. For example, if we take the square root function that is demonstrated in the Scala course, it is defined as follows: def sqrt(x: Double): Double = ... As you can see, Scala expects the values to be typed (aside, you don't actually always need to provide a return type in Scala). But in the Groovy function it is: def sqrt = {x-> ... } A groovy function can be defined and assigned to any variable, thereby allowing it to be passed around as a first class object. If we look at the complete solution for calculating the square root of a number (using Newton's method - To compute the square root of "x" we start with an estimate of the square root, "y" and continue to improve the the guess by taking the mean of x and x/y ) def sqrtIter(guess: Double, x: Double): Double = if (isGoodEnough(guess, x)) guess else sqrtIter(improve(guess, x), x) def improve(guess: Double, x: Double) = (guess + x / guess) / 2 def isGoodEnough(guess: Double, x: Double) = abs(guess * guess - x) < 0.001 def sqrt(x: Double) = srqtIter(1.0, x) So that's in Scala, if we try Groovy we will see we can achieve pretty much the same thing easily: //Improve the guess using Newton's method def improve = { guess, x -> (guess + x / guess) / 2 } //Check if our guess is good enough, within a chosen threshold def isGoodEnough = {guess, x -> abs(guess * guess - x) < 0.001 } //iterate over guesses until our guess is good enough def sqrtIter = { guess, x -> if (isGoodEnough(guess, x)) guess else sqrtIter(improve(guess, x), x) } //wrap everythin in the square root function call def sqrt = {x -> srqtIter(1.0, x)} Recursion (and tail-recursion): As FP avoids having mutable state, the most common approach to solve problems is to break the problem down in to simple functions and call them recursively - This avoids having to maintain state whilst iterating through loops, and each function call is given its input and produces an output. If we again consider an example from the Scala course, with the example of a simple function that calculates the factorial for a given number. def factorial ={ n -> if (n == 0) 1 else n * factorial(n - 1) } factorial(4) This simple function recursively calculates the factorial, continuing to call itself until all numbers to zero have been considered. As you can see, there is no mutable state - every call to the factorial function simply takes the input and returns an output (the value of n is never changed or re-assigned, n is simply used to calculate output values) There is a problem here, and that is as soon as you attempt to calculate the factorial of a significantly large enough number you will encounter a StackOverflow exception - this is because in the JVM every time a function is called, a frame is added to the stack, so working recursively its pretty easy to hit upon the limit of the stack and encounter this problem. The common way to solve this is by using Tail-Call recursion. This trick is simply to have the last code that is evaluated in the function to be the recursive call - normally in FP languages the compiler/interpreter will recognise this pattern and under the hood, it will really just run the code as a loop (e.g. if we know the very last piece of code in the block of code is calling itself, its really not that different to just having the block of code/function inside a loop construct) In the previous factorial example, it might look like the last code to be executed is the recursive callfactorial(n-1) - however, the value of that call is actually returned to the function and THENmultiplied by n - so actually the last piece of code to be evaluated in the function call is actually n * return value of factorial(n-1). Let's have a look at re-writing the function so it is tail-recursive. def factorial ={ n, accumulator=1 -> if (n == 1) accumulator else factorial(n-1, n*accumulator) } factorial(4) Now, using an accumulator, the last code to be evaluated in the function is our recursive function call. In most FP languages, including Scala, this is enough - however, the JVM doesn't automatically support tail-call recursion, so you actually need to use a rather clunkier approach in Groovy: def factorial ={ n, accumulator=1 -> if (n == 1) accumulator else factorial.trampoline(n-1, n*accumulator) }.trampoline() factorial(4) The use of the trampoline() method means that the function will now be called using tail-call recursion, so there should never be a StackOverflow exception. It's not as nice as in Scala or other languages, but the support is there so we can continue. Currying: This is like function composition - the idea being you take a generic function, and then you curry it with some value to make a more specific application of the function. For example, if we look at a function that given values x and y, it returns z which is the value x percent of y (e.g. given x=10, y=100, it returns the 10 percent of 100, z=10) def percentage = { percentage, x -> x/100 * percentage } The above simple function is a generic mechanism to get a percentage value of another, but if we consider that we wanted a common application of this function was to always calculate 10% of a given value - rather than write a slightly modified version of the function we can simply curry the function as follows: def tenPercent = percentage.curry(10) Now, if the function tenPercent(x) is called, it uses the original percentage() function, but curries the value 10 as the first argument. (If you need to curry other argument positions you can also use the rcurry() function to curry the right most argument, or ncurry() which also takes an argument position - check the Groovy docs on currying for more info) Immutability: Immutability is partially supported in Java normally with use of the final keyword (meaning variables can't be changed after being initially set on object instantiation). Groovy also provides a quick and easy @Immutable annotation that can be added to a class to easily make it immutable. But really, there is more to avoiding immutable state than just having classes as immutable - As we have functions as first class objects, we can easily assign variables and mutate them within a function - so this is more of a mindset or philosophy that you have to get used to. For example: def list = ['groovy', 'functional'] //This mutates the original list list.add('programming') //This creates a new list leaving the original unchanged def newList = list.plus(2, 'programming') The first example is probably more like the Groovy/Java code we are used to writing, but that is mutating the state of the list - where as the second approach leaves the original list unchanged. Map Reduce: As a final note, there are some functions in FP that are pretty common techniques - the most famous of which these days (in part thanks to Google) is Map-Reduce, but the trio of functions are actually Map, Reduce(also known as Fold) & Filter - you can read more about the functions here (or just google them!), but these functions actually correlate pretty nicely to core Groovy functions that you probably use a lot of (assuming you are groovy programmers). Map map is the easiest to understand of the three. It takes in two inputs - a function, and a list. It then applies this function to every element in the list. You can basically do the same thing with a list comprehension however. Sound familiar? This is basically the .collect{} function in Groovy Reduce/Fold fold takes in a function and folds it in between the elements of a list. It's a bit hard to understand at first This one is a bit more complicated to descibe, but is the same as the .inject{} function in groovy Filter filter is easy. It takes in a 'test' and a list, and it chucks out any elements of the list which don't satisfy that test. And another simple one - filtering out a list for desired elements, this is Groovy's .findAll{} function As I said at the start, I am new to FP and coming from an OO background, but hopefully the above isn't too far from the truth! As I get further through the Coursera course I will try to post again, maybe with some of the assignments attempted in Groovy to see how it really stands up. Some useful references: Groovy docs on FP: http://groovy.codehaus.org/Functional+Programming+with+Groovy Coursera Scala course: https://www.coursera.org/course/progfun

Recently I needed to quickly create some temporary JSON objects from some preexisting data. So, I turned to Json.NET, a very popular JSON framework for .NET. Since I got my data on the fly (think parsing some other data structures and you have lots of anonymous results), I didn’t want to create classes for such instances. Luckily, Json.NET supports serializing anonymous objects. First add the library via NuGet – it is called Newtonsoft.Json. For example: var obj = new { user = new { name = "John", age = 21, data = new[] { 1, 2, 3, 4 } } }; var result = JsonConvert.SerializeObject(obj, Formatting.Indented); If we did that in a console application, we could print out the result. The printout is: { "user": { "name": "John", "age": 21, "data": [ 1, 2, 3, 4 ] } } The code above works on Windows 8, Windows Phone 7 and 8. This way it is dead simple to create JSON representation of your anonymous data. Think REST services.

this post comes from ernie souhrada at the mysql performance blog. note: this is part 1 of what will be a two-part series on the performance implications of using in-flight data encryption. some of you may recall my security webinar from back in mid-august; one of the follow-up questions that i was asked was about the performance impact of enabling ssl connections. my answer was 25%, based on some 2011 data that i had seen over on yassl’s website, but i included the caveat that it is workload-dependent, because the most expensive part of using ssl is establishing the connection. not long thereafter, i received a request to conduct some more specific benchmarks surrounding ssl usage in mysql, and today i’m going to show the results. first, the testing environment. all tests were performed on an intel core i7-2600k 3.4ghz cpu (8 cores, ht included) with 32gb of ram and centos 6.4. the disk subsystem is a 2-disk raid-0 of samsung 830 ssds, although since we’re only concerned with measuring the overhead added by using ssl connections, we’ll only be conducting read-only tests with a dataset that fits completely in the buffer pool. the version of mysql used for this experiment is community edition 5.6.13, and the testing tools are sysbench 0.5 and perl. we conduct two tests, each one designed to simulate one of the most common mysql usage patterns. first, we examine connection pooling, often seen in the java world, where some small set of connections are established by, for example, the servlet container and then just passed around to the application as needed, and one-request-per-connection, typical in the lamp world, where the script that displays a given page might connect to the database, run a couple of queries, and then disconnect. test 1: connection pool for the first test, i ran sysbench in read-only mode at concurrency levels of 1, 2, 4, 8, 16, and 32 threads, first with no encryption and then with ssl enabled and key lengths of 1024, 2048, and 4096 bits. 8 sysbench tables were prepared, each containing 100,000 rows, resulting in a total data size of approximately 256mb. the size of my innodb buffer pool was 4gb, and before conducting each official measurement run, i ran a warm-up run to prime the buffer pool. each official test run lasted 10 minutes; this might seem short, but unlike, say, a pcie flash storage device, i would not expect the variable under observation to really change that much over time or need time to stabilize. the basic sysbench syntax used is shown below. #!/bin/bash for ssl in on off ; do for threads in 1 2 4 8 16 32 ; do sysbench --test=/usr/share/sysbench/oltp.lua --mysql-user=msandbox$ssl --mysql-password=msandbox \ --mysql-host=127.0.0.1 --mysql-port=5613 --mysql-db=sbtest --mysql-ssl=$ssl \ --oltp-tables-count=8 --num-threads=$threads --oltp-dist-type=uniform --oltp-read-only=on \ --report-interval=10 --max-time=600 --max-requests=0 run > sb-ssl_${ssl}-threads-${threads}.out done done if you’re not familiar with sysbench, the important thing to know about it for our purposes is that it does not connect and disconnect after each query or after each transaction. it establishes n connections to the database (where n is the number of threads) and runs queries though them until the test is over. this behavior provides our connection-pool simulation. the assumption, given what we know about where ssl is the slowest, is that the performance penalty here should be the lowest. first, let’s look at raw throughput, measured in queries per second: the average throughput and standard deviation (both measured in queries per second) for each test configuration is shown below in tabular format: # of threads ssl key size 1 2 4 8 16 32 ssl off 9250.18 (1005.82) 18297.61 (689.22) 33910.31 (446.02) 50077.60 (1525.37) 49844.49 (934.86) 49651.09 (498.68) 1024-bit 2406.53 (288.53) 4650.56 (558.58) 9183.33 (1565.41) 26007.11 (345.79) 25959.61 (343.55) 25913.69 (192.90) 2048-bit 2448.43 (290.02) 4641.61 (510.91) 8951.67 (1043.99) 26143.25 (360.84) 25872.10 (324.48) 25764.48 (370.33) 4096-bit 2427.95 (289.00) 4641.32 (547.57) 8991.37 (1005.89) 26058.09 (432.86) 25990.13 (439.53) 26041.27 (780.71) so, given that this is an 8-core machine and io isn’t a factor, we would expect throughput to max out at 8 threads, so the levelling-off of performance is expected. what we also see is that it doesn’t seem to make much difference what key length is used, which is also largely expected. however, i definitely didn’t think the encryption overhead would be so high. the next graph here is 95th-percentile latency from the same test: and in tabular format, the raw numbers (average and standard deviation): # of threads ssl key size 1 2 4 8 16 32 ssl off 1.882 (0.522) 1.728 (0.167) 1.764 (0.145) 2.459 (0.523) 6.616 (0.251) 27.307 (0.817) 1024-bit 6.151 (0.241) 6.442 (0.180) 6.677 (0.289) 4.535 (0.507) 11.481 (1.403) 37.152 (0.393) 2048-bit 6.083 (0.277) 6.510 (0.081) 6.693 (0.043) 4.498 (0.503) 11.222 (1.502) 37.387 (0.393) 4096-bit 6.120 (0.268) 6.454 (0.119) 6.690 (0.043) 4.571 (0.727) 11.194 (1.395) 37.26 (0.307) with the exception of 8 and 32 threads, the latency introduced by the use of ssl is constant at right around 5ms, regardless of the key length or the number of threads. i’m not surprised that there’s a large jump in latency at 32 threads, but i don’t have an immediate explanation for the improvement in the ssl latency numbers at 8 threads. test 2: connection time for the second test, i wrote a simple perl script to just connect and disconnect from the database as fast as possible. we know that it’s the connection setup which is the slowest part of ssl, and the previous test already shows us roughly what we can expect for ssl encryption overhead for sending data once the connection has been established, so let’s see just how much overhead ssl adds to connection time. the basic script to do this is quite simple (non-ssl version shown): #!/usr/bin/perl use dbi; use time::hires qw(time); $start = time; for (my $i=0; $i<100; $i++) { my $dbh = dbi->connect("dbi:mysql:host=127.0.0.1;port=5613", "msandbox","msandbox",undef); $dbh->disconnect; undef $dbh; } printf "%.6f\n", time - $start; as with test #1, i ran test #2 with no encryption and ssl encryption of 1024, 2048, and 4098 bits, and i conducted 10 trials of each configuration. then i took the elapsed time for each test and converted it to connections per second. the graph below shows the results from each run: here are the averages and standard deviations: encryption average connections per second standard deviation none 2701.75 165.54 1024-bit 77.04 6.14 2048-bit 28.183 1.713 4096-bit 5.45 0.015 yes, that’s right, 4096-bit ssl connections are 3 orders of magnitude slower to establish than unencrypted connections. really, the connection overhead for any level of ssl usage is quite high when compared to the unencrypted test, and it’s certainly much higher than my original quoted number of 25%. analysis and parting thoughts so, what do we take away from this? the first thing is, of course, is that ssl overhead is a lot higher than 25%, particularly if your application uses anything close to the one-connection-per-request pattern. for a system which establishes and maintains long-running connections, the initial connection overhead becomes a non-factor, regardless of the encryption strength, but there’s still a rather large performance penalty compared to the unencrypted connection. this leads directly into the second point, which is that connection pooling is by far a more efficient method of using ssl if your application can support it. but what if connection pooling isn’t an option, mysql’s ssl performance is insufficient, and you still need full encryption of data in-flight? run the encryption component of your system at a lower layer – a vpn with hardware crypto would be the fastest approach, but even something as simple as an ssh tunnel or openvpn *might* be faster than ssl within mysql. i’ll be exploring some of these solutions in a follow-up post. and finally… when in doubt, run your own benchmarks. i don’t have an explanation for why the yassl numbers are so different from these (maybe yassl is a faster ssl library than openssl, or maybe they used a different cipher – if you’re curious, the original 25% number came from slides 56-58 of this presentation ), but in any event, this does illustrate why it’s important to run tests on your own hardware and with your own workload when you’re interested in finding out how well something will perform rather than taking someone else’s word for it.

The following question will again test your knowledge of the Oracle Weblogic threading model. I’m looking forward for your comments and experience on the same. If you are a Weblogic administrator, I’m certain that you heard of this common problem: stuck threads. This is one of the most common problems you will face when supporting a Weblogic production environment. A Weblogic stuck thread simply means a thread performing the same request for a very long time and more than the configurable Stuck Thread Max Time. Question: How can you detect the presence of STUCK threads during and following a production incident? Answer: As we saw from our last article “Weblogic Thread Monitoring Tips”, Weblogic provides functionalities allowing us to closely monitor its internal self-tuning thread pool. It will also highlight you the presence of any stuck thread. This monitoring view is very useful when you do a live analysis but what about after a production incident? The good news is that Oracle Weblogic will also log any detected stuck thread to the server log. Such information includes details on the request and more importantly, the thread stack trace. This data is crucial and will allow you to potentially better understand the root cause of any slowdown condition that occurred at a certain time. < ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'> <[STUCK] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "608" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 608213 ms POST /App1/jsp/test.jsp HTTP/1.1 Accept: application/x-ms-application... Referer: http://.. Accept-Language: en-US User-Agent: Mozilla/4.0 .. Content-Type: application/x-www-form-urlencoded Accept-Encoding: gzip, deflate Content-Length: 539 Connection: Keep-Alive Cache-Control: no-cache Cookie: JSESSIONID= ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace: ................................... javax.servlet.http.HttpServlet.service(HttpServlet.java:727) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:184) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.... weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run() weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2281) weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2180) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491) weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) weblogic.work.ExecuteThread.run(ExecuteThread.java:221) Here is one more tip: the generation and analysis of a JVM thread dump will also highlight you stuck threads. As we can see from the snapshot below, the Weblogic thread state is now updated to STUCK, which means that this particular request is being executed since at least 600 seconds or 10 minutes. This is very useful information since the native thread state will typically remain to RUNNABLE. The native thread state will only get updated when dealing with BLOCKED threads etc. You have to keep in mind that RUNNABLE simply means that this thread is healthy from a JVM perspective. However, it does not mean that it truly is from a middleware or Java EE container perspective. This is why Oracle Weblogic has its own internal ExecuteThread state. Finally, if your organization or client is using any commercial monitoring tool, I recommend that you enable some alerting around both hogging thread and stuck thread. This will allow your support team to take some pro-active actions before the affected Weblogic managed server(s) become fully unresponsive.

I’m going to start by discussing the Spring WebMVC configuration and move on from there in future posts.

When putting together data sets to play around with, one of the more boring tasks is stripping out characters that you’re not interested in and more often than not those characters are white spaces. Since I’ve been building data sets using Clojure I wanted to write a function that would do this for me. I started out with the following string: (def word " with a little bit of space we can make it through the night ") which I wanted to format in such a way that there would be a maximum of one space between each word. I start out by using the trim function but that only removes white space from the beginning and end of a string: > (clojure.string/trim word) "with a little bit of space we can make it through the night" I wanted to get rid of the space in between ‘a’ and ‘little’ as well so I wrote the following code to split on a space and filter out any excess spaces that still remained before joining the words back together: > (clojure.string/join " " (filter #(not (clojure.string/blank? %)) (clojure.string/split word #" "))) "with a little bit of space we can make it through the night" I wanted to try and make it a bit easier to read by using the thread last (->>) macro but that didn’t work as well as I’d hoped because clojure.string/split doesn’t take the string in as its last parameter: > (->> (clojure.string/split word #" ") (filter #(not (clojure.string/blank? %))) (clojure.string/join " ")) "with a little bit of space we can make it through the night" I worked around it by creating a specific function for splitting on a space: (defn split-on-space [word] (clojure.string/split word #"\s")) which means we can now chain everything together nicely: > (->> word split-on-space (filter #(not (clojure.string/blank? %))) (clojure.string/join " ")) "with a little bit of space we can make it through the night" I couldn’t find a cleaner way to do this but I’m sure there is one and my googling just isn’t up to scratch so do let me know in the comments!

In the previous post, we have seen an example where dataProvider attribute has been used3 to test methods with different sets of input data for the same test method. TestNG provides another attribute dataProviderClass in conjunction with dataProvider to fetch the input data for the test methods from an external class. The actual class that holds input data is set to the dataProviderClass attribute and datProvider by itself holds the method name where the input data is actually fetched. Here is a quick example to show how to use dataProviderClass and dataProvide attribute Code Service Class ? view source print? 01.package com.skilledmonster.example; 02./** 03.* Simple calculator service to demonstrate TestNG Framework 04.* 05.* @author Jagadeesh Motamarri 06.* @version 1.0 07.*/ 08.public interface CalculatorService { 09.int sum(int a, int b); 10.int multiply(int a, int b); 11.int div(int a, int b); 12.int sub(int a, int b); 13.} Service Implementation Class ? view source print? 01.package com.skilledmonster.example; 02./** 03.* Simple calculator service implementation to demonstrate TestNG Framework 04.* 05.* @author Jagadeesh Motamarri 06.* @version 1.0 07.*/ 08.public class SimpleCalculator implements CalculatorService { 09.public int sum(int a, int b) { 10.return a + b; 11.} 12.public int multiply(int a, int b) { 13.return a * b; 14.} 15.public int div(int a, int b) { 16.return a / b; 17.} 18.public int sub(int a, int b) { 19.return a - b; 20.} 21.} Data Provider Class ? view source print? 01.package com.skilledmonster.common; 02.import org.testng.annotations.DataProvider; 03./** 04.* Data Provider class for TestNG test cases 05.* 06.* @author Jagadeesh Motamarri 07.* @version 1.0 08.*/ 09.public class TestNGDataProvider { 10./** 11.* Data Provider for testing sum of 2 numbers 12.* 13.* @return 14.*/ 15.@DataProvider 16.public static Object[][] testSumInput() { 17.return new Object[][] { { 5, 5 }, { 10, 10 }, { 20, 20 } }; 18.} 19./** 20.* Data Provider for testing multiplication of 2 numbers 21.* 22.* @return 23.*/ 24.@DataProvider 25.public static Object[][] testMultipleInput() { 26.return new Object[][] { { 5, 5 }, { 10, 10 }, { 20, 20 } }; 27.} 28.} Finally, test class that uses dataProviderClass attribute to feed the input data for the test methods ? package com.skilledmonster.example; import org.testng.Assert; import org.testng.annotations.BeforeClass; import org.testng.annotations.Test; import com.skilledmonster.common.TestNGDataProvider; /** * Example to demonstrate use of dataProviderClass and dataProvide attributes of TestNG framework * * @author Jagadeesh Motamarri * @version 1.0 */ public class TestNGAnnotationTestDataProviderExample { public CalculatorService service; @BeforeClass public void init() { System.out.println("@BeforeClass: The annotated method will be run before the first test method in the current class is invoked."); System.out.println("init service"); service = new SimpleCalculator(); } @Test(dataProviderClass = TestNGDataProvider.class, dataProvider = "testSumInput") public void testSum(int a, int b) { System.out.println("@Test : testSum()"); int result = service.sum(a, b); Assert.assertEquals(result, a + b); } @Test(dataProviderClass = TestNGDataProvider.class, dataProvider = "testMultipleInput") public void testMultiple(int a, int b) { System.out.println("@Test : testMultiple()"); int result = service.multiply(a, b); Assert.assertEquals(result, a * b); } } Output As shown in the above console output, each of the testSum() and testMutiple() methods are invoked with different sets of input data using an external class with dataProviderClass attribute. Advantage More flexibility and re-usability of commonly used data across several test classes. Download Download TestNG DataProvider Example

Here are some free HTML WYSIWYG editors based on the Mozilla Gecko Engine.

I wanted to do some date manipulation in Clojure recently and figured that since clj-time is a wrapper around Joda Time it’d probably do the trick. The first thing we need to do is add the dependency to our project file and then run lein reps to pull down the appropriate JARs. The project file should look something like this: project.clj (defproject ranking-algorithms "0.1.0-SNAPSHOT" :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"} :dependencies [[org.clojure/clojure "1.4.0"] [clj-time "0.6.0"]]) Now let’s load the clj-time.format namespace into the REPL since we know we’ll be parsing dates: > (require '(clj-time [format :as f])) The string that I want to convert into a date looks like this: (def string-date "18 September 2012") The first thing we should do is check whether there is an existing formatter that we can use by evaluating the following function: > (f/show-formatters) ... :hour-minute 06:45 :hour-minute-second 06:45:22 :hour-minute-second-fraction 06:45:22.473 :hour-minute-second-ms 06:45:22.473 :mysql 2013-09-20 06:45:22 :ordinal-date 2013-263 :ordinal-date-time 2013-263T06:45:22.473Z :ordinal-date-time-no-ms 2013-263T06:45:22Z :rfc822 Fri, 20 Sep 2013 06:45:22 +0000 ... There are a lot of different built in formatters but unfortunately I couldn’t find one that exactly matched our date format so we’ll have to write our own one. For that we’ll need to refresh our knowledge of Java date formatting: We end up with the following formatter: > (f/parse (f/formatter "dd MMM YYYY") string-date) # It took me much longer than it should have to remember that ‘MMM’ is the pattern to match a short form of a month but it’s just the same as what we’d have to do in Java but with some neat wrapper functions.

If you work with large, gridded datasets, you should probably be using NetCDF, the Network Common Data Form from Unidata: NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. Lots of high-end analysis software can be made to support NetCDF, and it is indispensable for working with gridded datasets that weigh in at tens of gigabytes or more. This brief post describes the easiest way to install the NetCDF libraries and the R ‘ncdf’ package on our favorite systems: CentOS, Ubuntu and Mac OSX. CentOS 6.x CentOS is the operating system of choice if you want a free, robust, open-source server to host your scientific analysis. It is basically an unbranded clone of Red Hat Enterprise Linux. The following instructions worked on CentOS 6.2. Installing System Libraries First, go to http://fedoraproject.org/wiki/EPEL and check for the latest version of the Extended Packages for Enterprise Linux (which contains NetCDF, HDF and many other useful packages). The latest version should be specified on this page. To download a local copy of EPEL and install NetCDF from it, just execute the following commands: sudo wget http://mirror.metrocast.net/fedora/epel/6/i386/epel-release-6-8.noarch.rpm sudo rpm -Uvh epel-release-6-8.noarch.rpm sudo yum --assumeyes install netcdf sudo yum --assumeyes install netcdf-devel Note that adding EPEL as a package archive in the second line does not automatically install all the packages in EPEL. We had to manually install netcdf and netcdf-dev. A list of other available packages is given at the EPEL wiki given above. Installing R Package ‘ncdf’ With the libraries in place, we can now install the ncdf package for our favorite statistical package — R. sudo wget http://cran.r-project.org/src/contrib/ncdf_1.6.6.tar.gz sudo R CMD INSTALL --configure-args="--with-netcdf-include=/usr/include --with-netcdf-lib=/usr/lib" ncdf_1.6.6.tar.gz Ubuntu 12.04 Ubuntu is easy to install and has a great user interface for linux systems. Ubuntu 12.0.4 is the most recent Long-Term-Stable release. Installing System Libraries The following instructions worked on Ubuntu 12.04 LTS. To install NetCDF libraries that allow reading, writing and manipulation, use apt-get, rather than downloading the source files and installing them yourself. To install, open a terminal and type: sudo apt-get install netcdf Installing R package ‘ncdf’ The base version of R on Ubuntu 12.04 slt is 2.14.1. Unfortunately, clicking the install button in RStudio and typing 'ncdf' will only work at the user level. The package will not be installed for all users or even show up in all of your RStudio projects. To install ncdf tools in the global library you must start up R as root and use the following command: install.packages(repos=c('http://cran.fhcrc.org/'),pkgs=c('ncdf'),lib="/usr/lib/R/site-library/") ‘http://cran.fhcrc.org/’ should be replaced by whichever CRAN mirror is closest to you. OSX 10.8.4 Macs run OSX, which is Unix based. The following instructions worked on OSX 10.8.4 — Mountain Lion. Installing System Libraries The absolute easiest way to install NetCDF on a mac requires Macports. Macports is a software package designed to make installing and compiling software easy. Macports .pkg and installation instructions are available here. Once Macports is installed, building and installing NetCDF libraries is a one-step job. sudo port install netcdf More details and instructions for installing Fortran and Python APIs are available here. Installing R Package ‘ncdf’ Having NetCDF command line tools is not required to use the ncdf R package. Simply download the package from CRAN (link), or by clicking on the “install packages” button in RStudio. This package allows reading, writing and manipulation of existing .nc files. However, the package’s ability to view the content of nc files before loading them into the R workspace is limited. For this reason, installing the NetCDF tools outlined in the first section of this post is extremely important. Command line tools such as “ncdump” are crucial to effectively working with NetCDF files.

ElasticSearch provides Java API, thus it executes all operations asynchronously by using client object.

So, I’ve been wanting to get back to playing with C# for a while, and finally have had the opportunity. I’ve also been wanting to play with the Task library in .NET and see if I could get it to do something interesting, well below is the result. The code below, running in a .NET 4 project, will run two SQL SELECT statements against the AdventureWorks2012 database. There are three tasks in here, ParallelTask 1 and 2, and a timing task. The Parallel task takes a Connection String and a query as inputs, and passes out a Status Message. One of the important points with a task is that the task has to be self contained. This is why the connection is instantiated within the task. I also added in a Timing task (ParallelTiming) so I could pass out a ping message. The whole thing is controlled by the code in the main section, which is used to start the three tasks, with their appropriate parameters. After this it awaits the tasks completing, then passes out the resulting return messages. Try it out; it’s good fun and all you need is SQL Server, AdventureWorks and something to build C# projects. You can download the code here Have fun! /// Parallel_SQL demonstration code /// From Nick Haslam /// http://blog.nhaslam.com /// 16/9/2013 using System; using System.Collections.Generic; using System.Data.SqlClient; using System.Linq; using System.Text; using System.Threading.Tasks; namespace Parallel_SQL { class Program { /// /// First Parallel task /// ///Connection string details ///Query to execute ///Status message to pass back /// static Task ParallelTask1(string sConnString, string sQuery, Action StatusMessage) { return Task.Factory.StartNew(() => { SqlConnection conn = new SqlConnection(sConnString); conn.Open(); StatusMessage(“Running Query”); SqlDataReader reader = null; SqlCommand sqlCommand = new SqlCommand(sQuery, conn); reader = sqlCommand.ExecuteReader(); while (reader.Read()) { StatusMessage(reader[0].ToString()); } return “Task 1 Complete”; }); } /// /// Second Parallel task /// ///Connection string details ///Query to execute ///Status message to pass back /// static Task ParallelTask2(string sConnString, string sQuery, Action StatusMessage) { return Task.Factory.StartNew(() => { SqlConnection conn = new SqlConnection(sConnString); conn.Open(); StatusMessage(“Running Query”); SqlDataReader reader = null; SqlCommand sqlCommand = new SqlCommand(sQuery, conn); reader = sqlCommand.ExecuteReader(); while (reader.Read()) { StatusMessage(reader[0].ToString()); } return “Task 2 Complete”; }); } /// /// Timing Task /// ///Milliseconds between ping ///Status message to pass back /// static Task ParallelTiming(int iMSPause, Action StatusMessage) { return Task.Factory.StartNew(() => { for (int i = 0; i < 10; i++) { System.Threading.Thread.Sleep(iMSPause); StatusMessage(“******************** PING ********************”); } return “Timing task done”; }); } static void Main(string[] args) { string sConnString = “server=.; Trusted_Connection=yes; database=AdventureWorks2012;”; try { var Task1Control = ParallelTask1(sConnString, “SELECT top 500 TransactionID FROM Production.TransactionHistory”, (update) => { Console.WriteLine(String.Format(“{0} – {1}”, DateTime.Now, update)); }); var Task2Control = ParallelTask2(sConnString, “SELECT top 500 SalesOrderDetailID FROM sales.SalesOrderDetail”, (update) => { Console.WriteLine(String.Format(“{0} – \t\t{1}”, DateTime.Now, update)); }); var TimingTaskControl = ParallelTiming(250, (update) => { Console.WriteLine(String.Format(“{0} – \t\t\t{1}”, DateTime.Now, update)); }); // Await Completion of the tasks Console.WriteLine(“Task 1 Status – {0}”, Task1Control.Result); Console.WriteLine(“Task 2 Status – {0}”, Task2Control.Result); Console.WriteLine(“Timing Task Status – {0}”, TimingTaskControl.Result); } catch (Exception e) { Console.WriteLine(e.ToString()); } Console.ReadKey(); } } }

When I was implementing the Elo Rating algorithm a few weeks ago one thing I needed to do was come up with a base ranking for each team. I started out with a set of teams that looked like this: (def teams #{ "Man Utd" "Man City" "Arsenal" "Chelsea"}) and I wanted to transform that into a map from the team to their ranking e.g. Man Utd -> {:points 1200} Man City -> {:points 1200} Arsenal -> {:points 1200} Chelsea -> {:points 1200} I had read the documentation of array-map, a function which can be used to transform a collection of pairs into a map, and it seemed like it might do the trick. I started out by building an array of pairs using mapcat: > (mapcat (fn [x] [x {:points 1200}]) teams) ("Chelsea" {:points 1200} "Man City" {:points 1200} "Arsenal" {:points 1200} "Man Utd" {:points 1200}) array-map constructs a map from pairs of values e.g. > (array-map "Chelsea" {:points 1200} "Man City" {:points 1200} "Arsenal" {:points 1200} "Man Utd" {:points 1200}) ("Chelsea" {:points 1200} "Man City" {:points 1200} "Arsenal" {:points 1200} "Man Utd" {:points 1200}) Since we have a collection of pairs rather than individual pairs we need to use the apply function as well: > (apply array-map ["Chelsea" {:points 1200} "Man City" {:points 1200} "Arsenal" {:points 1200} "Man Utd" {:points 1200}]) {"Chelsea" {:points 1200}, "Man City" {:points 1200}, "Arsenal" {:points 1200}, "Man Utd" {:points 1200} And if we put it all together we end up with the following: > (apply array-map (mapcat (fn [x] [x {:points 1200}]) teams)) {"Man Utd" {:points 1200}, "Man City" {:points 1200}, "Arsenal" {:points 1200}, "Chelsea" {:points 1200} It works but the function we pass to mapcat feels a bit clunky. Since we just need to create a collection of team/ranking pairs we can use the vector and repeat functions to build that up instead: > (mapcat vector teams (repeat {:points 1200})) ("Chelsea" {:points 1200} "Man City" {:points 1200} "Arsenal" {:points 1200} "Man Utd" {:points 1200}) And if we put the apply array-map code back in we still get the desired result: > (apply array-map (mapcat vector teams (repeat {:points 1200}))) {"Chelsea" {:points 1200}, "Man City" {:points 1200}, "Arsenal" {:points 1200}, "Man Utd" {:points 1200} Alternatively we could use assoc like this: > (apply assoc {} (mapcat vector teams (repeat {:points 1200}))) {"Man Utd" {:points 1200}, "Arsenal" {:points 1200}, "Man City" {:points 1200}, "Chelsea" {:points 1200} I also came across the into function which seemed useful but took in a collection of vectors: > (into {} [["Chelsea" {:points 1200}] ["Man City" {:points 1200}] ["Arsenal" {:points 1200}] ["Man Utd" {:points 1200}] ]) We therefore need to change the code to use map instead of mapcat: > (into {} (map vector teams (repeat {:points 1200}))) {"Chelsea" {:points 1200}, "Man City" {:points 1200}, "Arsenal" {:points 1200}, "Man Utd" {:points 1200} However, my favourite version so far uses the zipmap function like so: > (zipmap teams (repeat {:points 1200})) {"Man Utd" {:points 1200}, "Arsenal" {:points 1200}, "Man City" {:points 1200}, "Chelsea" {:points 1200} I’m sure there are other ways to do this as well so if you know any let me know in the comments.

TestNG method that is annotated with @BeforeClass annotation will be run before the first test method in the current class is invoked.

Intro If you want to manage your SQL Databases in Azure using tools that you’re a little more familiar and comfortable with – for example – SQL Management Studio, how do you go about connecting? You could read the help article from Microsoft, or you can follow my intuitive screen-based instructions, below: Assumptions 1. I’m assuming you have a version of SQL Management Studio already installed. I believe you’ll need at least SQL Server 2008 R2’s version or newer 2. I’m further assuming you’ve already created a SQL Database in Azure Steps to Connect SSMS to SQL Azure 1. Authenticate to the Azure Portal 2. Click on SQL Databases 3. Click on Servers 4. Click on the name of the Server you wish to connect to… 5. Click on Configure… If not already in place, click on ‘Add to the allowed IP addresses’ to add your current IP address (or specify an address you wish to connect from) and click ‘Save’ 6. Open SQL Management Studio and connect to Database services (usually comes up by default) Enter the fully qualified server name (.database.windows.net) Change to SQL Server Authentication Enter the login preferred (if a new database, the username you specified when yuo created the DB server) Enter the correct password 7. Hit the Connect button Troubleshooting Ensure you have the appropriate ports open outbound from your local network or connection (typically port 1433) Ensure you have allowed the correct public IP address you’re trying to connect from via the Azure Portal (steps 1-5 above) Ensure you are using the correct server name and user name For SSMS, this is the server name (in step 4) followed by .database.windows.net Ensure you are using SQL Server Authentication For SSMS the username format is If you forgot the password of your username, you can reset the password in the Azure Portal, in step 4, click on Dashboard: Lastly… You can click on the Database (in step 2) to see your connection options:

Advantages coming from dependency injection can be addictive. It's a lot easier to configure application structure using injections than doing all resolutions manually. It's hard to resign from it when we have some non-managed classes that are instantiated outside of the container - for example being part of other frameworks like Vaadin UI components or JPA entities. The latter are especially important when we're using Domain Driven Design. During DDD training ran by Slawek Sobotka we were talking about options to remove "bad coupling" from aggregate factories. I'm sure you'll admit, that it's better to have generic mechanism able to "energize" objects by dependencies defined by e.g. @Inject annotation than inject all necessary dependencies into particular factory and then pass them into object by constructor, builder or simple setters. Spring framework brings us two different solutions to achieve such requirement. I'll now describe them both of them. Let's start with with simpler one. It's especially powerful when we've case such as generic repository, mentioned earlier aggregate factory, or even any other factory just ensuring that we have only few places in our code where will instantiate objects outside of the container. In this case we can make use of AutowireCapableBeanFactory class. In particular we will be interested in two methods: void autowireBean(Object existingBean) Object initializeBean(Object existingBean, String beanName) The first one just populates our bean without applying specific post processors (e.g. @PostConstruct, etc). The second one additionally applies factory callbacks like setBeanName and setBeanFactory, as well as any other post processors with @PostConstruct of course. In our code it'll look like this: public abstract class GenericFactory { @Autowired private AutowireCapableBeanFactory autowireBeanFactory; public T createBean() { // creation logic autowireBeanFactory.autowireBean(createdBean); return createdBean; } } Simple and powerful - my favorite composition :) But what can we do when we have plenty places in our code where objects get born? That's for example case of building Vaadin layouts in Spring web application. Injecting custom bean configurer objects invoking autowireBean method won't be the peak of productivity. Happily Spring developers brought us @Configurable annotation. This annotation connected with aspects will configure each annotated object even if we will create it outside of the container using new operator. Like with any other aspects we can choose between load-time-waving (LTW) compile-time-waving (CTW). The first one is easier to configure but we become (due to instrumentation) dependent from application server, which can be undesirable in some cases. To use it we need to annotate our @Configuration class by @EnableLoadTimeWeaving (or add tag if you like Flintstones and XML configuration ;)) After completing configuration just annotate class by @Configurable: @Configurable public class MyCustomButton extends Button { @Autowired private MyAwesomeService myAwesomeService; // handlers making use of injected service } The second option is a little bit more complex to setup but after this it will be a lot lighter in runtime. Because now we want to load aspects till compilation we have to integrate aspectj compiler into our build. In Maven you need to add few dependencies: org.aspectj aspectjrt 1.7.3 org.springframework spring-aspects 3.2.4.RELEASE org.springframework spring-tx 3.2.4.RELEASE javax.persistence persistence-api 1.0 provided I hope you are curious why above you can see persistence-api. That was also strange for me, when I saw "can't determine annotations of missing type javax.persistence.Entity" error during aspectj compilation. The answer can be found in SpringFramework JIRA in issue SPR-6819. This happens when you configure spring-aspects as a aspectLibrary in aspectj-maven-plugin. Issue is unresolved for over three year so better get used to it :) The last thing we need to do is to include above-mentioned plugin into our plugins section. org.codehaus.mojo aspectj-maven-plugin 1.5 1.7 1.7 1.7 true org.springframework spring-aspects compile And that's all folks :)