One thing that Big Data certainly made happen is that it brought the database/infrastructure community and the data analysis/statistics/machine learning communities closer together. As always, each community had it’s own set of models, methods, and ideas about how to structure and interpret the world. You can still tell these differences when looking at current Big Data projects, and I think it’s important to be aware of the distinctions in order to better understand the relationships between different projects. Because, let’s face it, every project claims to re-invent Big Data. Hadoop and MapReduce being something like the founding fathers of Big Data, other’s projects have since appeared. Most notably, there are stream processing projects like Twitter’s Storm who move from batch-oriented processing to event-based processing which is more suited for real-time, low-latency processing. Spark is yet something different, a bit like Hadoop, but puts greater emphasis on iterative algorithms, and in-memory processing to achieve that landmark “100x faster than Hadoop” every current project seems to need to sport. Twitter’s summingbird project tries to bridge the gap between MapReduce and stream processing by providing us with a high-level set of operators which can then either run on MapReduce or Storm. However, both Spark or summingbird leave me sort of flat because you can see that they come from a database background, which means that there will still be a considerable gap to serious machine learning. So, what exactly is the difference? In the end, it’s the difference between relational and linear algebra. In the database world, you model relationships between objects, which you encode in tables, and foreign keys to link up entries between different tables. Probably the most important insight of the database world was to develop a query language, a declarative description of what you want to extract from your database, leaving the optimization of the query and the exact details of how to perform them efficiently to the database guys. The machine learning community, on the other hand, has its roots in linear algebra and probability theory. Objects are usually encoded as a feature vector, that is, a list of numbers describing different properties of an object. Data is often collected in matrices where each row corresponds to an object, and each column to a feature, not much unlike a table in a database. However, the operations you perform in order to do data analysis are quite different from the data base world. Take something as basic as linear regression: your try to learn a linear function f(x)=di=1wixi in a d-dimensional space (that is, where your objects are described by a d-dimensional vector) given n examples Xi, and Yi, where Xi are the features describing your objects and Yi is the real number you attach to Xi. One way to “learn” w is to tune it such that the quadratic error on the training examples is minimal. The solution can be written in closed form as w=(XXT)−1XY where X is the matrix built from the Xi (putting the Xi in the columns of X), and Y is the vector of outputs Yi. In order to solve this, you need to solve the linear equation (XXT)w=XY which can be done by one of a large number of algorithms, starting with Gaussian elimination, which you’ve probably learned in your undergrad studies, or the conjugate gradient algorithm, or by first computing a Cholesky decomposition. All of these algorithms have in common that they are iterative. They go through a number of operations, for example O(d3) for the Gaussian elimination case. They also need to store intermediate results. Gaussian elimination and Cholesky decomposition have rather elementary operations acting on individual entries, while the conjugate gradient algorithm performs a matrix-vector multiplication in each iteration. Most importantly, these algorithms can only be expressed very badly in SQL! It’s certainly not impossible, but you’d need to store your data in much different ways than you would in idiomatic database usage. So, it’s not about whether or not your framework can support iterative algorithms without significant latency, it’s about understanding that joins, group bys, and count() won’t get you far, but you need scalar products, matrix-vector and matrix-matrix multiplications. You don’t need indices for most ML algorithms, maybe except for being able to quickly find the k-nearest neighbors, because most algorithms tend to either take in the whole data set in each iteration or otherwise stream the whole set by some model which is iteratively updated like in stochastic gradient descent. I’m not sure projects like Spark or Stratosphere have fully grasped the significance of this yet. Database infrastructure-inspired Big Data has it’s place when it comes to extracting and preprocessing data, but eventually, you move from database land to machine learning land, which invariably means linear algebra land (or probability theory land, which often also reduces to linear algebra like computations). What often happens today is that you either painstakingly have to break down your linear algebra into MapReduce jobs, or you actively look for algorithms which fit the database view better. I think we’re still at the beginning of what is possible. Or, to be a bit more aggressive, claims that existing (infrastructure, database, parallelism inspired) frameworks provide you with sophistic data analytics are widely exaggerated. They take care of a very important problem by giving you a reliable infrastructure to scale your data analysis code, but there’s still a lot of work that needs to be done on your side. High-level DSLs like Apache Hive or Pig are a first step in this direction but still too much rooted in the database world IMHO. In summary, one should be aware of the difference between a framework which mostly is concerned with scaling and a tool which actually provides some piece of data analysis. And even if it comes with basic database-like analytics mechanisms, there is still a long way to go to do some serious data science. That’s why we’re also thinking that streamdrill occupies an interesting spot, because it is a bit of infrastructure, allowing you to process a serious amount of event data, but it also provides valuable analysis based on algorithms you wouldn’t want to implement yourself, even if you had some Big Data framework like Hadoop at hand. That’s an interesting direction I also would like to see more of in the future. Note: Just saw that Spark has a logistic regression example on their landing page. Well, doing matrix operations explicitly via map() on collections doesn’t count in my view ;)

simple talk - How to get SQL Railroad Diagrams from MSDN BNF syntax notation. On SQL Server Books-On-Line, in the Transact-SQL Reference (database Engine), every SQL Statement has its syntax represented in ‘Backus–Naur Form’ notation (BNF) syntax. For a programmer in a hurry, this should be ideal because It is the only quick way to understand and appreciate all the permutations of the syntax. It is a great feature once you get your eye in. It isn’t the only way to get the information; You can, of course, reverse-engineer an understanding of the syntax from the examples, but your understanding won’t be complete, and you’ll have wasted time doing it. BNF is a good start in representing the syntax: Oracle and SQLite go one step further, and have proper railroad diagrams for their syntax, which is a far more accessible way of doing it. There are three problems with the BNF on MSDN. Firstly, it is isn’t a standard version of BNF, but an ancient fork from EBNF, inherited from Sybase. Secondly, it is excruciatingly difficult to understand, and thirdly it has a number of syntactic and semantic errors. The page describing DML triggers, for example, currently has the absurd BNF error that makes it state that all statements in the body of the trigger must be separated by commas. There are a few other detail problems too. Here is the offending syntax for a DML trigger, pasted from MSDN. ... I’ve been trying to create railroad diagrams for all the important SQL Server SQL statements, as good as you’d find for Oracle, and have so far published the CREATE TABLE and ALTER TABLE railroad diagrams based on the BNF. Although I’ve been aware of them, I’ve never realised until recently how many errors there are. Then, Colin Daley created a translator for the SQL Server dialect of BNF which outputs standard EBNF notation used by the W3C. The example MSDN BNF for the trigger would be rendered as … ... Colin’s intention was to allow anyone to paste SQL Server’s BNF notation into his website-based parser, and from this generate classic railroad diagrams via Gunther Rademacher's Railroad Diagram Generator. Colin's application does this for you: you're not aware that you are moving to a different site. Because Colin's 'translator' it is a parser, it will pick up syntax errors. Once you’ve fixed the syntax errors, you will get the syntax in the form of a human-readable railroad diagram and, in this form, the semantic mistakes become flamingly obvious. Gunter’s Railroad Diagram Generator is brilliant. To be able, after correcting the MSDN dialect of BNF, to generate a standard EBNF, and from thence to create railroad diagrams for SQL Server’s syntax that are as good as Oracle’s, is a great boon, and many thanks to Colin for the idea. Here is the result of the W3C EBNF from Colin’s application then being run through the Railroad diagram generator. Now that’s much better, you’ll agree. This is pretty easy to understand, and at this point any error is immediately obvious. This should be seriously useful, and it is to me. However there is that snag. The BNF is generally incorrect, and you can’t expect the average visitor to mess about with it. The answer is, of course, to correct the BNF on MSDN and maybe even add railroad diagrams for the syntax. Stop giggling! I agree it won’t happen. In the meantime, we need to collaboratively store and publish these corrected syntaxes ourselves as we do them. How? GitHub? SQL Server Central? Simple-Talk? What should those of us who use the system do with our corrected EBNF so that anyone can use them without hassle? Grammar Translator If you are familiar with the Grammar Translator, go ahead and create railroad diagrams from the Transact-SQL Reference. Otherwise, please see the FAQ. In particular, be sure to try thetutorial. Welcome to Railroad Diagram Generator! This is a tool for creating syntax diagrams, also known as railroad diagrams, from context-free grammars specified in EBNF. Syntax diagrams have been used for decades now, so the concept is well-known, and some tools for diagram generation are in existence. The features of this one are usage of the W3C's EBNF notation, web-scraping of grammars from W3C specifications, online editing of grammars, diagram presentation in SVG, and it was completely written in web languages (XQuery, XHTML, CSS, JavaScript). There's nothing like a diagram to help grok something (and the MSDN BNF SQL stuff really makes my brain hurt...)

Hyphenation is always a big question, and the fact that it can vary for the same word causes significant confusion. This article will give you some tips and common usages.

Some months ago I wrote an article on how to transition between Scrum and a Lean Kanban operation. It's an important capability for an organization to have, because when a Scrum project finishes it is likely to enter a "leaner" BAU (Business As Usual) support phase. There are consequences arising from such a move which experienced Scrum hands may find surprising, and perhaps even a little off-putting. In this article we'll look at the shift in mindset that is required to do this. "Whoa! Something screwy has happened to our task board, it looks different" Kanban boards are subtly different to the task boards commonly used in Scrum. At first blush they might look similar. Both have columns showing the progress of user story "tickets" from a backlog through states such as in progress, peer review, in test, and done. In either case there might also be a blocked column, although it is equally acceptable to add a "blocked" sticker, or to simply invert the ticket on the board. As the name suggests, a task board will show the progress of the tasks that are needed to complete user stories. Often these tasks will be kept within horizontal swim lanes - one lane per user story. When all of the tasks are done, the user story will also move into done. Each user story therefore "chases" its tasks across the board. A Kanban board on the other hand - which is meant to deal with smaller and finer-grained pieces of work - will typically track the progress of user stories themselves across the board. The requirements should be well understood and there should be little appreciable depth to the solutioning; there will be few if any explicit tasks associated with the user stories. There is therefore no need for horizontal swim lanes to keep tasks and user stories aligned. You might also notice that Work in Progress limits are given particular emphasis in Lean Kanban. This is because scope is not timeboxed into sprints. The only way to throttle the rate of ticket throughput, and to keep it to manageable levels, is therefore by making sure that WIP limits are rigorously enforced. These are often annotated to the column headers on a Kanban board. For example, if there are 3 developers and 1 tester, the WIP for in progress would be 3, and 1 for in test. "Hey…there's just one backlog" That's right. Since there are no sprints in Lean Kanban, there can be no meaningful separation between a "sprint backlog" and a "product backlog". Instead there's just a single backlog of enqueued work items being brought into progress. This has repercussions for product ownership because you no longer have a clear separation between the prioritization that a team does for itself on a sprint backlog, and the prioritization done by a Product Owner on the product backlog. In effect you've just got a product backlog. In this situation clear product ownership can become more important then ever…or it can become a complete non-issue. "The Product Owner has too much power, he keeps jerking our chain" Since there is only one backlog, the Product Owner (or customer representative) must constantly reprioritize the user stories within it. The Product Owner needs to have more operational control in Lean Kanban than in Scrum. Developers can action tickets from the backlog on a daily or even hourly basis. There is no notion of getting a product backlog in shape before "the next sprint starts". Product Owners are therefore much more closely involved in day-to-day delivery than they would be in Scrum, and their involvement in daily standups becomes much more important. Note that the extent of a Product Owner's decision making should not extend beyond the backlog, and a good Kanban Leader will protect the team and its work in progress just like a good ScrumMaster would. "Now the Product Owner has disappeared altogether" Business as Usual work often boils down to the maintenance of existing systems post-delivery. Depending upon the level of demand, it's quite plausible to have one Lean-Kanban team responsible for the maintenance of multiple systems. In this situation there is no product being delivered as such, and consequently there is no clear product ownership. Instead, work items are raised as change requests and triaged by the team who then manage and prioritize their own backlog. This means that the team needs a strong and shared sense of direction and purpose. "There's no vision for this project" That's because a Lean Kanban operation typically isn't a project at all. A defined end point is likely to be missing… remember that it's covering "Business as Usual work". These are small, repeatable changes that may affect diverse systems and without any sort of narrative to bind them together. There'll certainly be a purpose and a rationale for operating a Lean Kanban… but don't expect a project vision. "We don't even seem to have decent sprint goals any more" Yep, they've gone too. Since there is no project vision and no sprints on a Lean Kanban, we won't have any "sprint goals" either. What we might get is a grouping of work requests that fall within a larger epic of changes…but if we do, it could well be a cause for concern. We must ask: are those related changes really representative of "Business as Usual" work, or are they too high risk? Do they constitute a project? "Lean Kanban work seems very bitty. I can't get a decent chunk to chew on" The diet of a Lean Kanban should consist of small, "digestible" pieces of work that do not require much breaking down in order to action them. By definition they must be well-understood and low-risk. A team must know how to handle them without the need for impact analysis or de-scoping. You're unlikely to get a meaty piece of work; you're more likely to be sucking these things up through a straw. Velocity and lead times are particularly significant metrics in Lean Kanban. Having said that, substantial and time consuming pieces of work can be taken on board if they satisfy the criteria of low risk and clear scope. An example would be the sort of work that conforms to a templated change. Of course, this sort of work might not appeal to an agile developer. So let's be clear: it takes a different temperament to do Lean Kanban BAU work than project work in Scrum. They are different skill sets. Agile developers who are happy doing one can find it unsettling, or even unrewarding, if they are switched to the other. "Why aren't we doing planning poker any more?" Without a sprint backlog there is no budget of story points to be brought into a sprint. This in turn means that estimation exercises such as planning poker lose much of their significance. In a Lean Kanban operation velocity can be measured not in terms of story points - either estimated or actual - but simply as the number of tickets actioned over a set period. This also provides an indication of the lead time before a ticket is handled. If tickets are of too variable a size - for example, if they include small ones as well as larger templated changes - then they can be awarded points for how long, or how much effort, they took. T-Shirt sizes is one approach. Remember that these points should represent the actuals, not estimates, so there's still no need for planning poker. Velocity can be averaged for each size. Alternatively the sizes can be mapped to points (e.g. small = 1, medium = 3, large = 7) and an aggregate velocity calculated. "Some of the BAU work that's been coming through looks like project work to me" You could well be right. It's important that you raise your suspicions with your team lead. There's often politics involved, but here's the lowdown. In many organizations "Business as Usual" work is classed - you could almost say "written off" - as an operational expenditure (OpEx), and is not drawn from the capital expenditure (CapEx) assigned to projects. Internal customers often have an incentive to sneak through initiatives as BAU work so as not to incur capital expense on their departmental budgets. This is indeed a political issue. But be on your guard otherwise your team could be hobbled with project work being slipped in on the sly. Be particularly wary of significant numbers of related changes, large changes, a seemingly high level of risk with any work items, or changes of uncertain scope. These suggest, but do not prove, that a fast one might be being pulled. Your team lead (who is analagous to a ScrumMaster) should try and defend against this, so if you as a team member have your suspicions, it's important to bring them to your lead's attention. Conclusion, and what's next In this post we've looked at the important differences between Lean Kanban and Scrum, and what that means for a team. We've also reviewed how a reasonably informed choice can be made between them. In my next post we'll look at a hybrid approach known as ScrumBan which can potentially address both project and BAU work. ScrumBan is becoming increasingly popular and has significant ramifications for project scalability.

Introduction I’ve talked before about Jetty as an embedded servlet container. Jetty also includes some useful utility servlet implementations, one of which isProxyServlet. ProxyServlet is a way to create an HTTP or HTTP/S proxy in very few lines of code. Even though it’s part of the Jetty project, it’s modularized to be independent of the Jetty server, so you can use it even in cases where the servlet won’t be run in Jetty. Motivation Why might you need a proxy servlet? One reason is to address issues raised by the same origin policy. In general, a script loaded from one site is not allowed to make requests from a different site. While it is possible to work around this (for example using JSONP) I tend to think a proxy is a more elegant solution as it doesn’t require exploiting a hole to download and evaluate arbitrary JavaScript. A proxy might also be useful to allow a user to access a web service without providing all the information necessary to access it. In our example, we’ll be providing a proxy for Google’s Places API without having to send the Google API key down to the browser. The proxy we’ll be looking at is a per-request proxy, so it’s not something that could conveniently be used for caching remote server responses in case of slow connections or server failures. Example The example is part of the Spring WebMVC application I use to present WebMVC and REST for a Java class. I’ve added the PlacesProxyServletand a basic HTML page to demonstrate fetching Google Places search results and using them in jQuery. Maven POM To get started, we need jetty-proxy in our pom.xml. Prior to Jetty 9, theProxyServlet class lived in jetty-servlets, but it’s been moved, probably to reduce the other Jetty dependencies that have to be pulled in. org.eclipse.jetty jetty-proxy ${jetty.version} Java Next, we create a class that extends ProxyServlet. We need to know the right URI to use for Google Places, and we need a Google API key. The best way to handle this is to allow them to be passed in from the servlet context using init-param, but I like to also allow them to be overridden using Java system properties. We start by overriding the init() method: public void init() throws ServletException { super.init(); ServletConfig config = getServletConfig(); placesUrl = config.getInitParameter("PlacesUrl"); apiKey = config.getInitParameter("GoogleApiKey"); // Allow override with system property try { placesUrl = System.getProperty("PlacesUrl", placesUrl); apiKey = System.getProperty("GoogleApiKey", apiKey); } catch (SecurityException e) { } if (null == placesUrl) { placesUrl = "https://maps.googleapis.com/maps/api/place/search/json"; } } To actually proxy the requests, the key method is rewriteURI. Again, this is new to Jetty 9; previously there was a method called proxyHttpURI that accomplished pretty much the same function. protected URI rewriteURI(HttpServletRequest request) { String query = request.getQueryString(); return URI.create(placesUrl + "?" + query + "&key=" + apiKey); } This method returns the “real” URI that the Jetty proxy servlet will call. All of the data from the client request is available. In this case, we just need the browser’s query parameters so we can pass them on to Google Places. Tweaks To actually get this to work with the Google Places API, there were a couple other changes required. First, the Places API enforces HTTP/S. Note that this doesn’t mean that our client has to connect to our proxy servlet using HTTP/S; regular HTTP is perfectly fine for that connection because our proxy servlet is making a brand new HTTP/S connection (using Jetty’s HttpClientclass). However, it does mean that we need to tell the Jetty HttpClient that it’s OK to use HTTP/S. We do this by overriding the method that theProxyServlet class uses to make a new HttpClient: protected HttpClient newHttpClient() { SslContextFactory sslContextFactory = new SslContextFactory(); HttpClient httpClient = new HttpClient(sslContextFactory); return httpClient; } Second, Google Places didn’t like the fact that the Jetty proxy servlet adds aHost header to the request with the name of the originating server. With this header, the Google Places server returns 404 in response to the request. Fortunately, this is easy to fix; we just have to remove that header before the request goes out. We can do this by overriding the customizeProxyRequestmethod that ProxyServlet thoughtfully provides for just such a problem: protected void customizeProxyRequest(Request proxyRequest, HttpServletRequest request) { proxyRequest.getHeaders().remove("Host"); } Updates to web.xml To get this servlet up and running, we need to add it to web.xml. In the case of the example application, this required updating to Servlet 3.0, since the Jetty proxy servlet wants to use asynchronous connections. This is a good thing in terms of increasing the number of simulataneous requests the proxy servlet can process, but it requires enabling that feature in web.xml: PlacesProxy org.anvard.webmvc.server.PlacesProxyServlet GoogleApiKey YOUR_KEY_HERE 1 true PlacesProxy /places The async-supported tag is important; the proxy servlet won’t work without it. Browser interface On the browser side, we need a way to query and then display the results. I cannibalized some example HTML and JavaScript I had lying around that did something similar with CometD. (Unfortunately, I can’t find the original source to provide a linkback.) The relevant jQuery part looks like this: $.getJSON("/places?location=39.016249,-77.122993&radius=1000&types=food&sensor=false", function ( data ) { console.log( data ); for (i = 0; i < data.results.length; i++) { result =data.results[i]; $('').html(result.name + '' + result.vicinity).appendTo('#contentList'); } }) .fail(function() { console.log( "error" ); }) .always(function() { $("#status").text("Complete."); }); The jQuery makes an AJAX call to the proxy servlet, which then makes a call to Google Places. The resulting JSON response data is sent through as-is. The (anonymous) “success” function then gets called. It iterates through the returned results, adding tags to the existing list for each result it finds. Conclusion Of course, a proxy servlet doesn’t have to be used for sites on the Internet. One of my motivations for creating the example application was to show how easy it was to REST-enable an existing standalone Java application. Many systems that use Java have multiple standalone Java applications, each performing some independent function. This would make it challenging to create a single unified web interface while still allowing each application to define its own REST API. Proxy servlets can help by making it look like there’s a single endpoint for all the various APIs, while not requiring any logic that knows about the contents of the interfaces.

There is a common misconception that if you have unique hashCode() you won't have collisions. While unique, or almost unique, hashCodes are good, this is not the end of the story. The problem is that the size of a HashMap is not unlimited (or at least 2^32 in size) This means the hashCode() number has to be reduced to a smaller number of bits. The way HashMap, and thus HashSet and LinkedHashMap ,work is to mutate the bits in the following manner h ^= (h >>> 20) ^ (h >>> 12); return h ^ (h >>> 7) ^ (h >>> 4); and then apply a mask for the lowest bits to select a bucket. The problem is that even with unique hashCode()s as Integer does, there will be values with different hash code map to the same bucket. You can research how Integer.hashCode() works ;) public static void main(String[] args) { Set integers = new HashSet<>(); for (int i = 0; i <= 400; i++) if ((hash(i) & 0x1f) == 0) integers.add(i); Set integers2 = new HashSet<>(); for (int i = 400; i >= 0; i--) if ((hash(i) & 0x1f) == 0) integers2.add(i); System.out.println(integers); System.out.println(integers2); } static int hash(int h) { // This function ensures that hashCodes that differ only by // constant multiples at each bit position have a bounded // number of collisions (approximately 8 at default load factor). h ^= (h >>> 20) ^ (h >>> 12); return h ^ (h >>> 7) ^ (h >>> 4); } this prints [373, 343, 305, 275, 239, 205, 171, 137, 102, 68, 34, 0] [0, 34, 68, 102, 137, 171, 205, 239, 275, 305, 343, 373] The entries as in the reverse order they were added as the HashMap is acting as a linked list, placing all entries into the same bucket. Solutions? A simple solution is to have a bucket turn into a tree instead of a linked list. In Java 8, it will do this for String keys, but this could be done for all Comparable types AFAIK. Another approach is to allow custom hashing strategies to allow the developer to avoid such problems, or to randomize the mutation on a per collection basis, amortizing the cost to the application. Other notes I would favour supporting 64-bit hash codes, esp for complex objects. This has a very low chance of collision in the hash code itself and supports very large data structures well. e.g. into the billions.

I’ve been doing some work that involves reading in CSV files, for which I’ve been using OpenCSV, and my initial approach was to read through the file line by line, parse the contents, and save it into a list of maps. This works when the contents of the file fit into memory, but is problematic for larger files where I needed to stream the file and process each line individually, rather than all of them after the file was loaded. I initially wrote a variation on totallylazy’s Strings#lines to do this, and while I was able to stream the file, I made a mistake somewhere which meant the number of maps on the heap was always increasing. After spending a few hours trying to fix this, Michael suggested that it’d be easier to use an iterator instead, and I ended up with the following code: public class ParseCSVFile { public static void main(String[] args) throws IOException { final CSVReader csvReader = new CSVReader( new BufferedReader( new FileReader( "/path/to/file.csv" ) ), '\t' ); final String[] fields = csvReader.readNext(); Iterator>() lazilyLoadedFile = return new Iterator>() { String[] data = csvReader.readNext(); @Override public boolean hasNext() { return data != null; } @Override public Map next() { final Map properties = new HashMap(); for ( int i = 0; i < data.length; i++ ) { properties.put(fields[i], data[i]); } try { data = csvReader.readNext(); } catch ( IOException e ) { data = null; } return properties; } @Override public void remove() { throw new UnsupportedOperationException(); } }; } } Although this code works, it’s not the most readable function I’ve ever written, so any suggestions on how to do this in a cleaner way are welcome.

With these changes, we can access the REST API equally well fromhttp://:9999 and https://:9998.

I have recently started the Coursera Functional Programming with Scala course (taught by Martin Odersky - the creator of Scala) - which is actually serving as an introduction to both FP and Scala at the same time having done neither before. The course itself is great, however, trying to watch the videos and take in both the new Scala syntax and the FP concepts at the same time can take a bit of effort. I wanted to work through some of the core FP concepts in a more familiar context, so am going to apply some of the lessons/principles/exercises in Groovy. Functions: If you have done any Groovy programming then you will have come across Groovy functions/closures. As Groovy is dynamically typed (compared to Scala's static typing), you can play it fairly fast and loose. For example, if we take the square root function that is demonstrated in the Scala course, it is defined as follows: def sqrt(x: Double): Double = ... As you can see, Scala expects the values to be typed (aside, you don't actually always need to provide a return type in Scala). But in the Groovy function it is: def sqrt = {x-> ... } A groovy function can be defined and assigned to any variable, thereby allowing it to be passed around as a first class object. If we look at the complete solution for calculating the square root of a number (using Newton's method - To compute the square root of "x" we start with an estimate of the square root, "y" and continue to improve the the guess by taking the mean of x and x/y ) def sqrtIter(guess: Double, x: Double): Double = if (isGoodEnough(guess, x)) guess else sqrtIter(improve(guess, x), x) def improve(guess: Double, x: Double) = (guess + x / guess) / 2 def isGoodEnough(guess: Double, x: Double) = abs(guess * guess - x) < 0.001 def sqrt(x: Double) = srqtIter(1.0, x) So that's in Scala, if we try Groovy we will see we can achieve pretty much the same thing easily: //Improve the guess using Newton's method def improve = { guess, x -> (guess + x / guess) / 2 } //Check if our guess is good enough, within a chosen threshold def isGoodEnough = {guess, x -> abs(guess * guess - x) < 0.001 } //iterate over guesses until our guess is good enough def sqrtIter = { guess, x -> if (isGoodEnough(guess, x)) guess else sqrtIter(improve(guess, x), x) } //wrap everythin in the square root function call def sqrt = {x -> srqtIter(1.0, x)} Recursion (and tail-recursion): As FP avoids having mutable state, the most common approach to solve problems is to break the problem down in to simple functions and call them recursively - This avoids having to maintain state whilst iterating through loops, and each function call is given its input and produces an output. If we again consider an example from the Scala course, with the example of a simple function that calculates the factorial for a given number. def factorial ={ n -> if (n == 0) 1 else n * factorial(n - 1) } factorial(4) This simple function recursively calculates the factorial, continuing to call itself until all numbers to zero have been considered. As you can see, there is no mutable state - every call to the factorial function simply takes the input and returns an output (the value of n is never changed or re-assigned, n is simply used to calculate output values) There is a problem here, and that is as soon as you attempt to calculate the factorial of a significantly large enough number you will encounter a StackOverflow exception - this is because in the JVM every time a function is called, a frame is added to the stack, so working recursively its pretty easy to hit upon the limit of the stack and encounter this problem. The common way to solve this is by using Tail-Call recursion. This trick is simply to have the last code that is evaluated in the function to be the recursive call - normally in FP languages the compiler/interpreter will recognise this pattern and under the hood, it will really just run the code as a loop (e.g. if we know the very last piece of code in the block of code is calling itself, its really not that different to just having the block of code/function inside a loop construct) In the previous factorial example, it might look like the last code to be executed is the recursive callfactorial(n-1) - however, the value of that call is actually returned to the function and THENmultiplied by n - so actually the last piece of code to be evaluated in the function call is actually n * return value of factorial(n-1). Let's have a look at re-writing the function so it is tail-recursive. def factorial ={ n, accumulator=1 -> if (n == 1) accumulator else factorial(n-1, n*accumulator) } factorial(4) Now, using an accumulator, the last code to be evaluated in the function is our recursive function call. In most FP languages, including Scala, this is enough - however, the JVM doesn't automatically support tail-call recursion, so you actually need to use a rather clunkier approach in Groovy: def factorial ={ n, accumulator=1 -> if (n == 1) accumulator else factorial.trampoline(n-1, n*accumulator) }.trampoline() factorial(4) The use of the trampoline() method means that the function will now be called using tail-call recursion, so there should never be a StackOverflow exception. It's not as nice as in Scala or other languages, but the support is there so we can continue. Currying: This is like function composition - the idea being you take a generic function, and then you curry it with some value to make a more specific application of the function. For example, if we look at a function that given values x and y, it returns z which is the value x percent of y (e.g. given x=10, y=100, it returns the 10 percent of 100, z=10) def percentage = { percentage, x -> x/100 * percentage } The above simple function is a generic mechanism to get a percentage value of another, but if we consider that we wanted a common application of this function was to always calculate 10% of a given value - rather than write a slightly modified version of the function we can simply curry the function as follows: def tenPercent = percentage.curry(10) Now, if the function tenPercent(x) is called, it uses the original percentage() function, but curries the value 10 as the first argument. (If you need to curry other argument positions you can also use the rcurry() function to curry the right most argument, or ncurry() which also takes an argument position - check the Groovy docs on currying for more info) Immutability: Immutability is partially supported in Java normally with use of the final keyword (meaning variables can't be changed after being initially set on object instantiation). Groovy also provides a quick and easy @Immutable annotation that can be added to a class to easily make it immutable. But really, there is more to avoiding immutable state than just having classes as immutable - As we have functions as first class objects, we can easily assign variables and mutate them within a function - so this is more of a mindset or philosophy that you have to get used to. For example: def list = ['groovy', 'functional'] //This mutates the original list list.add('programming') //This creates a new list leaving the original unchanged def newList = list.plus(2, 'programming') The first example is probably more like the Groovy/Java code we are used to writing, but that is mutating the state of the list - where as the second approach leaves the original list unchanged. Map Reduce: As a final note, there are some functions in FP that are pretty common techniques - the most famous of which these days (in part thanks to Google) is Map-Reduce, but the trio of functions are actually Map, Reduce(also known as Fold) & Filter - you can read more about the functions here (or just google them!), but these functions actually correlate pretty nicely to core Groovy functions that you probably use a lot of (assuming you are groovy programmers). Map map is the easiest to understand of the three. It takes in two inputs - a function, and a list. It then applies this function to every element in the list. You can basically do the same thing with a list comprehension however. Sound familiar? This is basically the .collect{} function in Groovy Reduce/Fold fold takes in a function and folds it in between the elements of a list. It's a bit hard to understand at first This one is a bit more complicated to descibe, but is the same as the .inject{} function in groovy Filter filter is easy. It takes in a 'test' and a list, and it chucks out any elements of the list which don't satisfy that test. And another simple one - filtering out a list for desired elements, this is Groovy's .findAll{} function As I said at the start, I am new to FP and coming from an OO background, but hopefully the above isn't too far from the truth! As I get further through the Coursera course I will try to post again, maybe with some of the assignments attempted in Groovy to see how it really stands up. Some useful references: Groovy docs on FP: http://groovy.codehaus.org/Functional+Programming+with+Groovy Coursera Scala course: https://www.coursera.org/course/progfun

The Windows Explorer (shell) provides extended file property information which can be quite valuable. The challenge was how to extract this information, given that the .NET Framework has somewhat limited support for this type of extraction?

Recently I needed to quickly create some temporary JSON objects from some preexisting data. So, I turned to Json.NET, a very popular JSON framework for .NET. Since I got my data on the fly (think parsing some other data structures and you have lots of anonymous results), I didn’t want to create classes for such instances. Luckily, Json.NET supports serializing anonymous objects. First add the library via NuGet – it is called Newtonsoft.Json. For example: var obj = new { user = new { name = "John", age = 21, data = new[] { 1, 2, 3, 4 } } }; var result = JsonConvert.SerializeObject(obj, Formatting.Indented); If we did that in a console application, we could print out the result. The printout is: { "user": { "name": "John", "age": 21, "data": [ 1, 2, 3, 4 ] } } The code above works on Windows 8, Windows Phone 7 and 8. This way it is dead simple to create JSON representation of your anonymous data. Think REST services.

this post comes from ernie souhrada at the mysql performance blog. note: this is part 1 of what will be a two-part series on the performance implications of using in-flight data encryption. some of you may recall my security webinar from back in mid-august; one of the follow-up questions that i was asked was about the performance impact of enabling ssl connections. my answer was 25%, based on some 2011 data that i had seen over on yassl’s website, but i included the caveat that it is workload-dependent, because the most expensive part of using ssl is establishing the connection. not long thereafter, i received a request to conduct some more specific benchmarks surrounding ssl usage in mysql, and today i’m going to show the results. first, the testing environment. all tests were performed on an intel core i7-2600k 3.4ghz cpu (8 cores, ht included) with 32gb of ram and centos 6.4. the disk subsystem is a 2-disk raid-0 of samsung 830 ssds, although since we’re only concerned with measuring the overhead added by using ssl connections, we’ll only be conducting read-only tests with a dataset that fits completely in the buffer pool. the version of mysql used for this experiment is community edition 5.6.13, and the testing tools are sysbench 0.5 and perl. we conduct two tests, each one designed to simulate one of the most common mysql usage patterns. first, we examine connection pooling, often seen in the java world, where some small set of connections are established by, for example, the servlet container and then just passed around to the application as needed, and one-request-per-connection, typical in the lamp world, where the script that displays a given page might connect to the database, run a couple of queries, and then disconnect. test 1: connection pool for the first test, i ran sysbench in read-only mode at concurrency levels of 1, 2, 4, 8, 16, and 32 threads, first with no encryption and then with ssl enabled and key lengths of 1024, 2048, and 4096 bits. 8 sysbench tables were prepared, each containing 100,000 rows, resulting in a total data size of approximately 256mb. the size of my innodb buffer pool was 4gb, and before conducting each official measurement run, i ran a warm-up run to prime the buffer pool. each official test run lasted 10 minutes; this might seem short, but unlike, say, a pcie flash storage device, i would not expect the variable under observation to really change that much over time or need time to stabilize. the basic sysbench syntax used is shown below. #!/bin/bash for ssl in on off ; do for threads in 1 2 4 8 16 32 ; do sysbench --test=/usr/share/sysbench/oltp.lua --mysql-user=msandbox$ssl --mysql-password=msandbox \ --mysql-host=127.0.0.1 --mysql-port=5613 --mysql-db=sbtest --mysql-ssl=$ssl \ --oltp-tables-count=8 --num-threads=$threads --oltp-dist-type=uniform --oltp-read-only=on \ --report-interval=10 --max-time=600 --max-requests=0 run > sb-ssl_${ssl}-threads-${threads}.out done done if you’re not familiar with sysbench, the important thing to know about it for our purposes is that it does not connect and disconnect after each query or after each transaction. it establishes n connections to the database (where n is the number of threads) and runs queries though them until the test is over. this behavior provides our connection-pool simulation. the assumption, given what we know about where ssl is the slowest, is that the performance penalty here should be the lowest. first, let’s look at raw throughput, measured in queries per second: the average throughput and standard deviation (both measured in queries per second) for each test configuration is shown below in tabular format: # of threads ssl key size 1 2 4 8 16 32 ssl off 9250.18 (1005.82) 18297.61 (689.22) 33910.31 (446.02) 50077.60 (1525.37) 49844.49 (934.86) 49651.09 (498.68) 1024-bit 2406.53 (288.53) 4650.56 (558.58) 9183.33 (1565.41) 26007.11 (345.79) 25959.61 (343.55) 25913.69 (192.90) 2048-bit 2448.43 (290.02) 4641.61 (510.91) 8951.67 (1043.99) 26143.25 (360.84) 25872.10 (324.48) 25764.48 (370.33) 4096-bit 2427.95 (289.00) 4641.32 (547.57) 8991.37 (1005.89) 26058.09 (432.86) 25990.13 (439.53) 26041.27 (780.71) so, given that this is an 8-core machine and io isn’t a factor, we would expect throughput to max out at 8 threads, so the levelling-off of performance is expected. what we also see is that it doesn’t seem to make much difference what key length is used, which is also largely expected. however, i definitely didn’t think the encryption overhead would be so high. the next graph here is 95th-percentile latency from the same test: and in tabular format, the raw numbers (average and standard deviation): # of threads ssl key size 1 2 4 8 16 32 ssl off 1.882 (0.522) 1.728 (0.167) 1.764 (0.145) 2.459 (0.523) 6.616 (0.251) 27.307 (0.817) 1024-bit 6.151 (0.241) 6.442 (0.180) 6.677 (0.289) 4.535 (0.507) 11.481 (1.403) 37.152 (0.393) 2048-bit 6.083 (0.277) 6.510 (0.081) 6.693 (0.043) 4.498 (0.503) 11.222 (1.502) 37.387 (0.393) 4096-bit 6.120 (0.268) 6.454 (0.119) 6.690 (0.043) 4.571 (0.727) 11.194 (1.395) 37.26 (0.307) with the exception of 8 and 32 threads, the latency introduced by the use of ssl is constant at right around 5ms, regardless of the key length or the number of threads. i’m not surprised that there’s a large jump in latency at 32 threads, but i don’t have an immediate explanation for the improvement in the ssl latency numbers at 8 threads. test 2: connection time for the second test, i wrote a simple perl script to just connect and disconnect from the database as fast as possible. we know that it’s the connection setup which is the slowest part of ssl, and the previous test already shows us roughly what we can expect for ssl encryption overhead for sending data once the connection has been established, so let’s see just how much overhead ssl adds to connection time. the basic script to do this is quite simple (non-ssl version shown): #!/usr/bin/perl use dbi; use time::hires qw(time); $start = time; for (my $i=0; $i<100; $i++) { my $dbh = dbi->connect("dbi:mysql:host=127.0.0.1;port=5613", "msandbox","msandbox",undef); $dbh->disconnect; undef $dbh; } printf "%.6f\n", time - $start; as with test #1, i ran test #2 with no encryption and ssl encryption of 1024, 2048, and 4098 bits, and i conducted 10 trials of each configuration. then i took the elapsed time for each test and converted it to connections per second. the graph below shows the results from each run: here are the averages and standard deviations: encryption average connections per second standard deviation none 2701.75 165.54 1024-bit 77.04 6.14 2048-bit 28.183 1.713 4096-bit 5.45 0.015 yes, that’s right, 4096-bit ssl connections are 3 orders of magnitude slower to establish than unencrypted connections. really, the connection overhead for any level of ssl usage is quite high when compared to the unencrypted test, and it’s certainly much higher than my original quoted number of 25%. analysis and parting thoughts so, what do we take away from this? the first thing is, of course, is that ssl overhead is a lot higher than 25%, particularly if your application uses anything close to the one-connection-per-request pattern. for a system which establishes and maintains long-running connections, the initial connection overhead becomes a non-factor, regardless of the encryption strength, but there’s still a rather large performance penalty compared to the unencrypted connection. this leads directly into the second point, which is that connection pooling is by far a more efficient method of using ssl if your application can support it. but what if connection pooling isn’t an option, mysql’s ssl performance is insufficient, and you still need full encryption of data in-flight? run the encryption component of your system at a lower layer – a vpn with hardware crypto would be the fastest approach, but even something as simple as an ssh tunnel or openvpn *might* be faster than ssl within mysql. i’ll be exploring some of these solutions in a follow-up post. and finally… when in doubt, run your own benchmarks. i don’t have an explanation for why the yassl numbers are so different from these (maybe yassl is a faster ssl library than openssl, or maybe they used a different cipher – if you’re curious, the original 25% number came from slides 56-58 of this presentation ), but in any event, this does illustrate why it’s important to run tests on your own hardware and with your own workload when you’re interested in finding out how well something will perform rather than taking someone else’s word for it.

qunit , used by projects like jquery and jquery mobile , is a rather popular javascript testing framework. for tests written using qunit, how do we measure its code coverage ? a possible solution which is quite easy to setup is to leverage the deadly combination of karma and istanbul . just like our previous adventure with jasmine code coverage , let's take a look at some simple code we need to test. this function my.sqrt is a reimplementation of math.sqrt which may throw an exception if the input is invalid. var my = { sqrt: function(x) { if (x < 0) throw new error("sqrt can't work on negative number"); return math.exp(math.log(x)/2); } }; a very simple qunit-based test for the above code is as follows. test("sqrt", function() { deepequal(my.sqrt(4), 2, "square root of 4 is 2"); }); manually running the test is easy as opening the test runner in a web browser: for a smoothed development workflow, an automated way to run the tests will be much preferred. this is where karma becomes very useful. karma also has the ability to launch a predetermined collection of browsers, or even to use phantomjs for a pure headless execution (suitable for smoke testing and/or continuous delivery). before we can use karma, installation is necessary: npm install karma karma-qunit karma-coverage karma requires a configuration file. for this purpose, the config file is very simple. as an illustration, the execution is done by phantomjs but it is easy to include other browsers as well. module.exports = function(config) { config.set({ basepath: '', frameworks: ['qunit'], files: [ '*.js', 'test/spec/*.js' ], browsers: ['phantomjs'], singlerun: true, reporters: ['progress', 'coverage'], preprocessors: { '*.js': ['coverage'] } }); }; now you can start karma with the above configuration, it would say that the test passes just fine. should you encounter some problems, you can look at an example repository i have setup github.com/ariya/coverage-qunit-istanbul-karma , it may be useful as a starting point or a reference for your own project. as a convenience, the test in that repository can be executed via npm test . what is more interesting here is that karma runs its coverage processor, as indicated by preprocessors in the above configuration. karma will run istanbul , a full-featured instrumenter and coverage tracker. essentially, istanbul grabs the original javascript source and injects extra instrumentation code so that it can gather the execution metrics once the process finishes (read also my previous blog post on javascript code coverage with istanbul ). in this karma and istanbul combo, the generated coverage report is available in the under the subdirectory coverage . the above report indicates that the single test for my.sqrt is still missing the test for an invalid input, thanks to branch coverage feature of istanbul. the i indicator next to the conditional statement tells us that the if branch was never taken. of course, once the issue is known, adding another test which will cover that branch is easy (left as an exercise for the reader). now that code coverage is tracker, perhaps you are ready for the next level? it is about setting the hard threshold so that future coverage regression will never happen. protect yourself and your team from carelessness, overconfidence, or honest mistakes!

in today’s tutorial, i’m going to show you the process of creating nearly an entire website with a new library – angularjs. however, i would like to introduce to you to angularjs first. angularjs is a magnificent framework by google for creating web applications. this framework lets you extend html’s syntax to express your application’s components clearly and succinctly, and lets you use standard html as your main template language. plus, it automatically synchronizes data from your ui with your js objects through 2-way data binding. if you’ve ever worked with jquery, the first thing to understand about angular is that this is a completely different instrument. jquery is a library, but angularjs is framework. when your code works with the library, it decides when to call a particular function or operator. in the case of the framework, you implement event handlers, and the framework decides at what moment it needs to invoke them. using this framework allows us to clearly distinguish between templates (dom), models, and functionality (in controllers). let’s come back to our template, take a look at our result: live demo download in package description this template is perfect for business sites. it consists of several static pages: projects, privacy, and about pages. each product has its own page. there is also a contact form for communication. that is all that is necessary for any small website. moreover, it is also a responsive template, thus it looks good on any device. i hope you liked the demo, so if you’re ready – let’s start making this application. please prepare a new folder for our project, and then create new folders in this directory: css – for stylesheet files images – for image files js – for javascript files (libraries, models, and controllers) pages – for internal pages stage 1. html the main layout consists of four main sections: a header with navigation, a hidden ‘contact us’ form, a main content section, and a footer. first we have to prepare a proper header: index.html as you can see, it’s an ordinary header. now – the header with the navigation: our projectsprivacy & termsaboutcontact us it's an ordinary logo, and the menu is the usual ul-li menu. the next section is more interesting – the ‘contact us’ form: contact us send message your message has been sent, thank you. finally, the last key element is the main content section: have you noticed the numerous ‘ng-’ directives? all these directives allow us to do various actions directly in the dom, for example: ng-class – the ngclass allows you to set css classes on an html element dynamically by databinding an expression that represents all classes to be added. ng-click – the ngclick allows you to specify custom behavior when element is clicked. ng-hide – the nghide directive shows and hides the given html element conditionally based on the expression provided to the nghide attribute. ng-include – fetches, compiles, and includes an external html fragment. ng-model – is a directive that tells angular to do two-way data binding. ng-show – the ngshow directive shows and hides the given html element conditionally based on the expression provided to the ngshow attribute. ng-submit – enables binding angular expressions to onsubmit events. stage 2. css in this rather large section, you can find all the styles used: css/style.css /* general settings */ html { min-height:100%; overflow-x:hidden; overflow-y:scroll; position:relative; width:100%; } body { background-color:#e6e6e6; color:#fff; font-weight:100; margin:0; min-height:100%; width:100%; } a { text-decoration:none; } a img { border:none; } h1 { font-size:3.5em; font-weight:100; } p { font-size:1.5em; } input,textarea { -webkit-appearance:none; background-color:#f7f7f7; border:none; border-radius:3px; font-size:1em; font-weight:100; } input:focus,textarea:focus { border:none; outline:2px solid #7ed7b9; } .left { float:left; } .right { float:right; } .btn { background-color:#fff; border-radius:24px; color:#595959; display:inline-block; font-size:1.4em; font-weight:400; margin:30px 0; padding:10px 30px; text-decoration:none; } .btn:hover { opacity:0.8; } .wrap { -moz-box-sizing:border-box; -webkit-box-sizing:border-box; box-sizing:border-box; margin:0 auto; max-width:1420px; overflow:hidden; padding:0 50px; position:relative; width:100%; } .wrap:before { content:''; display:inline-block; height:100%; margin-right:-0.25em; vertical-align:middle; } /* header section */ header { height:110px; } header .wrap { height:100%; } header .logo { margin-top:1px; } header nav { float:right; margin-top:17px; } header nav ul { margin:1em 0; padding:0; } header nav ul li { display:block; float:left; margin-right:20px; } header nav ul li a { border-radius:24px; color:#aaa; font-size:1.4em; font-weight:400; padding:10px 27px; text-decoration:none; } header nav ul li a.active { background-color:#c33c3a; color:#fff; } header nav ul li a.active:hover { background-color:#d2413f; color:#fff; } header nav ul li a:hover,header nav ul li a.activesmall { color:#c33c3a; } /* footer section */ footer .copyright { color:#adadad; margin-bottom:50px; margin-top:50px; text-align:center; } /* other objects */ .projectobj { color:#fff; display:block; } .projectobj .name { float:left; font-size:4em; font-weight:100; position:absolute; width:42%; } .projectobj .img { float:right; margin-bottom:5%; margin-top:5%; width:30%; } .paddrow { background-color:#dadada; color:#818181; display:none; padding-bottom:40px; } .paddrow.aboutrow { background-color:#78c2d4; color:#fff !important; display:block; } .paddrow .head { font-size:4em; font-weight:100; margin:40px 0; } .paddrow .close { cursor:pointer; position:absolute; right:50px; top:80px; width:38px; } .about { color:#818181; } .about section { margin:0 0 10%; } .about .head { font-size:4em; font-weight:100; margin:3% 0; } .about .subhead { font-size:2.5em; font-weight:100; margin:0 0 3%; } .about .txt { width:60%; } .about .image { width:26%; } .about .flleft { float:left; } .about .flright { float:right; } .projecthead.product { background-color:#87b822; } .projecthead .picture { margin-bottom:6%; margin-top:6%; } .projecthead .picture.right { margin-right:-3.5%; } .projecthead .text { position:absolute; width:49%; } .projecthead .centertext { margin:0 auto; padding-bottom:24%; padding-top:6%; text-align:center; width:55%; } .image { text-align:center; } .image img { vertical-align:top; width:100%; } .contactform { width:50%; } .input { -moz-box-sizing:border-box; -webkit-box-sizing:border-box; box-sizing:border-box; margin:1% 0; padding:12px 14px; width:47%; } .input.email { float:right; } button { border:none; cursor:pointer; } .textarea { -moz-box-sizing:border-box; -webkit-box-sizing:border-box; box-sizing:border-box; height:200px; margin:1% 0; overflow:auto; padding:12px 14px; resize:none; width:100%; } ::-webkit-input-placeholder { color:#a7a7a7; } :-moz-placeholder { color:#a7a7a7; } ::-moz-placeholder { /* ff18+ */ color:#a7a7a7; } :-ms-input-placeholder { color:#a7a7a7; } .loader { -moz-animation:loader_rot 1.3s linear infinite; -o-animation:loader_rot 1.3s linear infinite; -webkit-animation:loader_rot 1.3s linear infinite; animation:loader_rot 1.3s linear infinite; height:80px; width:80px; } @-moz-keyframes loader_rot { from { -moz-transform:rotate(0deg); } to { -moz-transform:rotate(360deg); } } @-webkit-keyframes loader_rot { from { -webkit-transform:rotate(0deg); } to { -webkit-transform:rotate(360deg); } } @keyframes loader_rot { from { transform:rotate(0deg); } to { transform:rotate(360deg); } } .view-enter,.view-leave { -moz-transition:all .5s; -o-transition:all .5s; -webkit-transition:all .5s; transition:all .5s; } .view-enter { left:20px; opacity:0; position:absolute; top:0; } .view-enter.view-enter-active { left:0; opacity:1; } .view-leave { left:0; opacity:1; position:absolute; top:0; } .view-leave.view-leave-active { left:-20px; opacity:0; } please note that css3 transitions are used, which means that our demonstration will only work modern browsers (ff, chrome, ie10+ etc) stage 3. javascript as i mentioned before, our main controller and the model are separated. the navigation menu can be handled here, and we also can operate with the contact form. js/app.js 'use strict'; // angular.js main app initialization var app = angular.module('example359', []). config(['$routeprovider', function ($routeprovider) { $routeprovider. when('/', { templateurl: 'pages/index.html', activetab: 'projects', controller: homectrl }). when('/project/:projectid', { templateurl: function (params) { return 'pages/' + params.projectid + '.html'; }, controller: projectctrl, activetab: 'projects' }). when('/privacy', { templateurl: 'pages/privacy.html', controller: privacyctrl, activetab: 'privacy' }). when('/about', { templateurl: 'pages/about.html', controller: aboutctrl, activetab: 'about' }). otherwise({ redirectto: '/' }); }]).run(['$rootscope', '$http', '$browser', '$timeout', "$route", function ($scope, $http, $browser, $timeout, $route) { $scope.$on("$routechangesuccess", function (scope, next, current) { $scope.part = $route.current.activetab; }); // onclick event handlers $scope.showform = function () { $('.contactrow').slidetoggle(); }; $scope.closeform = function () { $('.contactrow').slideup(); }; // save the 'contact us' form $scope.save = function () { $scope.loaded = true; $scope.process = true; $http.post('sendemail.php', $scope.message).success(function () { $scope.success = true; $scope.process = false; }); }; }]); app.config(['$locationprovider', function($location) { $location.hashprefix('!'); }]); pay attention here. when we request a page, it loads an appropriate page from the ‘pages’ folder: about.html, privacy.html, index.html. depending on the selected product, it opens one of the product pages: product1.html, product2.html, product3.html or product4.html in the second half, there are functions to slide the contact form and to handle its submit process (to the sendemail.php page). next is the controller file: js/controllers.js 'use strict'; // optional controllers function homectrl($scope, $http) { } function projectctrl($scope, $http) { } function privacyctrl($scope, $http, $timeout) { } function aboutctrl($scope, $http, $timeout) { } it is empty, because we have nothing to use here at the moment. stage 4. additional pages angularjs loads pages asynchronously, thereby increasing the speed. here are templates of all additional pages used in our project: pages/about.html about us script tutorials is one of the largest web development communities. we provide high quality content (articles and tutorials) which covers all the web development technologies including html5, css3, javascript (and jquery), php and so on. our audience are web designers and web developers who work with web technologies. additional information promo 1 lorem ipsum dolor sit amet, consectetur adipiscing elit. nunc et ligula accumsan, pharetra nibh nec, facilisis nulla. in pretium semper venenatis. in adipiscing augue elit, at venenatis enim suscipit a. fusce vitae justo tristique, ultrices mi metus. ..... pages/privacy.html privacy & terms by accessing this web site, you are agreeing to be bound by these web site terms and conditions of use, all applicable laws and regulations, and agree that you are responsible for compliance with any applicable local laws. if you do not agree with any of these terms, you are prohibited from using or accessing this site. the materials contained in this web site are protected by applicable copyright and trade mark law. other information header 1 lorem ipsum dolor sit amet, consectetur adipiscing elit. nunc et ligula accumsan, pharetra nibh nec, facilisis nulla. in pretium semper venenatis. in adipiscing augue elit, at venenatis enim suscipit a. fusce vitae justo tristique, ultrices mi metus. ..... pages/footer.html copyright © 2013 script tutorials pages/index.html product #1 product #2 product #3 product #4 finally, the product pages. all of them are prototypes, so i decided to publish only one of them. pages/index.html product 1 page lorem ipsum dolor sit amet, consectetur adipiscing elit. nunc et ligula accumsan, pharetra nibh nec, facilisis nulla. in pretium semper venenatis. in adipiscing augue elit, at venenatis enim suscipit a. fusce vitae justo tristique, ultrices mi metus. lorem ipsum dolor sit amet, consectetur adipiscing elit. nunc et ligula accumsan, pharetra nibh nec, facilisis nulla. in pretium semper venenatis. in adipiscing augue elit, at venenatis enim suscipit a. fusce vitae justo tristique, ultrices mi metus. download the app finishing touches – responsive styles all of these styles are needed to make our results look equally good on all possible mobile devices and monitors: @media (max-width: 1200px) { body { font-size:90%; } h1 { font-size:4.3em; } p { font-size:1.3em; } header { height:80px; } header .logo { margin-top:12px; width:200px; } header nav { margin-top:11px; } header nav ul li { margin-right:12px; } header nav ul li a { border-radius:23px; font-size: 1.3em; padding:10px 12px; } .wrap { padding:0 30px; } .paddrow .close { right:30px; } } @media (max-width: 900px) { .contactform { width:100%; } } @media (max-width: 768px) { body { font-size:80%; margin:0; } h1 { font-size:4em; } header { height:70px; } header .logo { margin-top:20px; width:70px; } header nav { margin-top:8px; } header nav ul li { margin-right:5px; } header nav ul li a { border-radius:20px; font-size:1.1em; padding:8px; } .wrap { padding:0 15px; } .projectobj .name { font-size:3em; } .paddrow { padding-bottom:30px; } .paddrow .head { font-size:3em; margin:30px 0; } .paddrow .close { right:20px; top:60px; width:30px; } .projecthead .picture { width:67%; } .projecthead .picture.right { margin-right:16.5%; } .projecthead .text { position:static; width:100%; } .projecthead .centertext { width:70%; } .view-enter,.view-leave { -webkit-transform:translate3d(0,0,0); transform:translate3d(0,0,0); } } @media (max-width: 480px) { body { font-size:70%; margin:0; } header { height:50px; } header .logo { display:none; } header nav { margin-top:3px; } header nav ul li { margin-right:3px; } header nav ul li a { border-radius:20px; font-size:1.3em; padding:5px 14px; } #contactbtn { display:none; } .wrap { padding:0 10px; } .paddrow { padding-bottom:20px; } .paddrow .head { margin:20px 0; } .paddrow .close { right:10px; top:45px; width:20px; } .about .image { margin:10% auto; width:60%; } .about .abicon { display:inline; } .projecthead .centertext { width:90%; } .about .txt,.input { width:100%; } .about .flleft,.about .flright,.input.email { float:none; } } live demo download in package conclusion that’s all for today. thanks for your patient attention, and if you really like what we did today, share it with all your friends in your social networks.

The following question will again test your knowledge of the Oracle Weblogic threading model. I’m looking forward for your comments and experience on the same. If you are a Weblogic administrator, I’m certain that you heard of this common problem: stuck threads. This is one of the most common problems you will face when supporting a Weblogic production environment. A Weblogic stuck thread simply means a thread performing the same request for a very long time and more than the configurable Stuck Thread Max Time. Question: How can you detect the presence of STUCK threads during and following a production incident? Answer: As we saw from our last article “Weblogic Thread Monitoring Tips”, Weblogic provides functionalities allowing us to closely monitor its internal self-tuning thread pool. It will also highlight you the presence of any stuck thread. This monitoring view is very useful when you do a live analysis but what about after a production incident? The good news is that Oracle Weblogic will also log any detected stuck thread to the server log. Such information includes details on the request and more importantly, the thread stack trace. This data is crucial and will allow you to potentially better understand the root cause of any slowdown condition that occurred at a certain time. < ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'> <[STUCK] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "608" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 608213 ms POST /App1/jsp/test.jsp HTTP/1.1 Accept: application/x-ms-application... Referer: http://.. Accept-Language: en-US User-Agent: Mozilla/4.0 .. Content-Type: application/x-www-form-urlencoded Accept-Encoding: gzip, deflate Content-Length: 539 Connection: Keep-Alive Cache-Control: no-cache Cookie: JSESSIONID= ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace: ................................... javax.servlet.http.HttpServlet.service(HttpServlet.java:727) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:184) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.... weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run() weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2281) weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2180) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491) weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) weblogic.work.ExecuteThread.run(ExecuteThread.java:221) Here is one more tip: the generation and analysis of a JVM thread dump will also highlight you stuck threads. As we can see from the snapshot below, the Weblogic thread state is now updated to STUCK, which means that this particular request is being executed since at least 600 seconds or 10 minutes. This is very useful information since the native thread state will typically remain to RUNNABLE. The native thread state will only get updated when dealing with BLOCKED threads etc. You have to keep in mind that RUNNABLE simply means that this thread is healthy from a JVM perspective. However, it does not mean that it truly is from a middleware or Java EE container perspective. This is why Oracle Weblogic has its own internal ExecuteThread state. Finally, if your organization or client is using any commercial monitoring tool, I recommend that you enable some alerting around both hogging thread and stuck thread. This will allow your support team to take some pro-active actions before the affected Weblogic managed server(s) become fully unresponsive.

for modern web application development, having dozens of unit tests is not enough anymore. the actual code coverage of those tests would reveal if the application is thoroughly stressed or not. for tests written using the famous jasmine test library, an easy way to have the coverage report is via istanbul and karma . for this example, let’s assume that we have a simple library sqrt.js which contains an alternative implementation of math.sqrt . note also how it will throw an exception instead of returning nan for an invalid input. var my = { sqrt: function(x) { if (x < 0) throw new error("sqrt can't work on negative number"); return math.exp(math.log(x)/2); } }; using jasmine placed under test/lib/jasmine-1.3.1 , we can craft a test runner that includes the following spec: describe("sqrt", function() { it("should compute the square root of 4 as 2", function() { expect(my.sqrt(4)).toequal(2); }); }); opening the spec runner in a web browser will give the expected outcome: so far so good. now let's see how the code coverage of our test setup can be measured. the first order of business is to install karma . if you are not familiar with karma, it is basically a test runner which can launch and connect to a specific set of web browsers, run your tests, and then gather the report. using node.js, what we need to do is: npm install karma karma-coverage before launching karma, we need to specify its configuration . it could be as simple as the following my.conf.js (most entries are self-explained). note that the tests are executed using phantomjs for simplicity, it is however quite trivial to add other web browsers such as chrome and firefox. module.exports = function(config) { config.set({ basepath: '', frameworks: ['jasmine'], files: [ '*.js', 'test/spec/*.js' ], browsers: ['phantomjs'], singlerun: true, reporters: ['progress', 'coverage'], preprocessors: { '*.js': ['coverage'] } }); }; running the tests, as well as performing code coverage at the same time, can be triggered via: node_modules/.bin/karma start my.conf.js which will dump the output like: info [karma]: karma v0.10.2 server started at http://localhost:9876/ info [launcher]: starting browser phantomjs info [phantomjs 1.9.2 (linux)]: connected on socket n9ndnhj0np92ntspgx-x phantomjs 1.9.2 (linux): executed 1 of 1 success (0.029 secs / 0.003 secs) as expected (from the previous manual invocation of the spec runner), the test passed just fine. however, the most particular interesting piece here is the code coverage report, it is stored (in the default location) under the subdirectory coverage . open the report in your favorite browser and there you'll find the coverage analysis report. behind the scene, karma is using istanbul , a comprehensive javascript code coverage tool (read also my previous blog post on javascript code coverage with istanbul ). istanbul parses the source file, in this example sqrt.js , using esprima and then adds some extra instrumentation which will be used to gather the execution statistics. the above report that you see is one of the possible outputs, istanbul can also generate lcov report which is suitable for many continuous integration systems (jenkins, teamcity, etc). an extensive analysis of the coverage data should also prevent any future coverage regression, check out my other post hard thresholds on javascript code coverage . one important thing about code coverage is branch coverage . if you pay attention carefully, our test above is still not exercising the situation where the input to my.sqrt is negative. there is a big "i" marking in the third-line of the code, this is istanbul telling us that the if branch is not taken at all (for the else branch, it will be an "e" marker). once this missing branch is noticed, improving the situation is as easy as adding one more test to the spec: it("should throw an exception if given a negative number", function() { expect(function(){ my.sqrt(-1); }). tothrow(new error("sqrt can't work on negative number")); }); once the test is executed again, the code coverage report looks way better and everyone is happy. if you have some difficulties following the above step-by-step instructions, take a look at a git repository i have prepared: github.com/ariya/coverage-jasmine-istanbul-karma . feel free to play with it and customize it to suit your workflow!

I’m going to start by discussing the Spring WebMVC configuration and move on from there in future posts.

In this article, I will show how to implement auto-completion using Hibernate Search. The same can be achieved using Solr or ElasticSearch. But I decided to use Hibernate Search as its the simplest to get started with, easily integrates with an existing application and leverages the same core - Lucene. And we get all of this without the overhead of managing Solr/ElasticSearch cluster. In all, I found Hibernate Search to be the go-to search engine for simple use cases. For our use case, we build a product title based auto-completion where often, the user queries are searches for product title. While typing, users should immediately see titles matching their requests, and Hibernate Search should do the hard work to filter the relevant documents in near real-time. Lets have the following JPA annotated Product entity class. public class Product { @Id @Column(name = "sku") private String sku; @Column(name = "upc") private String upc; @Column(name = "title") private String title; .... } We are interested in returning suggestions based on the 'title' field. Title will be indexed based on 2 strategies - N-Gram and Edge N-Gram. Edge N-Gram - This will match only from the left edge of the suggestion text. For this we use KeywordTokenizerFactory (emits the entire input as a single token) and EdgeNGramFilterFactory along with some regex cleansing. N-Gram matches from the start of every word, so that you can get right-truncated suggestions for any word in the text, not only from the first word. The main difference from N-gram is the tokenizer which is StandardTokenizerFactory along with NGramFilterFactory. Using these strategies, if the document field is "A brown fox" and the query is a) "A bro"- Will match b) "bro" - Will match Implementation: In the entity defined above, we can map 'title' property twice with the above strategies. Below are the annotations to instruct Hibernate to index 'title' twice. @Entity @Table(name = "item_master") @Indexed(index = "Products") @AnalyzerDefs({ @AnalyzerDef(name = "autocompleteEdgeAnalyzer", // Split input into tokens according to tokenizer tokenizer = @TokenizerDef(factory = KeywordTokenizerFactory.class), filters = { // Normalize token text to lowercase, as the user is unlikely to // care about casing when searching for matches @TokenFilterDef(factory = PatternReplaceFilterFactory.class, params = { @Parameter(name = "pattern",value = "([^a-zA-Z0-9\\.])"), @Parameter(name = "replacement", value = " "), @Parameter(name = "replace", value = "all") }), @TokenFilterDef(factory = LowerCaseFilterFactory.class), @TokenFilterDef(factory = StopFilterFactory.class), // Index partial words starting at the front, so we can provide // Autocomplete functionality @TokenFilterDef(factory = EdgeNGramFilterFactory.class, params = { @Parameter(name = "minGramSize", value = "3"), @Parameter(name = "maxGramSize", value = "50") }) }), @AnalyzerDef(name = "autocompleteNGramAnalyzer", // Split input into tokens according to tokenizer tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class), filters = { // Normalize token text to lowercase, as the user is unlikely to // care about casing when searching for matches @TokenFilterDef(factory = WordDelimiterFilterFactory.class), @TokenFilterDef(factory = LowerCaseFilterFactory.class), @TokenFilterDef(factory = NGramFilterFactory.class, params = { @Parameter(name = "minGramSize", value = "3"), @Parameter(name = "maxGramSize", value = "5") }), @TokenFilterDef(factory = PatternReplaceFilterFactory.class, params = { @Parameter(name = "pattern",value = "([^a-zA-Z0-9\\.])"), @Parameter(name = "replacement", value = " "), @Parameter(name = "replace", value = "all") }) }), @AnalyzerDef(name = "standardAnalyzer", // Split input into tokens according to tokenizer tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class), filters = { // Normalize token text to lowercase, as the user is unlikely to // care about casing when searching for matches @TokenFilterDef(factory = WordDelimiterFilterFactory.class), @TokenFilterDef(factory = LowerCaseFilterFactory.class), @TokenFilterDef(factory = PatternReplaceFilterFactory.class, params = { @Parameter(name = "pattern", value = "([^a-zA-Z0-9\\.])"), @Parameter(name = "replacement", value = " "), @Parameter(name = "replace", value = "all") }) }) // Def }) public class Product { .... } Explanation: 2 custom analyzers - autocompleteEdgeAnalyzer andautocompleteNGramAnalyzer have been defined as per theory in the previous section. Next, we apply these analyzers on the 'title' field to create 2 different indexes. Here is how we do it: @Column(name = "title") @Fields({ @Field(name = "title", index = Index.YES, store = Store.YES, analyze = Analyze.YES, analyzer = @Analyzer(definition = "standardAnalyzer")), @Field(name = "edgeNGramTitle", index = Index.YES, store = Store.NO, analyze = Analyze.YES, analyzer = @Analyzer(definition = "autocompleteEdgeAnalyzer")), @Field(name = "nGramTitle", index = Index.YES, store = Store.NO, analyze = Analyze.YES, analyzer = @Analyzer(definition = "autocompleteNGramAnalyzer")) }) private String title; Start indexing: public void index() throws InterruptedException { getFullTextSession().createIndexer().startAndWait(); } Once indexed, inspect the index using Luke and you should be able to see title analyzed and stored as N-Grams and Edge N-Grams. Search Query: private static final String TITLE_EDGE_NGRAM_INDEX = "edgeNGramTitle"; private static final String TITLE_NGRAM_INDEX = "nGramTitle"; @Transactional(readOnly = true) public synchronized List getSuggestions(final String searchTerm) { QueryBuilder titleQB = getFullTextSession().getSearchFactory() .buildQueryBuilder().forEntity(Product.class).get(); Query query = titleQB.phrase().withSlop(2).onField(TITLE_NGRAM_INDEX) .andField(TITLE_EDGE_NGRAM_INDEX).boostedTo(5) .sentence(searchTerm.toLowerCase()).createQuery(); FullTextQuery fullTextQuery = getFullTextSession().createFullTextQuery( query, Product.class); fullTextQuery.setMaxResults(20); @SuppressWarnings("unchecked") List results = fullTextQuery.list(); return results; } And we have a working suggester. What next? Expose the functionality via a REST API and integrate it with jQuery, examples of which can be easily found. You can also use the same strategy with Solr and ElasticSearch.

Feeling good to be back at the blog . Actually, I have been managing GDG Ahmedabad, delivering android talks, and managing workshops locally and outside my region. Last month, I was quite busy in organizing the “DevFest” event for GDG Ahmedabad, and then for the preparation of my two talks for the GDG Kathmandu DevFest. I was invited to deliver two talks at DevFest, which was organized by GDG Kathmandu. I have already published slides on my Speakerdeck. I am not sure whether you have already checked and learned from my speaker deck, but still give me a chance to write about Introduction to Android studio here. What is Android Studio? It’s an Android focused IDE, designed specially for Android development. It was launched on 16th May 2013, during Google's I/O 2013 event. Android studio contains all the Android SDK tools to design, test, debug and profile your app. By looking at the development tools and environment, we can see its similar to Eclipse with the ADT plug-in, but as I have mentioned above, it's an Android focused IDE, and there are many cool features available in Android Studio that can foster and increase your development productivity. One great thing is that it depends on the IntelliJ Idea IDE, which has proved itself to be a great IDE and has been in use by many Android engineers. What is the Difference Between IntelliJ Idea and Android Studio? Nothing, in regards to Android. If you use IntelliJ… Keep using it IntelliJ 13 will have the same stuff EAP of IntelliJ Idea 13 includes all the new stuff If Not… Give Android Studio a try You may have some questions in mind regarding IntelliJ and Android Studio. If so, check the FAQ section: IntelliJ IDEA and Android Studio FAQ. Let’s Download Android Studio You can download Android Studio from the android developer site: http://developer.android.com/sdk/installing/studio.html. Cool Features of Android Studio As I have mentioned, it's similar to Eclipse with the ADT plug-in, but Android Studio has many cool features that can help you to increase development productivity. Here are the cool features: Powerful code editing (smart editing, code re-factoring) Rich layout Editor (As soon as you drag and drop views on the layout, it shows you a preview in all the screens including Nexus 4, Nexus 7, Nexus 10 and many other resolutions. Layout designing can be done much faster way as compared to eclipse.) Gradle-based build support Maven Support Template-based wizards Lint tool analysis (The Android lint tool is a static code analysis tool that checks your Android project source files for potential bugs and optimization improvements for correctness, security, performance, usability, accessibility, and internationalization). You can experience all the cool features by using Android Studio yourself Awesome Stuff Inside Darcula Theme It's actually a black-based theme. While using Android Studio, I enjoy working in Darcula theme environment. By the way, Its Darcula theme, not Dracula. I am correcting this just because I have seen many people on Stackoverflow and Google+ saying Dracula. You can set the Darcula theme in Android Studio by: File > Settings > IDE Settings > Appearance > Theme: Darcula. Preview All the Screens We can consider this is as part of the Rich layout editor feature. With this privilege, users can design layouts and can check layouts by previewing in all the possible screens, such as Nexus 4, Nexus 7, Nexus and many other devices. It helps the user to improve layout designs while providing compatibility to various resolutions available. Device Framed Screen Capture It provides ability to directly generate a screenshot of your application. Yes, it was already included in the SDK, but Android Studio provides something more: Device frame (As frames for many Nexus devices are available, you can capture screenshot in whichever frame you like most) Drop shadow Screen glare Color Preview I like this feature very much and I have found this feature helpful while working on big projects. While using Eclipse, we have to have 3rd party color chooser and picker but this feature gives privilege to select color from in-build color chooser and can also have preview in Colors.xml file. Color Preview – Activity class While using Eclipse, it’s difficult to check which color we have used. Yes, we can imagine the color by its name, but an actual preview is much better. This feature was recently introduced in Android Studio, so you must have latest version installed. Hard Coded Strings Here is another feature I like and have found useful: Whenever you use any string resources from Strings.xml, it displays actual value instead of variable name. This setting comes by default, but in case you aren’t able to get hard coded strings in your activity class, then try any of the below ways. Settings > Editor > Code Folding > Android String References OR Select String and right click on it and then go to Folding > Collapse OR CTRL + Numpad ‘-’ Create Layout Variation This provides the ability to create layout variation directly. For example: layout for the large screen, layout for Xlarge screen, etc. The great thing is that the created variant layout gets stored in particular folders like layout-xlarge, layout-large-land, etc. Should I Use Android Studio? You might have explored all the cool features, or you are ready to explore right now. But questions might have arisen in your mind: “Should I use Android Studio,” or “should we start using Android Studio right now,” or “should I continue with IntelliJ or Eclipse?” My answer is a big NO to use Android Studio as your main IDE for Android development, because currently its EARLY ACCESS PREVIEW and it's maturing over days. Engineers have been working hard to improve this IDE. So, you should wait until the BETA comes out. I agree with Carlos Vega (commented over G+) on this point: “You should at least migrate to Intellij Idea 12 so that you get familiar with the IDE’s workflow and keyboard shortcuts. That way when Android Studio reach a more stable level, you can switch without a major learning curve.” Thanks, Carlos Vega, for the input. By the way, here is the presentation I delivered at the GDG Kathmandu DevFest.

The Builder pattern has been described in the Gang of Four “Design Patterns” book: The builder pattern is a design pattern that allows for the step-by-step creation of complex objects using the correct sequence of actions. The construction is controlled by a director object that only needs to know the type of object it is to create. A common implementation of using the Builder pattern is to have a fluent interface, with the following caller code: Person person = new PersonBuilder().withFirstName("John").withLastName("Doe") .withTitle(Title.MR).build(); This code snippet can be enabled by the following builder: public class PersonBuilder { private Person person = new Person(); public PersonBuilder withFirstName(String firstName) { person.setFirstName(firstName); return this; } // Other methods along the same model // ... public Person build() { return person; } } The job of the Builder is achieved: the Person instance is well-encapsulated and only the build() method finally returns the built instance. This is usually where most articles stop, pretending to have covered the subject. Unfortunately, some cases may arise that need deeper work. Let’s say we need some validation handling the final Person instance, e.g. the lastName attribute is to be mandatory. To provide this, we could easily check if the attribute is null in the build() method and throws an exception accordingly. public Person build() { if (lastName == null) { throw new IllegalStateException("Last name cannot be null"); } return person; } Sure, this resolves our problem. Unfortunately, this check happens at runtime, as developers calling our code will find (much to their chagrin). To go the way to true DSL, we have to update our design – a lot. We should enforce the following caller code: Person person1 = new PersonBuilder().withFirstName("John").withLastName("Doe").withTitle(Title.MR).build(); // OK Person person2 = new PersonBuilder().withFirstName("John").withTitle(Title.MR).build(); // Doesn't compile We have to update our builder so that it may either return itself, or an invalid builder that lacks the build() method as in the following diagram. Note the first PersonBuilder class is kept as the entry-point for the calling code doesn’t have to cope with Valid-/InvaliPersonBuilder if it doesn’t want to. This may translate into the following code: public class PersonBuilder { private Person person = new Person(); public InvalidPersonBuilder withFirstName(String firstName) { person.setFirstName(firstName); return new InvalidPersonBuilder(person); } public ValidPersonBuilder withLastName(String lastName) { person.setLastName(lastName); return new ValidPersonBuilder(person); } // Other methods, but NO build() methods } public class InvalidPersonBuilder { private Person person; public InvalidPersonBuilder(Person person) { this.person = person; } public InvalidPersonBuilder withFirstName(String firstName) { person.setFirstName(firstName); return this; } public ValidPersonBuilder withLastName(String lastName) { person.setLastName(lastName); return new ValidPersonBuilder(person); } // Other methods, but NO build() methods } public class ValidPersonBuilder { private Person person; public ValidPersonBuilder(Person person) { this.person = person; } public ValidPersonBuilder withFirstName(String firstName) { person.setFirstName(firstName); return this; } // Other methods // Look, ma! I can build public Person build() { return person; } } This is a huge improvement, as now developers can know at compile-time their built object is invalid. The next step is to imagine more complex use-case: Builder methods have to be called in a certain order. For example, a house should have foundations, a frame and a roof. Building the frame requires having built foundations, as building the roof requires the frame. Even more complex, some steps are dependent on previous steps (e.g. having a flat roof is only possible with a concrete frame) The exercise is left to interested readers. Links to proposed implementations welcome in comments. There’s one flaw with our design: just calling the setLastName() method is enough to qualify our builder as valid, so passing null defeats our design purpose. Checking for null value at runtime wouldn’t be enough for our compile-time strategy. The Scala language features may leverage an enhancement to this design called the type-safe builder pattern. Summary In real-life software, the builder pattern is not so easy to implement as quick examples found here and there Less is more: create an easy-to-use DSL is (very) hard Scala makes it easier for complex builder implementation’s designers than Java