This article wants to bring some hints about how to use HTML5 canvas with Apache Wicket web framework. Inside a Wicket application we want to have a panel with something drawn inside a HTML5 canvas. To make this happen we have to think about following: Do we really need HTML5? If we need HTML5, how to do it? What to do if browser version is an issue and it does not support HTML5? 1. First we should ask if we really need HTML5 If we need just an image then we should consider to draw inside a Java2D Graphics object. If we need some animation we should consider to draw inside a HTML5 canvas, but even in this case we need a simple Java2D image implementation if browser version is a concern and canvas is not supported. Wicket has a RenderedDynamicImageResource class which is very handy for this because we can do Java2D stuff inside render(Graphics2D g2) method. A simple example may look like the following: public class MyDynamicImageResource extends RenderedDynamicImageResource { private int width; private int height; private MyData data; public MyDynamicImageResource (int width, int height, MYData data) { super(width, height); this.width = width; this.height = height; this.data = data; } protected boolean render(Graphics2D g2) { g2.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON); g2.setRenderingHint(RenderingHints.KEY_RENDERING, RenderingHints.VALUE_RENDER_QUALITY); // your code } } Because Java2d is used, we can set anti-aliasing to make the image look good. Then we can use this dynamic resource to create our panel. We will use Wicket's NonCachingImage class, a subclass of Image that adds random noise to the url at every request to prevent the browser from caching the image. If you do not care that browser caches your image then you should use a simple Image instead. public class MyJava2DImagePanel extends Panel { private MyDynamicImageResource imageResource; public MyJava2DImagePanel(String id, final int width, final int height, final IModel model) { super(id, model); NonCachingImage image = new NonCachingImage("myImage", new PropertyModel(this, "imageResource")) { private static final long serialVersionUID = 1L; @Override protected void onBeforeRender() { imageResource = new MyDynamicImageResource(width, height, model.getObject()); super.onBeforeRender(); } }; add(image); } } Markup html file for MyJava2DImagePanel will contain the image: 2. If we need some animation for our image, then we should think to draw it on a HTML5 canvas. We should pay attention to draw things just once, meaning for example if we draw a text twice in same position , then our result will look ugly (pix-elated) because an anti-aliasing for canvas cannot be set as for Java2D Graphics object. First we need to create our java script code. We can obtain a Java 2d context and use it to draw our image. I won't talk about canvas context and its methods here. For animation we use jquery in following snippet, but you can use anything you like. Knowing two values (from, to) we can have for example a drawColor method which can paint different segments, creating this way a filling effect which takes in this example 1000ms : var myWidget = function(id, color) { var can = document.getElementById(id); var ctx = can.getContext('2d'); // clear canvas ctx.clearRect(0, 0, can.width, can.height); // draw your image on ctx ..... // animate color fill $({ n: from }).animate({ n: to}, { duration: 1000, step: function(now, fx) { drawColor(id, now); } }); } } Second we have to create our Wicket panel. Canvas is just a WebMarkupContainer and we set width and height through some AttributeAppenders: public class MyHTML5Panel extends Panel { private final ResourceReference MY_JS = new JavaScriptResourceReference(MyHTML5Panel.class, "my.js"); public MyHTML5Panel(String id, String width, String height, IModel model) { super(id, model); WebMarkupContainer container = new WebMarkupContainer("canvas"); container.setOutputMarkupId(true); container.add(new AttributeAppender("width", width)); container.add(new AttributeAppender("height", height)); add(container); } @Override public void renderHead(IHeaderResponse response) { response.renderOnLoadJavaScript(getJavascriptCall()); //include js file response.renderJavaScriptReference(MY_JS); } private String getJavascriptCall() { MyData data = getModel().getObject(); StringBuilder sb = new StringBuilder(); sb.append("myWidget(\""). append(get("canvas").getMarkupId()). append("\",\"").append(data.getColor()). append("\");"); return sb.toString(); } } renderHead(IHeaderResponse response) method from Panel can use the IHeaderResponse object to render our java script call. Also, on the response object we should render our java script reference file. We can use one of the following methods: /** * Renders javascript that is executed right after the DOM is built, before external resources * (e.g. images) are loaded. * * @param javascript */ public void renderOnDomReadyJavaScript(String javascript); /** * Renders javascript that is executed after the entire page is loaded. * * @param javascript */ public void renderOnLoadJavaScript(String javascript); There are situations when we should call one or another depending on our business. As an example, if we need to expose our wicket component to an external iframe, we must call onLoad instead of onDomReady to make it appear inside iframe because $(document).ready in the iframe seems to be fired too soon and the iframe content isn't even loaded yet. HTML markup file MyHTML5Panel.html will contain the canvas tag: 3. If we choose to use HTML5 panel but we also have to think about older browser that cannot support canvas tag, we will have to create both a Java2D and a HTML5 panel and see what to render by ourselves. A solution is to have a wrapper panel with a container which initially contains an EmptyPanel and we add a Wicket Behavior to the container. That behavior will choose what to render (html5 or simple image): ..... container = new WebMarkupContainer("container"); container.setOutputMarkupId(true); container.add(new EmptyPanel("image")); add(container); add(new MyHTML5Behavior()); ....... The following java-script code is a way to test if canvas tag is supported by browser: function isCanvasEnabled() { return !!document.createElement('canvas').getContext; } This function starts by creating a dummy element which is never attached to the page, so no one will ever see it. As soon as we create the dummy element, we test for the presence of a getContext() method. This method will only exist if browser supports the canvas API. Finally, we use the double-negative trick to force the result to a Boolean value (true or false). To call this java script and make the result available to Wicket we use wicketAjaxGet javascript method as seen in following code. We append a result parameter to callback url and inside respond method we can read the value of this parameter. class MyHTML5Behavior extends AbstractDefaultAjaxBehavior { private String width; private String height; private String PARAM = "Param"; public MyHTML5Behavior() { super(); } @Override public void renderHead(Component component, IHeaderResponse response) { super.renderHead(component, response); //include js file response.renderJavaScriptReference(MY_UTIL_JS); response.renderOnLoadJavaScript(getJavascript()); } @Override protected void respond(AjaxRequestTarget target) { String param = this.getComponent().getRequest().getRequestParameters().getParameterValue(PARAM).toString(); // test if html5 canvas tag is supported if (Boolean.parseBoolean(param)) { container.replace(new MyHTML5Panel("image", width, height, model).setOutputMarkupId(true)); } else { container.replace(new MyImagePanel("image", width, height, model).setOutputMarkupId(true)); } target.add(container); } // this javascript call will make the PARAM available to wicket and can be read in respond method private String getJavascript() { StringBuilder sb = new StringBuilder(); sb.append("var data = isCanvasEnabled();"); sb.append("wicketAjaxGet('" + getCallbackUrl() + "&" + PARAM + "='+ data" + ", null, null, function() { return true; })"); return sb.toString(); } } These are just some hints on how to use HTML5 canvas inside Apache Wicket framework. I hope it will help others.

JavaFX provides certain colors by default when rendering charts. There are situations, however, when one wants to customize these colors. In this blog post I look at changing the colors of a JavaFX pie chart using an example I intend to include in my presentation this afternoon at RMOUG Training Days 2013. Some Java-based charting APIs provided Java methods to set colors. JavaFX, born in the days of HTML5 prevalence, instead uses Cascading Style Sheets (CSS) to allow developers to adjust colors, symbols, placement, alignment and other stylistic issues used in their charts. I demonstrate using CSS to change colors here. In this post, I will look at two code samples demonstrating simple JavaFX applications that render pie charts based on data from Oracle's sample 'hr' schema. The first example does not specify colors and so uses JavaFX's default colors for pie slices and for the legend background. That example is shown next. EmployeesPerDepartmentPieChart (Default JavaFX Styling) package rmoug.td2013.dustin.examples; import javafx.application.Application; import javafx.scene.Scene; import javafx.scene.chart.PieChart; import javafx.scene.layout.StackPane; import javafx.stage.Stage; /** * Simple JavaFX application that generates a JavaFX-based Pie Chart representing * the number of employees per department. * * @author Dustin */ public class EmployeesPerDepartmentPieChart extends Application { final DbAccess databaseAccess = DbAccess.newInstance(); @Override public void start(final Stage stage) throws Exception { final PieChart pieChart = new PieChart( ChartMaker.createPieChartDataForNumberEmployeesPerDepartment( this.databaseAccess.getNumberOfEmployeesPerDepartmentName())); pieChart.setTitle("Number of Employees per Department"); stage.setTitle("Employees Per Department"); final StackPane root = new StackPane(); root.getChildren().add(pieChart); final Scene scene = new Scene(root, 800 ,500); stage.setScene(scene); stage.show(); } public static void main(final String[] arguments) { launch(arguments); } } When the above simple application is executed, the output shown in the next screen snapshot appears. I am now going to adapt the above example to use a custom "theme" of blue-inspired pie slices with a brown background on the legend. Only one line is needed in the Java code to include the CSS file that has the stylistic specifics for the chart. In this case, I added several more lines to catch and print out any exception that might occur while trying to load the CSS file. With this approach, any problems loading the CSS file will lead simply to output to standard error stating the problem and the application will run with its normal default colors. EmployeesPerDepartmentPieChartWithCssStyling (Customized CSS Styling) package rmoug.td2013.dustin.examples; import javafx.application.Application; import javafx.scene.Scene; import javafx.scene.chart.PieChart; import javafx.scene.layout.StackPane; import javafx.stage.Stage; /** * Simple JavaFX application that generates a JavaFX-based Pie Chart representing * the number of employees per department and using style based on that provided * in CSS stylesheet chart.css. * * @author Dustin */ public class EmployeesPerDepartmentPieChartWithCssStyling extends Application { final DbAccess databaseAccess = DbAccess.newInstance(); @Override public void start(final Stage stage) throws Exception { final PieChart pieChart = new PieChart( ChartMaker.createPieChartDataForNumberEmployeesPerDepartment( this.databaseAccess.getNumberOfEmployeesPerDepartmentName())); pieChart.setTitle("Number of Employees per Department"); stage.setTitle("Employees Per Department"); final StackPane root = new StackPane(); root.getChildren().add(pieChart); final Scene scene = new Scene(root, 800 ,500); try { scene.getStylesheets().add("chart.css"); } catch (Exception ex) { System.err.println("Cannot acquire stylesheet: " + ex.toString()); } stage.setScene(scene); stage.show(); } public static void main(final String[] arguments) { launch(arguments); } } The chart.css file is shown next: chart.css /* Find more details on JavaFX supported named colors at http://docs.oracle.com/javafx/2/api/javafx/scene/doc-files/cssref.html#typecolor */ /* Colors of JavaFX pie chart slices. */ .data0.chart-pie { -fx-pie-color: turquoise; } .data1.chart-pie { -fx-pie-color: aquamarine; } .data2.chart-pie { -fx-pie-color: cornflowerblue; } .data3.chart-pie { -fx-pie-color: blue; } .data4.chart-pie { -fx-pie-color: cadetblue; } .data5.chart-pie { -fx-pie-color: navy; } .data6.chart-pie { -fx-pie-color: deepskyblue; } .data7.chart-pie { -fx-pie-color: cyan; } .data8.chart-pie { -fx-pie-color: steelblue; } .data9.chart-pie { -fx-pie-color: teal; } .data10.chart-pie { -fx-pie-color: royalblue; } .data11.chart-pie { -fx-pie-color: dodgerblue; } /* Pie Chart legend background color and stroke. */ .chart-legend { -fx-background-color: sienna; } Running this CSS-styled example leads to output as shown in the next screen snapshot. The slices are different shades of blue and the legend's background is "sienna." Note that while I used JavaFX "named colors," I could have also used "#0000ff" for blue, for example. I did not show the code here for my convenience classes ChartMaker and DbAccess. The latter simply retrieves the data for the charts from the Oracle database schema via JDBC and the former converts that data into the Observable collections appropriate for the PieChart(ObservableList) constructor. It is important to note here that, as Andres Almiray has pointed out, it is not normally appropriate to execute long-running processes from the main JavaFX UI thread (AKA JavaFX Application Thread) as I've done in this and other other blog post examples. I can get away with it in these posts because the examples are simple, the database retrieval is quick, and there is not much more to the chart rendering application than that rendering so it is difficult to observe any "hanging." In a future blog post, I intend to look at the better way of handling the database access (or any long-running action) using the JavaFX javafx.concurrent package (which is well already well described in Concurrency in JavaFX). JavaFX allows developers to control much more than simply chart colors with CSS. Two very useful resources detailing what can be done to style JavaFX charts with CSS are the Using JavaFX Charts section Styling Charts with CSS and the JavaFX CSS Reference Guide. CSS is becoming increasingly popular as an approach to styling web and mobile applications. By supporting CSS styling in JavaFX, the same styles can easily be applied to JavaFX apps as the HTML-based applications they might coexist with.

Futures are very important abstraction, even more these day than ever due to growing demand for asynchronous, event-driven, parallel and scalable systems.

In Management 3.0 classes I let participants play an exercise called Moving Motivators, which uses the CHAMPFROGS model for intrinsic motivation. This model is loosely based on the book The 16 Basic Desires by Steven Reiss. I simplified Reiss’ model by removing some very basic desires, such as family, romance, and vengeance, which I considered somewhat less desirable within the context of a team. (Though this simplification makes the model less applicable to the crew of Battlestar Galactica, and other teams on TV.) By renaming a few of the key terms I came up with this list of 10 intrinsic desires… CHAMPFROGS Curiosity: I have plenty of things to investigate and to think about. Honor: I feel proud that my personal values are reflected in how I work. Acceptance: The people around me approve of what I do and who I am. Mastery: My work challenges my competence but it is still within my abilities. Power: There’s enough room for me to influence what happens around me. Freedom: I am independent of others with my work and my responsibilities. Relatedness: I have good social contacts with the people in my work. Order: There are enough rules and policies for a stable environment. Goal: My purpose in life is reflected in the work that I do. Status: My position is good, and recognized by the people who work with me. I told many people that I have no idea what champfrogs means. But it’s a nice mnemonic that enables me to remember the 10 intrinsic motivators for team members. Sometimes people point out to me that there are other models for intrinsic motivation available: Maslow’s Hierarchy of Needs Psychologist Abraham Maslow came up with his famous theory in 1943: Self-actualization: similar to Curiosity, Mastery, Freedom Esteem: similar to Honor, Power, Goal, Status Love/belonging: similar to Relatedness, Acceptance Safety: similar to Order Physiological: similar to a few that I deleted. I simply made a best guess of the correlation with Maslow’s model and Champfrogs, so please don’t interpret my mapping as a law. Also note that scientists have dismissed the hierarchical nature of Maslow’s model as unscientific. Personally, I find the 10 motivators easier to discuss than Maslow’s hierarchy, which is why I prefer the Champfrogs model. SCARF by David Rock Dr. David Rock, the founder of the NeuroLeadership Institute, came up with this model: Status: same as in Champfrogs Certainty: equivalent to Order Autonomy: equivalent to Freedom Relatedness: same as in Champfrogs Fairness: similar to Honor (not sure about this one) It seems to me that SCARF is simply half of CHAMPFROGS. The motivators that are missing are Curiosity, Acceptance, Mastery, Power, and Goal. Personally I find those too important to ignore, which is why I prefer Champfrogs over Scarf when discussing motivation in a team. Self-Determination Theory by Deci & Ryan Professor in psychology Edward L. Deci, together with Richard M. Ryan, proposed the following model: Competence: equivalent to Mastery Relatedness: same as in Champfrogs Autonomy: equivalent to Freedom This model lists even fewer intrinsic motivators for people. I’m sure it is a fine model, but I find it too limited for practical exploration in teams. Note that Daniel Pink, in his book Drive, popularized Self-Determination Theory and actually changed it to Autonomy, Mastery and Purpose. I am not the only one to point out that Daniel Pink replaced Relatedness with Purpose, but they’re all in the Champfrogs model anyway, so who cares? Moving Motivators Regardless of all the small differences I mentioned above, there's one that is obviously the most important... CHAMPFROGS has pictures on cards! :-) If you’re interested in playing with the CHAMPFROGS model, you may want to download the free PDF or order the “official” cards. Your team would not be the first to have a bit of fun with them.

I find myself working more and more with Spring these days, and what I find raises questions. This week, my thoughts are turned toward beans overwriting, that is registering more than one bean with the same name. In the case of a simple project, there’s no need for this; but when building a a plugin architecture around a core, it may be a solution. Here are some facts I uncovered and verified regarding beans overwriting. Single bean id per file The id attribute in the Spring bean file is of type ID, meaning you can have only a single bean with a specific ID in a specific Spring beans definition file. Overwriting bean dependent on context fragments loading order As opposed to classpath loading where the first class takes priority over those others further on the classpath, it’s the last bean of the same name that is finally used. That’s why I called it overwriting. Reversing the fragment loading order proves that. Fragment assembling methods define an order Fragments can be assembled from statements in the Spring beans definition file or through an external component (e.g. the Spring context listener in a web app or test classes). All define a deterministic order. As a side note, though I formerly used import statements in my projects (in part to take advantage of IDE support), experience taught me it can bite you in the back when reusing modules: I’m in favor of assembling through external components now. Names Spring lets you define names in addition to ids (which is a cheap way of putting illegals characters fors ID). Those names also overwrites ids. Aliases Spring lets you define aliases of existing beans: those aliases also overwrites ids. Scope overwriting This one is really mean: by overwriting a bean, you also overwrite scope. So, if the original bean had a specified scope and you do not specify the same, tough luck: you just probably changed the application behavior. Not only are perhaps not known by your development team, but the last one is the killer reason not to overwrite beans. It’s too easy to forget scoping the overwritten bean. In order to address plugins architecture, and given you do not want to walk the OSGi path, I would suggest what I consider a KISS (yet elegant) solution. Let us use simple Java properties in conjunction with ProperyPlaceholderConfigurer. The main Spring Beans definition file should define placeholders for beans that can be overwritten and read two defined properties file: one wrapped inside the core JAR and the other on a predefined path (eventually set by a JVM property). Both property files have the same structure: fully-qualified interface names as keys and fully-qualified implementations names as values. This way, you define default implementations in the internal property file and let uses overwrite them in the external file (if necessary). As an added advantage, it shields users from Spring so they are not tied to the framework. Sources for this article can be found in Maven/Eclipse format here.

Yes, it is long time since i posted… Was just trying to see how a XML can be converted to JSON and to HashMap. The situation is very imaginary. import java.io.File; import java.io.FileInputStream; import java.io.IOException; import java.io.InputStream; import java.util.ArrayList; import java.util.List; import java.util.Map; import net.sf.json.JSON; import net.sf.json.xml.XMLSerializer; import org.apache.commons.io.IOUtils; import org.codehaus.jackson.JsonGenerationException; import org.codehaus.jackson.map.JsonMappingException; import org.codehaus.jackson.map.ObjectMapper; import org.codehaus.jackson.type.TypeReference; public class XML2JSONConvertor { public static void main(String[] args) throws Exception { InputStream is = new FileInputStream(new File( “e:\\jagannathan\\personal\\java-projects\\secondtest.xml”)); String xml = IOUtils.toString(is); XMLSerializer xmlSerializer = new XMLSerializer(); JSON json = xmlSerializer.read(xml); System.out.println(json.toString(2)); printJSON(json.toString(2)); } public static void printJSON(String jsonString) { ObjectMapper mapper = new ObjectMapper(); try { Map jsonInMap = mapper.readValue(jsonString, new TypeReference>() { }); List keys = new ArrayList(jsonInMap.keySet()); for (String key : keys) { System.out.println(key + “: ” + jsonInMap.get(key)); } } catch (JsonGenerationException e) { e.printStackTrace(); } catch (JsonMappingException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } } } Dependencies net.sf.json-lib json-lib 2.4 jdk15 commons-io commons-io 2.3 compile xom xom 1.2.5 org.codehaus.jackson jackson-mapper-asl 1.9.0 The Input XML Jags Inc Jagan Male 24-jul Satya Male 24-apr The output 7 Feb, 2013 7:20:50 PM net.sf.json.xml.XMLSerializer getType INFO: Using default type string { “name”: “Jags Inc”, “employees”: [ { "name": "Jagan", "sex": "Male", "dob": "24-jul" }, { "name": "Satya", "sex": "Male", "dob": "24-apr" } ] } name: Jags Inc employees: [{name=Jagan, sex=Male, dob=24-jul}, {name=Satya, sex=Male, dob=24-apr}]

once upon a time there was a munich pizza baker who developed a technique to beam pizza out of bright sunshine. he can produce more than a thousand pizzas per second and needs a channel to sell this amount of pizza and decides to build an online shop. mario’s initial idea is to sell pizzas, but now he is thinking about introduction of new product lines like beverages, salads and pasta. before we take a look to the validation of mario´s idea, lets take a short look at the existing online shop. mario’s online shop is based on mongodb , apache wicket and spring . mongodb is a document-oriented nosql-database . mongodb stores records not in tables as a relational database but in bson documents, which is a binary version of json (java script object notation) and very similar to the object structure in mario’s application. the usage of mongodb makes his development easier and deployment faster. the figure shows a json document which is very similar to a java object: a json document property with the according value corresponds to the java object property with the appropriate value. you can add or remove properties in your java object and this will automatically change your database schema. so there is no need to put your java object model into a relational schema via hibernate. mario also decided to build his online shop only with open-source technologies like apache wicket and spring. wicket is a very common lightweight component-based web application framework and it is closely patterned after stateful gui frameworks such as javafx . the spring framework is an open source application framework and inversion of control container for the java platform and does not impose any specific programming model. spring has become popular in the java community as an alternative to, replacement for, or even addition to the enterprise javabean (ejb) model. because of this architecture mario is able to deploy its application in a lightweight application server like tomcat or jetty . this figure shows the system landscape of mario. mario has two major system on the lefthand site there is his online shop and on the righthand site there is ‘pas’ a famous billing system. in the middle is hadoop that connects both systems together. in the business world an application normally does not stand alone. in most cases an application must communicate with others. the lean architecture of marios online shop enables him to connect the billing system ‘pas’ to his online shop. spring for apache hadoop provides this integration between the two systems online shop and ‘pas’. hadoop supports data-intensive distributed applications and implements a computational paradigm named mapreduce, where the computation is divided into many small fragments, each of them may be executed or re-executed on any node in the cluster of commodity hardware. mario uses hadoop as an etl layer that enables him to transfer gigabytes of order information into the billing system. in this case hadoop makes it possible for a financial controller to verify if all orders were billed correctly. in addition to the online shop feature mario has a real-time sales dashboard that enables him to track his sales in real time. the dashboard displays daily and monthly sales statistics for each pizza and contains a map with the geographical overview of customer activity and competitor locations. here is a walkthrough of the shop : now lets talk about mario’s incredible new idea : mario wants to sell even more pizza! and other products as well. mario decides to use lean startup methods in order to test the possible introduction of new product lines and plans an experiment to validate his new idea using a scientific approach and pure facts instead of hunches. mario´s core assumption is that customers wants to buy other products than pizza – drinks, salads and pasta. furthermore he is worried about pricing. mario contacts all customers to complete a survey and provides an incentive for the participation, a free pizza to every customer who responds to the survey. the result of the survey validated mario’s assumption – customers want to buy beverages, salads and pasta. but he also found out that his customers are willing to pay higher prices for high-quality products and that they simply love his easy shopping flow. currently a pizza order can be completed with three clicks only, so there is new riskiest assumption to validate: will a more complex shopping flow affect his sales? the figures shows a validation board. a validation board is a deceptively simple tool for testing out product ideas. furthermore a validation board tracks pivots which follows from customer feedback. mario decides to introduce beverages, salads and pasta product lines and thinks about a possibility, how he can handle the extension of the product line without destroying the easy shopping flow. that’s why mario thinks a recommendation engine is the right way for him. panels for recommendations can be integrated in the online shop without changing the shopping flow. mario hired a statistician to help him implement a recommender system for his online shop for better cross-selling. he also defined new measurement points to validate his new idea . therefore he tracks the conversion rate of orders as well as cross-selling rates and every event in the online shop is already tracked in realtime. so mario can very easily perform further experiments in order to verify more assumptions. follow the blog to see how the story continues or come to mongodb usergroup meetup in munich , february 20, 2013 or mongodb days in berlin , february 26, 2013 to get a live presentation. our talk sheds light on how to build an online recommendation engine based on mongodb and apache mahout. we’ll show which recommenders must be built to reach mario’s goal and how these can be integrated in mario’s shop infrastructure.

today, i thought a lot about how to examine different databases. choosing a database is often a daunting task. there's a lot of confusion, a 'theorem', and more than all, the immortal proverb 'not one size fits all'. as if it helps. one of the first things that you realize, when examining nosql distributed databases (and how could you not)is that these days databases are like cars: they're all good. old fashioned sql databases can scale in and out, horizontally sharded over several machines to achieve high availability. nosql systems claim to be consistent. what difference then does it make what database would you choose? the availability and consistency that i mentioned comes, of course, from the misunderstood cap theorem , that - so people say - states that you can only choose 2 out of the 3 consistency: every read would get you the most recent write availability: every node (if not failed) always executes queries partition-tolerance: even if the connections between nodes are down, the other two (a & c) promises, are kept. usually its depicted in a nicely equilaterl triangle, as this one from ofirm : there's a nice proof and explanation of it in this 4 minute video here . but if we think about it, and also see some of brewer's (the theorem author) later remarks , we'll see that the 2 out of 3 is really 1 out of 2: it's really just a vs c! and this is simply because: availability is achieved by replicating the data across different machines consistency is achieved by updating several nodes before allowing further reads total partitioning, meaning failure of part of the system is rare. however, we could look at a delay, a latency, of the update between nodes, as a temporary partitioning . it will then cause a temporary decision between a and c: on systems that allow reads before updating all the nodes, we will get high availability on systems that lock all the nodes before allowing reads, we will get consistency that's it! and since this decision is temporary, it exists only for the duration of the delay, some may say that we are really contrasting latency (another word for availability) against consistency. by the way, there's no distributed system that wants to live with "paritioning" - if it does, it's not distributed. that is why putting sql in this triangle may lead to confusion.

Even from highly experienced technologists I often hear talk about how certain operations cause a CPU cache to "flush". This seems to be illustrating a very common fallacy about how CPU caches work, and how the cache sub-system interacts with the execution cores. In this article I will attempt to explain the function CPU caches fulfil, and how the cores, which execute our programs of instructions, interact with them. For a concrete example I will dive into one of the latest Intel x86 server CPUs. Other CPUs use similar techniques to achieve the same ends. Most modern systems that execute our programs are shared-memory multi-processor systems in design. A shared-memory system has a single memory resource that is accessed by 2 or more independent CPU cores. Latency to main memory is highly variable from 10s to 100s of nanoseconds. Within 100ns it is possible for a 3.0GHz CPU to process up to 1200 instructions. Each Sandy Bridge core is capable of retiring up to 4 instructions-per-cycle (IPC) in parallel. CPUs employ cache sub-systems to hide this latency and allow them to exercise their huge capacity to process instructions. Some of these caches are small, very fast, and local to each core; others are slower, larger, and shared across cores. Together with registers and main-memory, these caches make up our non-persistent memory hierarchy. Next time you are developing an important algorithm, try pondering that a cache-miss is a lost opportunity to have executed ~500 CPU instructions! This is for a single-socket system, on a multi-socket system you can effectively double the lost opportunity as memory requests cross socket interconnects. Memory Hierarchy Figure 1. For the circa 2012 Sandy Bridge E class servers our memory hierarchy can be decomposed as follows: Registers: Within each core are separate register files containing 160 entries for integers and 144 floating point numbers. These registers are accessible within a single cycle and constitute the fastest memory available to our execution cores. Compilers will allocate our local variables and function arguments to these registers. When hyperthreading is enabled these registers are shared between the co-located hyperthreads. Memory Ordering Buffers (MOB): The MOB is comprised of a 64-entry load and 36-entry store buffer. These buffers are used to track in-flight operations while waiting on the cache sub-system. The store buffer is a fully associative queue that can be searched for existing store operations, which have been queued when waiting on the L1 cache. These buffers enable our fast processors to run asynchronously while data is transferred to and from the cache sub-system. When the processor issues asynchronous reads and writes then the results can come back out-of-order. The MOB is used to disambiguate the ordering for compliance to the published memory model. Level 1 Cache: The L1 is a core-local cache split into separate 32K data and 32K instruction caches. Access time is 3 cycles and can be hidden as instructions are pipelined by the core for data already in the L1 cache. Level 2 Cache: The L2 cache is a core-local cache designed to buffer access between the L1 and the shared L3 cache. The L2 cache is 256K in size and acts as an effective queue of memory accesses between the L1 and L3. L2 contains both data and instructions. L2 access latency is 12 cycles. Level 3 Cache: The L3 cache is shared across all cores within a socket. The L3 is split into 2MB segments each connected to a ring-bus network on the socket. Each core is also connected to this ring-bus. Addresses are hashed to segments for greater throughput. Latency can be up to 38 cycles depending on cache size. Cache size can be up to 20MB depending on the number of segments, with each additional hop around the ring taking an additional cycle. The L3 cache is inclusive of all data in the L1 and L2 for each core on the same socket. This inclusiveness, at the cost of space, allows the L3 cache to intercept requests thus removing the burden from private core-local L1 & L2 caches. Main Memory: DRAM channels are connected to each socket with an average latency of ~65ns for socket local access on a full cache-miss. This is however extremely variable, being much less for subsequent accesses to columns in the same row buffer, through to significantly more when queuing effects and memory refresh cycles conflict. 4 memory channels are aggregated together on each socket for throughput, and to hide latency via pipelining on the independent memory channels. NUMA: In a multi-socket server we have non-uniform memory access. It is non-uniform because the required memory maybe on a remote socket having an additional 40ns hop across the QPI bus. Sandy Bridge is a major step forward for 2-socket systems over Westmere and Nehalem. With Sandy Bridge the QPI limit has been raised from 6.4GT/s to 8.0GT/s, and two lanes can be aggregated thus eliminating the bottleneck of the previous systems. For Nehalem and Westmere the QPI link is only capable of ~40% the bandwidth that could be delivered by the memory controller for an individual socket. This limitation made accessing remote memory a choke point. In addition, the QPI link can now forward pre-fetch requests which previous generations could not. Associativity Levels Caches are effectively hardware based hash tables. The hash function is usually a simple masking of some low-order bits for cache indexing. Hash tables need some means to handle a collision for the same slot. The associativity level is the number of slots, also known as ways or sets, which can be used to hold a hashed version of an address. Having more levels of associativity is a trade off between storing more data vs. power requirements and time to search each of the ways. For Sandy Bridge the L1 and L2 are 8-way and the L3 is 12-way associative. Cache Coherence With some caches being local to cores, we need a means of keeping them coherent so all cores can have a consistent view of memory. The cache sub-system is considered the "source of truth" for mainstream systems. If memory is fetched from the cache it is never stale; the cache is the master copy when data exists in both the cache and main-memory. This style of memory management is known as write-back whereby data in the cache is only written back to main-memory when the cache-line is evicted because a new line is taking its place. An x86 cache works on blocks of data that are 64-bytes in size, known as a cache-line. Other processors can use a different size for the cache-line. A larger cache-line size reduces effective latency at the expense of increased bandwidth requirements. To keep the caches coherent the cache controller tracks the state of each cache-line as being in one of a finite number of states. The protocol Intel employs for this is MESIF, AMD employs a variant know as MOESI. Under the MESIF protocol each cache-line can be in 1 of the 5 following states: Modified: Indicates the cache-line is dirty and must be written back to memory at a later stage. When written back to main-memory the state transitions to Exclusive. Exclusive: Indicates the cache-line is held exclusively and that it matches main-memory. When written to, the state then transitions to Modified. To achieve this state a Request-For-Ownership (RFO) message is sent which involves a read plus an invalidate broadcast to all other copies. Shared: Indicates a clean copy of a cache-line that matches main-memory. Invalid: Indicates an unused cache-line. Forward: Indicates a specialised version of the shared state i.e. this is the designated cache which should respond to other caches in a NUMA system. To transition from one state to another, a series of messages are sent between the caches to effect state changes. Previous to Nehalem for Intel, and Opteron for AMD, this cache coherence traffic between sockets had to share the memory bus which greatly limited scalability. These days the memory controller traffic is on a separate bus. The Intel QPI, and AMD HyperTransport, buses are used for cache coherence between sockets. The cache controller exists as a module within each L3 cache segment that is connected to the on-socket ring-bus network. Each core, L3 cache segment, QPI controller, memory controller, and integrated graphics sub-system are connected to this ring-bus. The ring is made up of 4 independent lanes for: request, snoop, acknowledge, and 32-bytes data per cycle. The L3 cache is inclusive in that any cache-line held in the L1 or L2 caches is also held in the L3. This provides for rapid identification of the core containing a modified line when snooping for changes. The cache controller for the L3 segment keeps track of which core could have a modified version of a cache-line it owns. If a core wants to read some memory, and it does not have it in a Shared, Exclusive, or Modified state; then it must make a read on the ring bus. It will then either be read from main-memory if not in the cache sub-systems, or read from L3 if clean, or snooped from another core if Modified. In any case the read will never return a stale copy from the cache sub-system, it is guaranteed to be coherent. Concurrent Programming If our caches are always coherent then why do we worry about visibility when writing concurrent programs? This is because within our cores, in their quest for ever greater performance, data modifications can appear out-of-order to other threads. There are 2 major reasons for this. Firstly, our compilers can generate programs that store variables in registers for relatively long periods of time for performance reasons, e.g. variables used repeatedly within a loop. If we need these variables to be visible across cores then the updates must not be register allocated. This is achieved in C by qualifying a variable as "volatile". Beware that C/C++ volatile is inadequate for telling the compiler to order other instructions. For this you need fences/barriers. The second major issue with ordering we have to be aware of is a thread could write a variable and then, if it reads it shortly after, could see the value in its store buffer which may be older than the latest value in the cache sub-system. This is never an issue for algorithms following the Single Writer Principle but is an issue for the likes of the Dekker and Peterson lock algorithms. To overcome this issue, and ensure the latest value is observed, the thread must wait for the store buffer to drain on that core. This can be achieved by issuing a fence instruction. The write of a volatile variable in Java, in addition to never being register allocated, is accompanied by a full fence instruction. This fence instruction on x86 has a significant performance impact by preventing progress on the issuing thread until the store buffer is drained. Fences on other processors can have more efficient implementations that simply put a marker in the store buffer for the search boundary, e.g. the Azul Vega does this. If you want to ensure memory ordering across Java threads when following the Single Writer Principle, and avoid the store fence, it is possible by using the j.u.c.Atomic(Int|Long|Reference).lazySet() method, as opposed to setting a volatile variable. The Fallacy Returning to the fallacy of "flushing the cache" as part of a concurrent algorithm. I think we can safely say that we never "flush" the CPU cache within our user space programs. I believe the source of this fallacy is the need to flush, mark or drain to a point, the store buffer for some classes of concurrent algorithms so the latest value can be observed on a subsequent load operation. For this we require a memory ordering fence and not a cache flush. Another possible source of this fallacy is that L1 caches, or the TLB, may need to be flushed based on address indexing policy on a context switch. ARM, previous to ARMv6, did not use address space tags on TLB entries thus requiring the whole L1 cache to be flushed on a context switch. Many processors require the L1 instruction cache to be flushed for similar reasons, in many cases this is simply because instruction caches are not required to be kept coherent. The bottom line is, context switching is expensive and a bit off topic, so in addition to the cache pollution of the L2, a context switch can also cause the TLB and/or L1 caches to require a flush. Intel x86 processors require only a TLB flush on context switch.

Want to check out what life is like as a developer? Check out this post for a series of GIFs demonstrating the dev life.

Hi, I have recently created www.php.is-best.net out of an actual business necessity: The need to quickly group lines of code into php functions. What www.php.is-best.net allows you to do is to quickly extract methods out of your code. Just Grab and paste any php script inside the designated area and WHALLA! you're done. A new method will be generated out of your code.

imagine you have a csv file that you want to manipulate. here’s a sample file we can play with: lopez,charlie,2002,11,21 parker,ward,1995,04,08 henderson,russell,2007,10,01 our goal is to transform this into the following form by combining the last three columns: lopez,charlie,20021121 parker,ward,19950408 henderson,russell,20071001 in linux this would take all of two seconds (excuse the awkward awk command): shell$ awk -f"," '{ print $1","$2","$3$4$5 }' people.txt what if you wanted to quickly do the same in hdfs - and let’s assume you want to write the results back to hdfs. one approach would be to use the hdfs cli to stream the inputs into awk, and stream the awk output back into hdfs. you could do this with the hdfs cat and put - options (note that adding a hyphen after put instructs the put command to stream data from standard input to hdfs): shell$ hadoop fs -cat people.txt | awk -f"," '{ print $1","$2","$3$4$5 }' | hadoop fs -put - people-coalesed.txt btw, if your input and output files are lzop-compressed then this command would work: shell$ hadoop fs -cat people.txt.lzo | lzop -dc | awk -f"," '{ print $1","$2","$3$4$5 }' | \ lzop -c | hadoop fs -put - people-coalesed.txt.lzo this is great if your file isn’t too large, but if it’s multiple gigabytes in length then you probably want to harness the power of mapreduce to get this done in a jiffy! the words “in a jiffy” and “mapreduce” aren’t commonly used together, so what do we do? well you could crack open pig or hive and write some custom user-defined functions, but this means you end up in java which we want to avoid. hadoop streaming comes to the rescue in these situations. let’s first create our awk script which will be executed: shell$ cat people.awk #!/bin/awk -f begin { fs = "," } { print $1","$2","$3$4$5 } in linux, if you make this awk script executable, you could execute is as follows: shell$ ./people.awk people.txt in mapreduce-land we don’t need to join data in this particular example, so we don’t need to run any reducers. call your awk script from mappers via hadoop streaming with this command: shell$ hadoop_home=/usr/lib/hadoop shell$ ${hadoop_home}/bin/hadoop \ jar ${hadoop_home}/contrib/streaming/*.jar \ -d mapreduce.job.reduces=0 \ -d mapred.reduce.tasks=0 \ -input people.txt \ -output people-coalesed \ -mapper people.awk \ -file people.awk a few options in the hadoop streaming command are worth examining: finally - to get lzo into the picture you need to add -inputformat , -d mapred.output.compress and -d mapred.output.compression.codec arguments: shell$ hadoop_home=/usr/lib/hadoop shell$ ${hadoop_home}/bin/hadoop \ jar ${hadoop_home}/contrib/streaming/*.jar \ -d mapreduce.job.reduces=0 \ -d mapred.reduce.tasks=0 \ -d mapred.output.compress=true \ -d stream.map.input.ignorekey=true \ -d mapred.output.compression.codec=com.hadoop.compression.lzo.lzopcodec \ -inputformat com.hadoop.mapred.deprecatedlzotextinputformat \ -input people.txt.lzo \ -output people-coalesed \ -mapper people.awk \ -file people.awk

ActiveMq is a powerful open source messaging broker, and is very easy and straightforward to use with Spring as the below classes and XML will prove. The example below is the bar minimum needed to get up and running with transactions and message converters. On the sending side, the ActiveMq connection factory needs to be created with the url of the broker. This in turn is used to create the Spring JMS connection factory and as no session cache property is supplied the default cache is one. The template is then used in turn to create the Message Sender class: An example sending class is below. It uses the convertandSend method of the JmsTemplate class. As there is no destination arg, the message will be sent to the default destination which was set up in the XML file: import java.util.Map; import org.springframework.jms.core.JmsTemplate; public class MessageSender { private final JmsTemplate jmsTemplate; public MessageSender(final JmsTemplate jmsTemplate) { this.jmsTemplate = jmsTemplate; } public void send(final Map map) { jmsTemplate.convertAndSend(map); } } On the receiving side, there needs to be a listener container. The simplest example of this is the SimpleMessageListenerContainer. This requires a connection factory, a destination (or destination name) and a message listener. An example of the Spring configuration for the receiving messages is below: The listening/receiving class needs to extend javax.jms.MessageListener and implement the onMessage method: import javax.jms.MapMessage; import javax.jms.Message; import javax.jms.MessageListener; public class MessageReceiver implements MessageListener { public void onMessage(final Message message) { if (message instanceof MapMessage) { final MapMessage mapMessage = (MapMessage) message; // do something } } } To then send a message would be as simple as getting the sending bean from the bean factory as shown in the below code: MessageSender sender = (MessageSender) factory.getBean("messageSender"); Map map = new HashMap(); map.put("Name", "MYNAME"); sender.send(map); Will try to expand and build up JMS and Spring articles with examples of using transactions and other brokers like MQSeries.

Resin has supported caching, session replication (another form of caching), and http proxy caching in cluster environments for over ten years. When you use Resin caching, you are using the same platform that has the speed and scalability of custom services written in C like NginX with the usability of Java, and the industry platform Java EE. JCache JSR 107 is a distributed cache that has a similar interface to the HashMap that you know and love. To be more specific, the Cache object in JCache looks like a java.util.ConncurrentHashMap. In addition, JCache JSR 107 defines integration with CDI (as well as Spring and Guice). You can decorate services with interceptors that apply caching to the services just by defining annotations. Resin 4 has support for JCache, and JCache support is required for Java EE 7. Let's look at a small example to see how easy is to get started with JCache. package hello.world; import javax.cache.Cache; import javax.cache.CacheBuilder; import javax.cache.CacheManager; import javax.cache.Caching; ... @WebServlet("/HelloServlet") public class HelloServlet extends HttpServlet { Cache cache; public Cache cache() { if (cache == null) { //building a cache CacheManager manager = Caching.getCacheManager("cacheManagerHello"); CacheBuilder builder = manager.createCacheBuilder("a"); cache = builder.build(); } return cache; } protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setContentType("text/html"); response.getWriter().append(" "); String helloMessage = cache().get("hello message"); if (helloMessage == null) { helloMessage = new StringBuilder(20) .append("Hello World ! ") .append(System.currentTimeMillis()).toString(); cache().put("hello message", helloMessage); // <-------------- putting results in the cache } response.getWriter().append(helloMessage); response.getWriter().append(" "); } } The above works out fairly well, but what if we want to periodically change the helloMessage. Let's say we get 2,000 requests a second, but every 10 seconds or so we would like to regenerate the helloMessage. The message might be: Hello World ! 1358979745996 Later we would want it to change. If we wanted it to change every 10 seconds after it was last accessed, we would do this: cache = builder.setExpiry(ExpiryType.ACCESSED, new Duration(TimeUnit.SECONDS, 10)).build(); For this example, we want to change it every 10 seconds after is was last modified. We would set up the timeout on the creation as follows: cache = builder.setExpiry(ExpiryType.MODIFIED, new Duration(TimeUnit.SECONDS, 10)).build(); This would go right in the cache method we defined earlier. public Cache cache() { if (cache == null) { CacheManager manager = Caching.getCacheManager("cacheManagerHello"); CacheBuilder builder = manager.createCacheBuilder("b"); cache = builder.setExpiry(ExpiryType.MODIFIED, new Duration(TimeUnit.SECONDS, 10)).build(); } return cache; } Resin's JCache implementation is built on top Resin distributed cache architecture. You get replication, and data redundancy built in. Bill Digman is a Java EE / Servlet enthusiast and Open Source enthusiast who loves working with Caucho's Resin Servlet Container, a Java EE Web Profile Servlet Container. Caucho's Resin OpenSource Servlet Container Java EE Web Profile Servlet Container Caucho's Resin 4.0 JCache blog post

A question that constantly comes up from people that care about writing good code, is: what’s the right size for a method or function, or a class, or a package or any other chunk of code? At some point any piece of code can be too big to understand properly – but how big is too big? It starts at the method or function level. In Code Complete, Steve McConnell says that the theoretical best maximum limit for a method or function is the number of lines that can fit on one screen (i.e., that a developer can see at one time). He then goes on to reference studies from the 1980s and 1990s which found that the sweet spot for functions is somewhere between 65 lines and 200 lines: routines this size are cheaper to develop and have fewer errors per line of code. However, at some point beyond 200 lines you cross into a danger zone where code quality and understandability will fall apart: code that can’t be tested and can’t be changed safely. Eventually you end up with what Michael Feathers calls “runaway methods”: routines that are several hundreds or thousands of lines long and that are constantly being changed and that continuously get bigger and scarier. Patrick Duboy looks deeper into this analysis on method length, and points to a more modern study from 2002 that shows that code with shorter routines has fewer defects overall, which matches with most people’s intuition and experience. Smaller must be better Bob Martin takes the idea that “if small is good, then smaller must be better” to an extreme in Clean Code: The first rule of functions is that they should be small. The second rule of functions is that they should be smaller than that. Functions should not be 100 lines long. Functions should hardly ever be 20 lines long. Martin admits that “This is not an assertion that I can justify. I can’t produce any references to research that shows that very small functions are better.” So like many other rules or best practices in the software development community, this is a qualitative judgement made by someone based on their personal experience writing code – more of an aesthetic argument – or even an ethical one – than an empirical one. Style over substance. The same “small is better” guidance applies to classes, packages and subsystems – all of the building blocks of a system. In Code Complete, a study from 1996 found that classes with more routines had more defects. Like functions, according to Clean Code, classes should also be “smaller than small”. Some people recommend that 200 lines is a good limit for a class – not a method, or as few as 50-60 lines (in Ben Nadel’s Object Calisthenics exercise)and that a class should consist of “less than 10” or “not more than 20” methods. The famous C3 project – where Extreme Programming was born – had 12 methods per class on average. And there should be no more than 10 classes per package. PMD, a static analysis tool that helps to highlight problems in code structure and style, defines some default values for code size limits: 100 lines per method, 1000 lines per class, and 10 methods in a class. Checkstyle, a similar tool, suggests different limits: 50 lines in a method, 1500 lines in a class. Rule of 30 Looking for guidelines like this led me to the “Rule of 30” in Refactoring in Large Software Projects by Martin Lippert and Stephen Roock: If an element consists of more than 30 subelements, it is highly probable that there is a serious problem: a) Methods should not have more than an average of 30 code lines (not counting line spaces and comments). b) A class should contain an average of less than 30 methods, resulting in up to 900 lines of code. c) A package shouldn’t contain more than 30 classes, thus comprising up to 27,000 code lines. d) Subsystems with more than 30 packages should be avoided. Such a subsystem would count up to 900 classes with up to 810,000 lines of code. e) A system with 30 subsystems would thus possess 27,000 classes and 24.3 million code lines. What does this look like? Take a biggish system of 1 million NCLOC. This should break down into: 30,000+ methods 1,000+ classes 30+ packages Hopefully more than 1 subsystem How many systems in the real world look like this, or close to this – especially big systems that have been around for a few years? Are these rules useful? How should you use them? Using code size as the basis for rules like this is simple: easy to see and understand. Too simple, many people would argue: a better indicator of when code is too big is cyclomatic complexity or some other measure of code quality. But some recent studies show that code size actually is a strong predictor of complexity and quality – that “complexity metrics are highly correlated with lines of code, and therefore the more complex metrics provide no further information that could not be measured simplify with lines of code”. In "Beyond Lines of Code: Do we Need more Complexity Metrics" in Making Software, the authors go so far as to say that lines of code should be considered always as the "first and only metric" for defect prediction, development and maintenance models. Recognizing that simple sizing rules are arbitrary, should you use them, and if so how? I like the idea of rough and easy-to-understand rules of thumb that you can keep in the back of your mind when writing code or looking at code and deciding whether it should be refactored. The real value of a guideline like the Rule of 30 is when you're reviewing code and identifying risks and costs. But enforcing these rules in a heavy handed way on every piece of code as it is being written is foolish. You don’t want to stop when you’re about to write the 31st line in a method – it would slow down work to a crawl. And forcing everyone to break code up to fit arbitrary size limits will make the code worse, not better – the structure will be dominated by short-term decisions. As Jeff Langer points out in his chapter discussing Ken Beck’s four rules of Simple Design in Clean Code: “Our goal is to keep our overall system small while we are also keeping our functions and classes small. Remember however that this rule is the lowest priority of the four rules of Simple Design. So, although it’s important to keep class and function count low, it’s more important to have tests, eliminate duplication, and express yourself.” Sometimes it will take more than 30 lines (or 20 or 5 or whatever the cut-off is) to get a coherent piece of work done. It’s more important to be careful in coming up with the right abstractions and algorithms and to write clean clear code – if a cut-off guideline on size helps to do that, use it. If it doesn't, then don’t bother.

Testing threads is hard, very hard and this makes writing good integration tests for multithreaded systems under test... hard. This is because in JUnit there's no built in synchronisation between the test code, the object under test and any threads. This means that problems usually arise when you have to write a test for a method that creates and runs a thread. One of the most common scenarios in this domain is in making a call to a method under test, which starts a new thread running before returning. At some point in the future when the thread's job is done you need assert that everything went well. Examples of this scenario could include asynchronously reading data from a socket or carrying out a long and complex set of operations on a database. For example, the ThreadWrapper class below contains a single public method: doWork(). Calling doWork() sets the ball rolling and at some point in the future, at the discretion of the JVM, a thread runs adding data to a database. public class ThreadWrapper { /** * Start the thread running so that it does some work. */ public void doWork() { Thread thread = new Thread() { /** * Run method adding data to a fictitious database */ @Override public void run() { System.out.println("Start of the thread"); addDataToDB(); System.out.println("End of the thread method"); } private void addDataToDB() { // Dummy Code... try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } } }; thread.start(); System.out.println("Off and running..."); } } A straightforward test for this code would be to call the doWork() method and then check the database for the result. The problem is that, owing to the use of a thread, there's no co-ordination between the object under test, the test and the thread. A common way of achieving some co-ordination when writing this kind of test is to put some kind of delay in between the call to the method under test and checking the results in the database as demonstrated below: public class ThreadWrapperTest { @Test public void testDoWork() throws InterruptedException { ThreadWrapper instance = new ThreadWrapper(); instance.doWork(); Thread.sleep(10000); boolean result = getResultFromDatabase(); assertTrue(result); } /** * Dummy database method - just return true */ private boolean getResultFromDatabase() { return true; } } In the code above there is a simple Thread.sleep(10000) between two method calls. This technique has the benefit of being incredabile simple; however it's also very risky. This is because it introduces a race condition between the test and the worker thread as the JVM makes no guarantees about when threads will run. Often it'll work on a developer's machine only to fail consistently on the build machine. Even if it does work on the build machine it atificially lengthens the duration of the test; remember that quick builds are important. The only sure way of getting this right is to synchronise the two different threads and one technique for doing this is to inject a simple CountDownLatch into the instance under test. In the example below I've modified the ThreadWrapper class's doWork() method adding the CountDownLatch as an argument. public class ThreadWrapper { /** * Start the thread running so that it does some work. */ public void doWork(final CountDownLatch latch) { Thread thread = new Thread() { /** * Run method adding data to a fictitious database */ @Override public void run() { System.out.println("Start of the thread"); addDataToDB(); System.out.println("End of the thread method"); countDown(); } private void addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } } private void countDown() { if (isNotNull(latch)) { latch.countDown(); } } private boolean isNotNull(Object obj) { return latch != null; } }; thread.start(); System.out.println("Off and running..."); } } he Javadoc API describes a count down latch as: A synchronization aid that allows one or more threads to wait until a set of operations being performed in other threads completes. A CountDownLatch is initialized with a given count. The await methods block until the current count reaches zero due to invocations of the countDown() method, after which all waiting threads are released and any subsequent invocations of await return immediately. This is a one-shot phenomenon -- the count cannot be reset. If you need a version that resets the count, consider using a CyclicBarrier. A CountDownLatch is a versatile synchronization tool and can be used for a number of purposes. A CountDownLatch initialized with a count of one serves as a simple on/off latch, or gate: all threads invoking await wait at the gate until it is opened by a thread invoking countDown(). A CountDownLatchinitialized to N can be used to make one thread wait until N threads have completed some action, or some action has been completed N times. A useful property of a CountDownLatch is that it doesn't require that threads calling countDown wait for the count to reach zero before proceeding, it simply prevents any thread from proceeding past an await until all threads could pass. The idea here is that the test code will never check the database for the results until the run() method of the worker thread has called latch.countdown(). This is because the test code thread is blocking at the call to latch.await(). latch.countdown() decrements latch's count and once this is zero the blocking call the latch.await() returns and the test code continues executing, safe in the knowledge that any results which should be in the database, are in the database. The test can then retrieve these results and make a valid assertion. Obviously, the code above merely fakes the database connection and operations. The thing is you may not want to, or need to, inject a CountDownLatch directly into your code; after all it's not used in production and it doesn't look particularly clean or elegant. One quick way around this is to simply make the doWork(CountDownLatch latch) method package private and expose it through a public doWork() method. public class ThreadWrapper { /** * Start the thread running so that it does some work. */ public void doWork() { doWork(null); } @VisibleForTesting void doWork(final CountDownLatch latch) { Thread thread = new Thread() { /** * Run method adding data to a fictitious database */ @Override public void run() { System.out.println("Start of the thread"); addDataToDB(); System.out.println("End of the thread method"); countDown(); } private void addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } } private void countDown() { if (isNotNull(latch)) { latch.countDown(); } } private boolean isNotNull(Object obj) { return latch != null; } }; thread.start(); System.out.println("Off and running..."); } } The code above uses Google's Guava @VisibleForTesting annotation to tell us that the doWork(CountDownLatch latch) method visibility has been relaxed slightly for testing purposes. Now I realise that making a method call package private for testing purposes in highly controversial; some people hate the idea, whilst others include it everywhere. I could write a whole blog on this subject (and may do one day), but for me it should be used judiciously, when there's no other choice, for example when you're writing characterisation tests for legacy code. If possible it should be avoided, but never ruled out. After all tested code is better than untested code. With this in mind the next iteration of ThreadWrapper designs out the need for a method marked as @VisibleForTesting together with the need to inject a CountDownLatch into your production code. The idea here is to use the Strategy Pattern and separate the Runnable implementation from the Thread. Hence, we have a very simple ThreadWrapper public class ThreadWrapper { /** * Start the thread running so that it does some work. */ public void doWork(Runnable job) { Thread thread = new Thread(job); thread.start(); System.out.println("Off and running..."); } } and a separate job: public class DatabaseJob implements Runnable { /** * Run method adding data to a fictitious database */ @Override public void run() { System.out.println("Start of the thread"); addDataToDB(); System.out.println("End of the thread method"); } private void addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } } } You'll notice that the DatabaseJob class doesn't use a CountDownLatch. How is it synchronised? The answer lies in the test code below... public class ThreadWrapperTest { @Test public void testDoWork() throws InterruptedException { ThreadWrapper instance = new ThreadWrapper(); CountDownLatch latch = new CountDownLatch(1); DatabaseJobTester tester = new DatabaseJobTester(latch); instance.doWork(tester); latch.await(); boolean result = getResultFromDatabase(); assertTrue(result); } /** * Dummy database method - just return true */ private boolean getResultFromDatabase() { return true; } private class DatabaseJobTester extends DatabaseJob { private final CountDownLatch latch; public DatabaseJobTester(CountDownLatch latch) { super(); this.latch = latch; } @Override public void run() { super.run(); latch.countDown(); } } } The test code above contains an inner class DatabaseJobTester, which extends DatabaseJob. In this class the run() method has been overridden to include a call to latch.countDown() after our fake database has been updated via the call to super.run(). This works because the test passes a DatabaseJobTester instance to the doWork(Runnable job) method adding in the required thread testing capability. The idea of sub-classing objects under test is something I've mentioned before in one of my blogs on testing techniques and is a really powerful technique. So, to conclude: Testing threads is hard. Testing anonymous inner classes is almost impossible. Using Thead.sleep(...) is a risky idea and should be avoided. You can refactor out these problems using the Strategy Pattern. Programming is the Art of Making the Right Decision ...and that relaxing a method's visibility for testing may or may not be a good idea, but more on that later... The code above is available on Github in the captain debug repository (git://github.com/roghughe/captaindebug.git) under the unit-testing-threads project.

Lets continue the "Heroes of Java" series. Today's interview has been planned nearly since the launch of the series and I knew that it would be a tough one to get. I know Marcus since a few years now and he is always busy providing the best diagnostic tools to Java developers. Thanks for finally joining, Marcus! It is a pleasure to have you here! Marcus Hirt is one of the founders of Appeal Virtual Machines, the company that created the JRockit JVM. He is currently working as Team Lead for the Java Mission Control team. In his spare time he enjoys coding on his many pet projects, composing music, and scuba diving. Marcus has contributed JRockit related articles, whitepapers, tutorials, and webinars to the JRockit community, and has been an appreciated speaker at various conferences, such as Oracle Open World and Java One. He is also one of the two authors behind a popular book about JVM technology (link to my review). General Part Who are you? I am a computer aficionado with a strikingly unmodern and lengthy romance with typed languages, profiling and diagnostics. I have three kids and a lovely wife, so right now there isn't much spare time to go around. When there was, I used to compose music, play the piano, scuba dive and do martial arts. Your official job title at your company? Consulting Member of Technical Staff Do you care about it? I care about being appreciated for my work. The title itself means nothing. Do you speak foreign languages? Which ones? Swedish is my native tongue and my preferred language for anything that is not computer related. That said, since most of the terminology in our business is in English, I actually prefer English when talking shop. I am half Swiss, and I did spend some time at Real Gymnasium Kirchenfeld in Bern. I haven't used my German since then though, so it is beyond rusty. How long is your daily "bootstrap" process? (Coffee, news, email) If you don't count email, it is almost non-existent. However, in my role as a team leader, email is chewing up a good portion of the morning these days. Thanks to the excellent mass transit system in Stockholm, that is usually taken care of before I arrive at the office. At least during the winters. During the summers I usually drive my motorcycle to work. Twitter You have a twitter handle? Why? I indiscriminately sign up for all social services. Then I find that I don't use most of them. Twitter is a bit of a exception, since I do tend to read what others write. When I do tweet it is mostly about new obscure and/or unsupported features in the Hotspot JDK. My twitter handle is @hirt. Whom are you following in general? I mostly follow people that I know and respect in the Java community. Do you have a personal "policy" for twitter? I try to avoid it at work. Does your company restrict or encourage you with your twitter usage? Oracle has neither actively encouraged nor restricted my twitter usage. The only time I can recall a company actively encouraging me to engage in some official social capacity was some years ago, when BEA tried to encourage people to blog. I’ve since moved away from the official company blog, because of a tooling issue. Work What's your daily development setup? (OS/IDE/VC/other Tools) Since the first target platform for JRockit was Windows, I've stuck with Windows at work. I am using Windows 7/Eclipse/Perforce and Visual Studio. At home I am using Mac OS X/Eclipse/Git&Perforce and XCode. Which is the tool providing most productivity to your work? These days: Eclipse. No doubt. Your preferred way of interacting with co-workers? Face to face for longer discussions. IM is good for smaller things, since you can choose when to handle the interrupt, whilst still being fairly interactive. What's your favorite way of managing your todo's? Pen and paper. Stone age, right? If you could make a wish for a job at your favorite company: What would that be? Whichever would give me the resources to attack some of the high impact development projects on my "want to do" list. Oracle is currently quite a good place to be. Java You're programming in Java. Why? To be honest, I am not exclusively programming in Java. When I do, it is because it is one of the programming environments in which I find myself to be the most productive. It may not be the least verbose or most elegant of languages, but the tooling and debugging capabilities are top notch. Not to mention that some intrinsic features of the language itself, such as the memory management, makes it easier to write error free code. Also, since there has been competition around the JVM for more than a decade, the JVMs for Java are really quite sophisticated. Not to mention fast. What's least fun with Java? It is unnecessarily verbose (more type inference please), type erasure (ever sent in a class to your generic type to have a chance of knowing what runtime type it is?), and any and all things that makes the illusion of an all powerful runtime break down. In a perfect world, a Java programmer should not have to worry about the details of the JVM configuration. For instance, why should I need to estimate how much space I will need for constants and class metadata (perm gen)? Thankfully there is work being done on this as we speak; the perm gen is scheduled for removal in JDK 8. I think there is a lot to be said for improving the usability of the JVM. If you could change one thing with Java, what would that be? There are many who want Java to be everything to everyone. I don't subscribe to that view. Instead, let's make it easy to run whatever language you want on the JVM. That said, if I could change something about the implementation, I would probably want a thread local garbage collector. One with insanely good heuristics as to when to back off and stop handling an object thread locally. Then there are some other things, but since I may start working on them soon, I would rather keep them to myself for now. :) What's your personal favorite in dynamic languages? Ruby is cute. I especially like the implementation on the JVM (JRuby). Which programming technique has moved you forwards most and why? When I first started my education at the Royal Institute of Technology, I had already programmed in various languages, such as Pascal, C and assembly. I really thought I had things figured out, until I came to the first computer science course. There I got confronted with SICP and Scheme. That was IMHO a genius move by the computer science department. All the cocky kids with prior experience, such as myself, got a rich serving of humble-pie. Functional programming taught me very elegant ways of expressing myself. Kudos to MIT and Sussman et al. What was the biggest project you've ever worked on? JRockit and JRockit Mission Control. I was one of the founders of Appeal (Appeal Virtual Machines & Appeal Software Solutions), a big project in itself. Which was the worst programming mistake you did? Well, maybe not strictly a programming mistake, but one of the worst red-face issues I've done is when a JRockit performance counter was slightly misspelled - the 'o' was accidentally dropped from *count. The bug report stated that "the customer was not amused". I must admit I was though. Another fun, deliberate, "mistake" was when I added the following three lines to one of the Mission Control property files: ------ # :) We just felt that we needed this one translated... # /The MC team Template_DEFAULT_TEMPLATE_NAME=All your base are belong to us! ------ I was hoping to get a cynical remark back from the translation team, but they just translated it the best they could. Heh. Finally, one of the worst programming mistakes in recent history was in a small start-up project. A hash code calculation error caused some subtle errors to one of many data points in a running production system. I finally solved the problem when I got fiber installed at home and got bold. In desperation I started a node with jdwp turned on, and I then proceeded to set break points and evaluate code remotely over an ssh tunnel. The latency was so low that it almost felt like a local debugging session. Crazy, but you gotta love Java for providing you with options. ;)

This article will provide a code snippet and explains how to return Dictionary as result from a LINQ Query in C#. There are times when you want to retrieve only the ID(distinct) and the name from the database table using LINQ . In scenarios like this , one can use the ToDictionary method to place the necessary properties to the dictionary and return them. Below is a sample sourecode demonstrating the usage of ToDictionary method in LINQ Query public class BlockbusterMovie { public string Name { get; set; } public int ID { get; set; } } public class BlockbusterMovies : List { public BlockbusterMovies() { Add(new BlockbusterMovie { Name = "Vishwaroopam", ID = 1 }); Add(new BlockbusterMovie { Name = "Endhiran", ID = 2 }); Add(new BlockbusterMovie { Name = "Thuppaki", ID = 3 }); Add(new BlockbusterMovie { Name = "Mankatha", ID = 4 }); } } The BlockbusterMovies class has the collection of movies which is used in the below code snippet to return the dictionary based on the ID and Name. private void Form1_Load(object sender, EventArgs e) { List movies = new BlockbusterMovies(); var LstMovies = movies.ToDictionary(Field => Field.ID, mc => mc.Name); }

Ehcache in Spring 5 Minutes with Spring Cache Spring 3.1 and Ehcache Spring 3.1 and Ehcache 2 Spring 3.1 Cache Ehcache and Hibernate Docs Spring and JPA Cache Issues

NetBeans workspace coloring has traditionally preferred dark text over bright background. However, under some circumstances, e.g., in low-light conditions a darker workspace coloring may be beneficial or at least perceived as easier on the eyes. For those users preferring dark workspaces there were only insufficient options available so far. It has been possible to choose a dark editor color style, but such choice would not be reflected outside editor window - the workspace would thus suffer from the strong contrast between the dark editor panel and the bright background of the rest of the workspace. Although it is generally possible to change the look and feel of the operating system as a whole, this is not always desirable and always remains suboptimal with respect to NetBeans; for instance the design of icons can not be adjusted in this way. In current NetBeans daily builds (February 2013) we provide two new Look-and-Feels to address this issue. They offer two slightly different visual styles: Dark Nimbus provides dark content panels on brighter grayish workspace, while Dark Metal provides more uniformly dark workspace. The designs aim at providing well-balanced and legible dark IDE workspace as a whole. To switch NetBeans to one of the new LaFs do the following: in Tools->Options->Miscellaneous->Windows choose either Dark Metal or Dark Nimbus in the "Preferred look and feel" combo box. Then in Tools->Options->Fonts & Colors choose Norway Today in the "Profile" combo box. Restart NetBeans. The two new LaFs represent work in progress, but have already reached a reasonably usable state. We will keep them improving. Support for the two dark LaFs is too new and as such could not make it to NetBeans 7.3 release schedule, but will be added to NetBeans 7.3.1. It is already included in daily builds . A simpler version of Dark Nimbus is also available as downloadable plugin . Please report any remaining issues (especially legibility issues, but also improperly colored UI component leftovers) as part of or in relation to Bugzilla Issue #225542.