The post titled Installing Packages described the basics of package installation with R. The process is wonderfully simple when everything goes well. But it can be maddening when it does not. Error messages give a hint as to what went wrong but do not necessarily tell you how to resolve the problem. This post will collect some of the error messages we’ve encountered while installing R packages and describe the reasons for the error and the workarounds we’ve found. 1) Older version of R Warning message: In install.packages(c("sp")) : package ‘sp’ is not available This is the message that you get when the CRAN package you’re interested in requires a more recent version of R than you have. Remember, the default behavior ofinstall.packages() is to grab the latest version of a package. In this case you have to poke around in the “Old sources” link on the CRAN page for that package and use trial-and-error to find an older version of the package that will work with your version of R. You should start by determining what version of R you have: 1 2 $R--version Rversion2.8.1(2008-12-22) This version of R was released at the end of 2008 and any version of the “sp” package released in 2008 should work. At least some of the 2009 releases should also work. Perusing the sp archive, we might try installing version 0.9-37, the last of the 0.9-3x series which was released in May of 2009: 1 2 3 4 $wget http://cran.r-project.org/src/contrib/Archive/sp/sp_0.9-37.tar.gz $sudo CMD INSTALL sp_0.9-37.tar.gz ... $# Success! 2) Unable to execute files in /tmp directory ERROR: 'configure' exists but is not executable -- see the 'R Installation and Administration Manual' By default, R uses the /tmp directory to install packages. On security conscious machines, the /tmp directory is often marked as “noexec” in the /etc/fstab file. This means that no file under /tmp can ever be executed. Packages that require compilation or that have self-inflating data will fail with the error above. One such package isRJSONIO. The solution is to set the TMPDIR environment variable which R will use as the compilation directory. For csh shell: 1 2 $mkdir~/tmp $setenv TMPDIR~/tmp And for bash: 1 2 $mkdir~/tmp $export TMPDIR=~/tmp Other problems Please leave comments describing other package installation problems and solutions you’ve encountered to help us build a more complete listing.

When I want to insert figures generated in R into a LaTeX document, it looks better if I first remove the white space around the figure. Unfortunately, R does not make this easy as the graphs are generated to look good on a screen, not in a document. There are two things that can be done to fix this problem. First, you can reduce the white space generated by R. I use the following function when saving figures in R. savepdf <- function(file, width=16, height=10) { fname <- paste("figures/",file,".pdf",sep="") pdf(fname, width=width/2.54, height=height/2.54, pointsize=10) par(mgp=c(2.2,0.45,0), tcl=-0.4, mar=c(3.3,3.6,1.1,1.1)) } The width and height are in centimetres. The ratio is about right for a beamer presentation, and also to fit two figures on an A4 page. Then I use the commands savepdf("filename") # Plotting commands here dev.off() That will generate a pdf figure of about the right size and shape for a document, and with narrow margins of white space, and save it in my figures sub-directory. The second trick is to trim the pdf files so there is no white space left. On a unix system, this is easily achieved as follows. pdfcrop filename.pdf filename.pdf There are probably windows and mac versions of the same, but I haven’t used them. Adobe Acrobat will also crop pdfs, but not from the command line as far as I know. To apply pdfcrop to every file in a directory (using unix), save the following to a file called cropall.sh: #!/bin/bash for FILE in ./*.pdf; do pdfcrop "${FILE}" "${FILE}" done Make the file executable and run it. In my post on Makefiles, I explain how to include pdfcrop within a Makefile. If you just use pdfcrop without first reducing the white space in R, the proportions come out a little odd. So I tend to use both approaches together.

There are times when you want to retrieve the ID of the last inserted record when using Entity Framework. For example: Employee emp = new Employee(); emp.ID = -1; emp.Name = "Senthil Kumar B"; emp.Expertise = "ASP.NET MVC" EmployeeContext context = new EmployeeContext(); context.AddObject(emp); context.SaveChanges(); In the above example , if i need to retrieve the ID of the employee that was inserted , all that i need to do is use the emp.ID property once the data is saved as shown below. Employee emp = new Employee(); emp.ID = -1; emp.Name = "Senthil Kumar B"; emp.Expertise = "ASP.NET MVC" EmployeeContext context = new EmployeeContext(); context.AddObject(emp); context.SaveChanges(); int empID = emp.ID;

JDK6 update 7 and onward include a tool called VisualVM. VisualVM is a visual tool with monitoring and profiling capabilities for the JVM. With VisualVM you can: Monitor heap usage Monitor CPU usage Monitor Threads Initiate garbage collections Profile CPU and memory And more… Although VisualVM is distributed with the Oracle JDK, it can also be used to monitor IBM JVM’s. VisualVM is not able to connect to the IBM JVM locally. JMX must be used instead. To enable JMX monitoring on the IBM JVM open the WebSphere administrative console and: navigate to: Server -> Server Types -> WebSphere application servers ->[SERVER_NAME] Expand Java and Process Management and click Process definition Click Java Virtual Machine In the Generic JVM arguments field append the following properties: -Djavax.management.builder.initial= -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 Restart the server To start monitoring the JVM with VisualVM start VisualVM by navigating to [JDK_HOME]\bin and start jvisualvm.exe (please note that when VisualVM is downloaded as a separate package the executable is called visualvm.exe instead). In VisualVM click File -> Add JMX Connection. Specify localhost:1099 in the connection field and click OK. If everything went OK, you should see the localhost:1099 connection under the Local node in the tree on the left. Double-click this node to start monitoring. See the following screenshot for an example: When using a JMX connection to monitor the JVM please be aware that not all functionality can be used compared to monitoring a local JVM. Profiling memory is for example not possible. The above configuration was tested on: Windows 7 64-bit IBM JDK1.6 64 bit WebSphere Application Server version 7.0 Note: Before JDK6 update 7, VisualVM can also be downloaded separately from http://visualvm.java.net/download.html Note: To actually test if port 1099 is listening for connections use (on Windows) the netstat –a command and check wether the port is present and listening.

For people in hurry get the code from Github. In continuation of my earlier blog on spring-test-mvc junit testing Spring Security layer with InMemoryDaoImpl, in this blog I will discuss how to use achieve method level access control. Please follow the steps in this blog to setup spring-test-mvc and run the below test case. mvn test -Dtest=com.example.springsecurity.web.controllers.SecurityControllerTest The JUnit test case looks as below, @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(loader = WebContextLoader.class, value = { "classpath:/META-INF/spring/services.xml", "classpath:/META-INF/spring/security.xml", "classpath:/META-INF/spring/mvc-config.xml" }) public class SecurityControllerTest { @Autowired CalendarService calendarService; @Test public void testMyEvents() throws Exception { Authentication auth = new UsernamePasswordAuthenticationToken("[email protected]", "user1"); SecurityContext securityContext = SecurityContextHolder.getContext(); securityContext.setAuthentication(auth); calendarService.findForUser(0); SecurityContextHolder.clearContext(); } @Test(expected = AuthenticationCredentialsNotFoundException.class) public void testForbiddenEvents() throws Exception { calendarService.findForUser(0); } } @Test(expected=AccessDeniedException.class) public void testWrongUserEvents() throws Exception { Authentication auth = new UsernamePasswordAuthenticationToken("[email protected]", "user2"); SecurityContext securityContext = SecurityContextHolder.getContext(); securityContext.setAuthentication(auth); calendarService.findForUser(0); SecurityContextHolder.clearContext(); } If you notice, if the user did not login or if the user is trying to access another users information it will throw an exception. The interface access control is as below, public interface CalendarService { @PreAuthorize("hasRole('ROLE_ADMIN') or principal.id == #userId") List findForUser(int userId); } The PreAuthorize only works on interface so that any implementation that implements this interface has this access control. I hope this blog helps you.

The Lamborghini of Key-Value stores Camel is the best of bread Integration framework and in this post I'm going to show you how to make it even more powerful by leveraging another great project - Redis. Camel 2.11 is on its way to be released soon with lots of new features, bug fixes and components. Couple of these new components are authored by me, redis-component being my favourite one. Redis - a ligth key/value store is an amazing piece of Italian software designed for speed (same as Lamborghini - a two-seater Italian car designed for speed). Written in C and having an in-memory closer to the metal nature, Redis performs extremely well (Lamborgini's motto is "Closer to the Road"). Redis is often referred to as a data structure server since keys can contain strings, hashes, lists and sorted sets. A fast and light data structure server is like a super sportscars for software engineers - it just flies. If you want to find out more about Redis' and Lamborghini's unique performance characteristics google around and you will see for yourself. Getting started with Redis is easy: download, make, and start a redis-server. After these steps, you ready to use it from your Camel application. The component uses internally Spring Data which in turn uses Jedis driver, but with possibility to switch to other Redis drivers. Here are few use cases where the camel-redis component is a good fit: Idempotent Repository The term idempotent is used in mathematics to describe a function that produces the same result if it is applied to itself. In Messaging this concepts translates into the a message that has the same effect whether it is received once or multiple times. In Camel this pattern is implemented using the IdempotentConsumer class which uses an Expression to calculate a unique message ID string for a given message exchange; this ID can then be looked up in the IdempotentRepository to see if it has been seen before; if it has the message is consumed; if its not then the message is processed and the ID is added to the repository. RedisIdempotentRepository is using a set structure to store and check for existing Ids. ${in.body.id} Caching One of the main uses of Redis is as LRU cache. It can store data inmemory as Memcached or can be tuned to be durable flushing data to a log file that can be replayed if the node restarts.The various policies when maxmemory is reached allows creating caches for specific needs: volatile-lru remove a key among the ones with an expire set, trying to remove keys not recently used. volatile-ttl remove a key among the ones with an expire set, trying to remove keys with short remaining time to live. volatile-random remove a random key among the ones with an expire set. allkeys-lru like volatile-lru, but will remove every kind of key, both normal keys or keys with an expire set. allkeys-random like volatile-random, but will remove every kind of keys, both normal keys and keys with an expire set. Once your Redis server is configured with the right policies and running, the operation you need to do are SET and GET: SET keyOne valueOne Interap pub/sub with Redis Camel has various components for interacting between routes: direct: provides direct, synchronous invocation in the same camel context. seda: asynchronous behavior, where messages are exchanged on a BlockingQueue, again in the same camel context. vm: asynchronous behavior like seda, but also supports communication across CamelContext as long as they are in the same JVM. Complex applications usually consist of more than one standalone Camel instances running on separate machines. For this kind of scenarios, Camel provides jms, activemq, combination of AWS SNS with SQS, for messaging between instances. Redis has a simpler solution for the Publish/Subscribe messaging paradigm. Subscribers subscribes to one or more channels, by specifying the channel names or using pattern matching for receiving messages from multiple channels. Then the publisher publishes the messages to a channel, and Redis makes sure it reaches all the matching subscribers. PUBLISH testChannel Test Message Other usages Guaranteed Delivery: Camel supports this EIP using JMS, File, JPA and few other components. Here Redis can be used as lightweight key-value persistent store with its transaction support. The Claim Check from the EIP patterns allows you to replace message content with a claim check (a unique key), which can be used to retrieve the message content at a later time. The message content can be stored temporarily in Redis. Redis is also very popular for implementing counters, leaderboards, tagging systems and many more functionalities. Now, with two swiss army knives under your belt, the integrations to make are limited only by your imagination.

Scala offers a functionality known as Structural Types which allows to set a behaviour very similar to what dynamic languages allow to do when they support Duck Typing (http://en.wikipedia.org/wiki/Duck_typing) The main difference is that it is a type safe, static typed implementation checked up at compile time. This means that you can create a function (or method) that receives an expected duck. But at compile time it would be checked that anything that is passed can actually quack like a Duck. Here is an example. Let’s say we want to create a function that expects anything that can quack like a duck. This is how we would do it in Scala with Structural Typing: def quacker(duck: {def quack(value: String): String}) { println (duck.quack("Quack")) } You can see that in the definition of the function we are not expecting a particular class or type. We are specifying an Structural Type which in this case means that we are expecting any type that has a method with the signature quack(string: String): String. So all the following examples will work with that function: object BigDuck { def quack(value: String) = { value.toUpperCase } } object SmallDuck { def quack(value: String) = { value.toLowerCase } } object IamNotReallyADuck { def quack(value: String) = { "prrrrrp" } } quacker(BigDuck) quacker(SmallDuck) quacker(IamNotReallyADuck) You can see that there is no interface or anything being implemented by any of the three objects we have defined. They simply have to define the method quack in order to work in our function. If you run the code above you get the output: QUACK quack prrrrrp If on the other hand you try to create an object without a quack method and try to call the function with that object you would get a compile error. For example trying to do this: object NoQuaker { } quacker(NoQuaker) You would get the error: error: type mismatch; found : this.NoQuaker.type required: AnyRef{def quack(value: String): String} quacker(NoQuaker) Also, you don’t even need to create a new type or class. You could use AnyRef to create an object with the quack method. Like this: val x = new AnyRef { def quack(value: String) = { "No type needed "+ value } } and you can use that object to call the function: quacker(x) You can also specify in the function that expects the structural type, the the parameter object must respond to more than one method. Like this: def quacker(duck: {def quack(value: String): String; def walk(): String}) { println (duck.quack("Quack")) } There you are saying that any object you pass to the function needs to respond to both methods quack and walk. This is also checked at compile time. Under the covers the use of Structural Types in this way will be handled by reflection. This means that it is a more expensive operation than the standard method call. So use only when it actually makes sense to use it.

Dependency inversion is the idea that interfaces should depend on abstractions not on specifics. According to Wikipedia, the principle states: A. High-level modules should not depend on low-level modules. Both should depend on abstractions. B. Abstractions should not depend upon details. Details should depend upon abstractions. Of course the second part of this principle is impossible if read literally. You can't have an abstraction until you know what details are to be covered, and so the abstraction and details are both co-dependent. If the covered details change sufficiently the abstraction will become either leaky or inadequate and so it is worth seeing these as intertwined to some extent. The focus on abstraction is helpful because it suggests that the interface contract should be designed in such a way that neither side really has to understand any internal details of the other in order to make things work. Both sides depend on well-encapsulated API's and neither side has to worry about what the other side is really doing. This is what is meant by details depending on abstractions rather than the other way around. This concept is quite applicable beyond object oriented programming because it covers a very basic aspect of API contract design, namely how well an API should encapsulate behavior. This principle is first formulated in its current form in the object oriented programming paradigm but is generally applicable elsewhere. SQL as an Abstraction Layer, or Why RDBMS are Still King There are plenty of reasons to dislike SQL, such as the fact that nulls are semantically ambiguous. As a basic disclaimer I am not holding SQL up to be a paragon of programming languages or even db interfaces, but I think it is important to discuss what SQL does right in this regard. SQL is generally understood to be a declarative language which approximates relational mathematics for database access purposes. With SQL, you specify what you want returned, not how to get it, and the planner determines the best way to get it. SQL is thus an interface language rather than a programming language per se. With SQL, you can worry about the logical structure, leaving the implementation details to the db engine. SQL queries are basically very high level specifications of operations, not detailed descriptions of how to do something efficiently. Even update and insert statements (which are by nature more imperative than select statements) leave the underlying implementation entirely to the database management system. I think that this, along with many concessions the language has made to real-world requirements (such as bags instead of sets and the addition of ordering to bags) largely account for the success of this language. SQL, in essence, encapsulates a database behind a mature mathematical, declarative model in the same way that JSON and REST do (in a much less comprehensive way) in many NoSQL db's. In essence SQL provides encapsulation, interface, and abstraction in a very full-featured way and this is why it has been so successful. SQL Abstraction as Imperfect One obvious problem with treating SQL as an abstraction layer in its own right is that one is frequently unable to write details in a way that is clearly separate from the interface. Often storage tables are hit directly, and therefore there is little separation between logical detail and logical interface, and so this can break down when database complexity reaches a certain size. Approaches to managing this problem include using stored procedures or user defined functions, and using views to encapsulate storage tables. Stored Procedures and User Defined Functions Done Wrong Of the above methods, stored procedures and functional interfaces have bad reputations frequently because of bad experiences that many people have with them. These include developers pushing too much logic into stored procedures, and the fact that defining functional interfaces in this way usually produces a very tight binding between database code and application code, often leading to maintainability problems. The first case is quite obvious, and includes the all-too-frequent case of trying to send emails directly from stored procedures (always a bad idea). This mistake leads to certain types of problems, including the fact that ACID-compliant operations may be mixed with non-ACID-compliant ones, leading to cases where a transaction can only be partially rolled back. Oops, we didn't actually record the order as shipped, but we told the customer it was..... MySQL users will also note this is an argument against mixing transactional and nontransactional backend table types in the same db..... However that problem is outside the scope of this post. Additionally, MySQL is not well suited for many applications against a single set of db relations. The second problem, though, is more insidious. The traditional way stored procedures and user defined functions are typically used, the application has to be deeply aware of the interface to the database, but the rollout for these aspects is different leading to the possibility or service interruptions, and a need to very carefully and closely time rollout of db changes with application changes. As more applications use the database, this becomes harder and the chance of something being overlooked becomes greater. For this reason the idea that all operations must go through a set of stored procedures is a decision fraught with hazard as the database and application environment evolves. Typically it is easier to manage backwards-compatibility in schemas than it is in functions and so a key question is how many opportunities you have to create new bugs when a new column is added. There are, of course, more hazards which I have dealt with before, but the point is that stored procedures are potentially harmful and a major part of the reason is that they usually form a fairly brittle contract with the application layer. In a traditional stored procedure, adding a column to be stored will require changing the number of variables in the stored procedure's argument list, the queries to access it, and each application's call to that stored procedure. In this way, they provide (in the absence of other help) at best a leaky abstraction layer around the database details. This is the sort of problem that dependency inversion helps to avoid. Stored Procedures and User Defined Functions Done Right Not all stored procedures are done wrong. In the LedgerSMB project we have at least partially solved the abstraction/brittleness issue by looking to web services for inspiration. Our approach provides an additional mapping layer and dynamic query generation around a stored procedure interface. By using a service locator pattern, and overloading the system tables in PostgreSQL as the service registry, we solve the problem of brittleness. Our approach of course is not perfect and it is not the only possibility. One shortcoming is that our approach is that the invocation of the service locator is relatively spartan. We intend to allow more options there in the future. However one thing I have noticed is the fact that there are far fewer places where bugs can hide and therefore faster and more robust development takes place. Additionally a focus on clarity of code in stored procedures has eliminated a number of important performance bottlenecks, and it limits the number of places where a given change propagates to. Other Important Options in PostgreSQL Stored procedures are not the only abstraction mechanisms available from PostgreSQL. In addition to views, there are also other interesting ways of using functions to accomplish this without insisting that all access goes through stored procedures. In addition these methods can be freely mixed to produce very powerful, intelligent database systems. Such options include custom types, written in C, along with custom operators, functions and the like. These would then be stored in columns and SQL can be used to provide an abstraction layer around the types. In this way SQL becomes the abstraction and the C programs become the details. A future post will cover the use of ip4r in network management with PostgreSQL db's as an example of what can be done here. Additionally, things like triggers and notifications can be used to ensure that appropriate changes trigger other changes in the same transaction or, upon transaction commit, hand off control to other programs in subsequent transactions (allowing for independent processing and error control for things like sending emails). Recommendations Rather than specific recommendations, the overall point here is to look at the database itself as a an application running in an application server (the RDBMS) and design it as an application with an appropriate API. There are many ways to do this, from writing components in C and using SQL as an abstraction mechanism to writing things in SQL and using stored procedures as a mechanism. One could even write code in SQL and still use SQL as an abstraction mechanism. The key point however is to be aware of the need for discoverable abstraction, a need which to date things like ORMs and stored procedures often fill very imperfectly. A well designed db with appropriate abstraction in interfaces, should be able to be seen as an application in its own right, engineered as such, and capable of serving multiple client apps through a robust and discoverable API. As with all things, it starts by recognizing the problems and putting solutions as priorities from the design stage onward.

As I mentioned in a previous post I’ve been playing around with some football related data over the last few days and one query I ran (using cypher) was to find all the players who’ve been sent off this season in the Premiership. The model in the graph around sending offs looks like this: My initial query looked like this: START player = node:players('name:*') MATCH player-[:sent_off_in]-game-[:in_month]-month RETURN player.name, month.name First we get the names of all the players which are stored in an index and then we follow relationships to the games they were sent off in and then find which months those games were played in. That query returns: +----------------------------+ | player.name | month.name | +----------------------------+ | "Jenkinson" | "February" | | "Chico" | "September" | | "Odemwingie" | "September" | | "Agger" | "August" | | "Cole" | "December" | | "Whitehead" | "August" | ... +----------------------------+ I thought it’d be interesting to see how many sending offs there were in each month which we’d achieve in SQL by making use of a GROUP BY. cypher has a bunch of aggregation functions which allow us to achieve the same outcome. In our case we want to use the COUNT function and we want our grouping key to be the month of the year so we need to include that as part of our RETURN statement as well: START player = node:players('name:*') MATCH player-[:sent_off_in]-game-[:in_month]-month RETURN COUNT(player.name) AS numberOfReds, month.name ORDER BY numberOfReds DESC which returns: +----------------------------+ | numberOfReds | month.name | +----------------------------+ | 7 | "October" | | 6 | "December" | | 4 | "September" | | 4 | "November" | | 3 | "August" | | 2 | "January" | | 2 | "February" | +----------------------------+ As far as I can tell anything which isn’t an aggregate function is used as part of the grouping key which means we could include more than one field in our grouping key. This isn’t particularly relevant for us for this particular query but would become useful if we add the teams that the players play for. I extended the graph to included a player’s statistics for each game which also includes a relationship indicating which team they played for in a specific game. The model now looks like this: It does now look quite a bit more complicated but this was the best way I could think of modelling player specific details for a match. I couldn’t see another way of modelling the fact that a player played for a certain team in a match which I want to use for some other queries but if you can see a simpler way please let me know. To get a list of the red cards and the name of the team the offender played for we can write the following query: START player = node:players('name:*') MATCH player-[:sent_off_in]-game-[:in_month]-month, game-[:in_match]-stats-[:stats]-player, stats-[:played_for]-team RETURN player.name, month.name, team.name ORDER BY month.name The original query traversed a path from a player to games they were sent off in and then from the games to the month the game was played in. We’ve now added a traversal from the game to the game stats for that player and we also traverse from the game stats to the team node that the player played for in that game. When we run this we get the following results: +--------------------------------------------+ | player.name | month.name | team.name | +--------------------------------------------+ | "Agger" | "August" | "Liverpool" | | "Whitehead" | "August" | "Stoke" | ... | "Shotton" | "December" | "Stoke" | | "Nzonzi" | "December" | "Stoke" | | "Jenkinson" | "February" | "Arsenal" | ... | "Ivanovic" | "October" | "Chelsea" | | "Torres" | "October" | "Chelsea" | +--------------------------------------------+ So we can see that Stoke got 2 players sent off in December and Chelsea got 2 sent off in October. We can write the following query to return a result set which uses team and month as the grouping key i.e. we count how many paths there are which have the same team and month: START player = node:players('name:*') MATCH player-[:sent_off_in]-game-[:in_month]-month, game-[:in_match]-stats-[:stats]-player, stats-[:played_for]-team RETURN month.name, team.name, COUNT(player.name) AS numberOfReds ORDER BY numberOfReds DESC When we run that query we see the following results: +--------------------------------------------+ | month.name | team.name | numberOfReds | +--------------------------------------------+ | "December" | "Stoke" | 2 | | "October" | "Chelsea" | 2 | ... | "August" | "Stoke" | 1 | | "November" | "Tottenham" | 1 | | "December" | "Everton" | 1 | +--------------------------------------------+ This is all explained in more detail in the documentation but I thought it’d be interesting to write about it from the perspective of someone more used to writing SQL and trying to work out how to achieve the same thing in cypher.

This article wants to bring some hints about how to use HTML5 canvas with Apache Wicket web framework. Inside a Wicket application we want to have a panel with something drawn inside a HTML5 canvas. To make this happen we have to think about following: Do we really need HTML5? If we need HTML5, how to do it? What to do if browser version is an issue and it does not support HTML5? 1. First we should ask if we really need HTML5 If we need just an image then we should consider to draw inside a Java2D Graphics object. If we need some animation we should consider to draw inside a HTML5 canvas, but even in this case we need a simple Java2D image implementation if browser version is a concern and canvas is not supported. Wicket has a RenderedDynamicImageResource class which is very handy for this because we can do Java2D stuff inside render(Graphics2D g2) method. A simple example may look like the following: public class MyDynamicImageResource extends RenderedDynamicImageResource { private int width; private int height; private MyData data; public MyDynamicImageResource (int width, int height, MYData data) { super(width, height); this.width = width; this.height = height; this.data = data; } protected boolean render(Graphics2D g2) { g2.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON); g2.setRenderingHint(RenderingHints.KEY_RENDERING, RenderingHints.VALUE_RENDER_QUALITY); // your code } } Because Java2d is used, we can set anti-aliasing to make the image look good. Then we can use this dynamic resource to create our panel. We will use Wicket's NonCachingImage class, a subclass of Image that adds random noise to the url at every request to prevent the browser from caching the image. If you do not care that browser caches your image then you should use a simple Image instead. public class MyJava2DImagePanel extends Panel { private MyDynamicImageResource imageResource; public MyJava2DImagePanel(String id, final int width, final int height, final IModel model) { super(id, model); NonCachingImage image = new NonCachingImage("myImage", new PropertyModel(this, "imageResource")) { private static final long serialVersionUID = 1L; @Override protected void onBeforeRender() { imageResource = new MyDynamicImageResource(width, height, model.getObject()); super.onBeforeRender(); } }; add(image); } } Markup html file for MyJava2DImagePanel will contain the image: 2. If we need some animation for our image, then we should think to draw it on a HTML5 canvas. We should pay attention to draw things just once, meaning for example if we draw a text twice in same position , then our result will look ugly (pix-elated) because an anti-aliasing for canvas cannot be set as for Java2D Graphics object. First we need to create our java script code. We can obtain a Java 2d context and use it to draw our image. I won't talk about canvas context and its methods here. For animation we use jquery in following snippet, but you can use anything you like. Knowing two values (from, to) we can have for example a drawColor method which can paint different segments, creating this way a filling effect which takes in this example 1000ms : var myWidget = function(id, color) { var can = document.getElementById(id); var ctx = can.getContext('2d'); // clear canvas ctx.clearRect(0, 0, can.width, can.height); // draw your image on ctx ..... // animate color fill $({ n: from }).animate({ n: to}, { duration: 1000, step: function(now, fx) { drawColor(id, now); } }); } } Second we have to create our Wicket panel. Canvas is just a WebMarkupContainer and we set width and height through some AttributeAppenders: public class MyHTML5Panel extends Panel { private final ResourceReference MY_JS = new JavaScriptResourceReference(MyHTML5Panel.class, "my.js"); public MyHTML5Panel(String id, String width, String height, IModel model) { super(id, model); WebMarkupContainer container = new WebMarkupContainer("canvas"); container.setOutputMarkupId(true); container.add(new AttributeAppender("width", width)); container.add(new AttributeAppender("height", height)); add(container); } @Override public void renderHead(IHeaderResponse response) { response.renderOnLoadJavaScript(getJavascriptCall()); //include js file response.renderJavaScriptReference(MY_JS); } private String getJavascriptCall() { MyData data = getModel().getObject(); StringBuilder sb = new StringBuilder(); sb.append("myWidget(\""). append(get("canvas").getMarkupId()). append("\",\"").append(data.getColor()). append("\");"); return sb.toString(); } } renderHead(IHeaderResponse response) method from Panel can use the IHeaderResponse object to render our java script call. Also, on the response object we should render our java script reference file. We can use one of the following methods: /** * Renders javascript that is executed right after the DOM is built, before external resources * (e.g. images) are loaded. * * @param javascript */ public void renderOnDomReadyJavaScript(String javascript); /** * Renders javascript that is executed after the entire page is loaded. * * @param javascript */ public void renderOnLoadJavaScript(String javascript); There are situations when we should call one or another depending on our business. As an example, if we need to expose our wicket component to an external iframe, we must call onLoad instead of onDomReady to make it appear inside iframe because $(document).ready in the iframe seems to be fired too soon and the iframe content isn't even loaded yet. HTML markup file MyHTML5Panel.html will contain the canvas tag: 3. If we choose to use HTML5 panel but we also have to think about older browser that cannot support canvas tag, we will have to create both a Java2D and a HTML5 panel and see what to render by ourselves. A solution is to have a wrapper panel with a container which initially contains an EmptyPanel and we add a Wicket Behavior to the container. That behavior will choose what to render (html5 or simple image): ..... container = new WebMarkupContainer("container"); container.setOutputMarkupId(true); container.add(new EmptyPanel("image")); add(container); add(new MyHTML5Behavior()); ....... The following java-script code is a way to test if canvas tag is supported by browser: function isCanvasEnabled() { return !!document.createElement('canvas').getContext; } This function starts by creating a dummy element which is never attached to the page, so no one will ever see it. As soon as we create the dummy element, we test for the presence of a getContext() method. This method will only exist if browser supports the canvas API. Finally, we use the double-negative trick to force the result to a Boolean value (true or false). To call this java script and make the result available to Wicket we use wicketAjaxGet javascript method as seen in following code. We append a result parameter to callback url and inside respond method we can read the value of this parameter. class MyHTML5Behavior extends AbstractDefaultAjaxBehavior { private String width; private String height; private String PARAM = "Param"; public MyHTML5Behavior() { super(); } @Override public void renderHead(Component component, IHeaderResponse response) { super.renderHead(component, response); //include js file response.renderJavaScriptReference(MY_UTIL_JS); response.renderOnLoadJavaScript(getJavascript()); } @Override protected void respond(AjaxRequestTarget target) { String param = this.getComponent().getRequest().getRequestParameters().getParameterValue(PARAM).toString(); // test if html5 canvas tag is supported if (Boolean.parseBoolean(param)) { container.replace(new MyHTML5Panel("image", width, height, model).setOutputMarkupId(true)); } else { container.replace(new MyImagePanel("image", width, height, model).setOutputMarkupId(true)); } target.add(container); } // this javascript call will make the PARAM available to wicket and can be read in respond method private String getJavascript() { StringBuilder sb = new StringBuilder(); sb.append("var data = isCanvasEnabled();"); sb.append("wicketAjaxGet('" + getCallbackUrl() + "&" + PARAM + "='+ data" + ", null, null, function() { return true; })"); return sb.toString(); } } These are just some hints on how to use HTML5 canvas inside Apache Wicket framework. I hope it will help others.

JavaFX provides certain colors by default when rendering charts. There are situations, however, when one wants to customize these colors. In this blog post I look at changing the colors of a JavaFX pie chart using an example I intend to include in my presentation this afternoon at RMOUG Training Days 2013. Some Java-based charting APIs provided Java methods to set colors. JavaFX, born in the days of HTML5 prevalence, instead uses Cascading Style Sheets (CSS) to allow developers to adjust colors, symbols, placement, alignment and other stylistic issues used in their charts. I demonstrate using CSS to change colors here. In this post, I will look at two code samples demonstrating simple JavaFX applications that render pie charts based on data from Oracle's sample 'hr' schema. The first example does not specify colors and so uses JavaFX's default colors for pie slices and for the legend background. That example is shown next. EmployeesPerDepartmentPieChart (Default JavaFX Styling) package rmoug.td2013.dustin.examples; import javafx.application.Application; import javafx.scene.Scene; import javafx.scene.chart.PieChart; import javafx.scene.layout.StackPane; import javafx.stage.Stage; /** * Simple JavaFX application that generates a JavaFX-based Pie Chart representing * the number of employees per department. * * @author Dustin */ public class EmployeesPerDepartmentPieChart extends Application { final DbAccess databaseAccess = DbAccess.newInstance(); @Override public void start(final Stage stage) throws Exception { final PieChart pieChart = new PieChart( ChartMaker.createPieChartDataForNumberEmployeesPerDepartment( this.databaseAccess.getNumberOfEmployeesPerDepartmentName())); pieChart.setTitle("Number of Employees per Department"); stage.setTitle("Employees Per Department"); final StackPane root = new StackPane(); root.getChildren().add(pieChart); final Scene scene = new Scene(root, 800 ,500); stage.setScene(scene); stage.show(); } public static void main(final String[] arguments) { launch(arguments); } } When the above simple application is executed, the output shown in the next screen snapshot appears. I am now going to adapt the above example to use a custom "theme" of blue-inspired pie slices with a brown background on the legend. Only one line is needed in the Java code to include the CSS file that has the stylistic specifics for the chart. In this case, I added several more lines to catch and print out any exception that might occur while trying to load the CSS file. With this approach, any problems loading the CSS file will lead simply to output to standard error stating the problem and the application will run with its normal default colors. EmployeesPerDepartmentPieChartWithCssStyling (Customized CSS Styling) package rmoug.td2013.dustin.examples; import javafx.application.Application; import javafx.scene.Scene; import javafx.scene.chart.PieChart; import javafx.scene.layout.StackPane; import javafx.stage.Stage; /** * Simple JavaFX application that generates a JavaFX-based Pie Chart representing * the number of employees per department and using style based on that provided * in CSS stylesheet chart.css. * * @author Dustin */ public class EmployeesPerDepartmentPieChartWithCssStyling extends Application { final DbAccess databaseAccess = DbAccess.newInstance(); @Override public void start(final Stage stage) throws Exception { final PieChart pieChart = new PieChart( ChartMaker.createPieChartDataForNumberEmployeesPerDepartment( this.databaseAccess.getNumberOfEmployeesPerDepartmentName())); pieChart.setTitle("Number of Employees per Department"); stage.setTitle("Employees Per Department"); final StackPane root = new StackPane(); root.getChildren().add(pieChart); final Scene scene = new Scene(root, 800 ,500); try { scene.getStylesheets().add("chart.css"); } catch (Exception ex) { System.err.println("Cannot acquire stylesheet: " + ex.toString()); } stage.setScene(scene); stage.show(); } public static void main(final String[] arguments) { launch(arguments); } } The chart.css file is shown next: chart.css /* Find more details on JavaFX supported named colors at http://docs.oracle.com/javafx/2/api/javafx/scene/doc-files/cssref.html#typecolor */ /* Colors of JavaFX pie chart slices. */ .data0.chart-pie { -fx-pie-color: turquoise; } .data1.chart-pie { -fx-pie-color: aquamarine; } .data2.chart-pie { -fx-pie-color: cornflowerblue; } .data3.chart-pie { -fx-pie-color: blue; } .data4.chart-pie { -fx-pie-color: cadetblue; } .data5.chart-pie { -fx-pie-color: navy; } .data6.chart-pie { -fx-pie-color: deepskyblue; } .data7.chart-pie { -fx-pie-color: cyan; } .data8.chart-pie { -fx-pie-color: steelblue; } .data9.chart-pie { -fx-pie-color: teal; } .data10.chart-pie { -fx-pie-color: royalblue; } .data11.chart-pie { -fx-pie-color: dodgerblue; } /* Pie Chart legend background color and stroke. */ .chart-legend { -fx-background-color: sienna; } Running this CSS-styled example leads to output as shown in the next screen snapshot. The slices are different shades of blue and the legend's background is "sienna." Note that while I used JavaFX "named colors," I could have also used "#0000ff" for blue, for example. I did not show the code here for my convenience classes ChartMaker and DbAccess. The latter simply retrieves the data for the charts from the Oracle database schema via JDBC and the former converts that data into the Observable collections appropriate for the PieChart(ObservableList) constructor. It is important to note here that, as Andres Almiray has pointed out, it is not normally appropriate to execute long-running processes from the main JavaFX UI thread (AKA JavaFX Application Thread) as I've done in this and other other blog post examples. I can get away with it in these posts because the examples are simple, the database retrieval is quick, and there is not much more to the chart rendering application than that rendering so it is difficult to observe any "hanging." In a future blog post, I intend to look at the better way of handling the database access (or any long-running action) using the JavaFX javafx.concurrent package (which is well already well described in Concurrency in JavaFX). JavaFX allows developers to control much more than simply chart colors with CSS. Two very useful resources detailing what can be done to style JavaFX charts with CSS are the Using JavaFX Charts section Styling Charts with CSS and the JavaFX CSS Reference Guide. CSS is becoming increasingly popular as an approach to styling web and mobile applications. By supporting CSS styling in JavaFX, the same styles can easily be applied to JavaFX apps as the HTML-based applications they might coexist with.

Futures are very important abstraction, even more these day than ever due to growing demand for asynchronous, event-driven, parallel and scalable systems.

I find myself working more and more with Spring these days, and what I find raises questions. This week, my thoughts are turned toward beans overwriting, that is registering more than one bean with the same name. In the case of a simple project, there’s no need for this; but when building a a plugin architecture around a core, it may be a solution. Here are some facts I uncovered and verified regarding beans overwriting. Single bean id per file The id attribute in the Spring bean file is of type ID, meaning you can have only a single bean with a specific ID in a specific Spring beans definition file. Overwriting bean dependent on context fragments loading order As opposed to classpath loading where the first class takes priority over those others further on the classpath, it’s the last bean of the same name that is finally used. That’s why I called it overwriting. Reversing the fragment loading order proves that. Fragment assembling methods define an order Fragments can be assembled from statements in the Spring beans definition file or through an external component (e.g. the Spring context listener in a web app or test classes). All define a deterministic order. As a side note, though I formerly used import statements in my projects (in part to take advantage of IDE support), experience taught me it can bite you in the back when reusing modules: I’m in favor of assembling through external components now. Names Spring lets you define names in addition to ids (which is a cheap way of putting illegals characters fors ID). Those names also overwrites ids. Aliases Spring lets you define aliases of existing beans: those aliases also overwrites ids. Scope overwriting This one is really mean: by overwriting a bean, you also overwrite scope. So, if the original bean had a specified scope and you do not specify the same, tough luck: you just probably changed the application behavior. Not only are perhaps not known by your development team, but the last one is the killer reason not to overwrite beans. It’s too easy to forget scoping the overwritten bean. In order to address plugins architecture, and given you do not want to walk the OSGi path, I would suggest what I consider a KISS (yet elegant) solution. Let us use simple Java properties in conjunction with ProperyPlaceholderConfigurer. The main Spring Beans definition file should define placeholders for beans that can be overwritten and read two defined properties file: one wrapped inside the core JAR and the other on a predefined path (eventually set by a JVM property). Both property files have the same structure: fully-qualified interface names as keys and fully-qualified implementations names as values. This way, you define default implementations in the internal property file and let uses overwrite them in the external file (if necessary). As an added advantage, it shields users from Spring so they are not tied to the framework. Sources for this article can be found in Maven/Eclipse format here.

Yes, it is long time since i posted… Was just trying to see how a XML can be converted to JSON and to HashMap. The situation is very imaginary. import java.io.File; import java.io.FileInputStream; import java.io.IOException; import java.io.InputStream; import java.util.ArrayList; import java.util.List; import java.util.Map; import net.sf.json.JSON; import net.sf.json.xml.XMLSerializer; import org.apache.commons.io.IOUtils; import org.codehaus.jackson.JsonGenerationException; import org.codehaus.jackson.map.JsonMappingException; import org.codehaus.jackson.map.ObjectMapper; import org.codehaus.jackson.type.TypeReference; public class XML2JSONConvertor { public static void main(String[] args) throws Exception { InputStream is = new FileInputStream(new File( “e:\\jagannathan\\personal\\java-projects\\secondtest.xml”)); String xml = IOUtils.toString(is); XMLSerializer xmlSerializer = new XMLSerializer(); JSON json = xmlSerializer.read(xml); System.out.println(json.toString(2)); printJSON(json.toString(2)); } public static void printJSON(String jsonString) { ObjectMapper mapper = new ObjectMapper(); try { Map jsonInMap = mapper.readValue(jsonString, new TypeReference>() { }); List keys = new ArrayList(jsonInMap.keySet()); for (String key : keys) { System.out.println(key + “: ” + jsonInMap.get(key)); } } catch (JsonGenerationException e) { e.printStackTrace(); } catch (JsonMappingException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } } } Dependencies net.sf.json-lib json-lib 2.4 jdk15 commons-io commons-io 2.3 compile xom xom 1.2.5 org.codehaus.jackson jackson-mapper-asl 1.9.0 The Input XML Jags Inc Jagan Male 24-jul Satya Male 24-apr The output 7 Feb, 2013 7:20:50 PM net.sf.json.xml.XMLSerializer getType INFO: Using default type string { “name”: “Jags Inc”, “employees”: [ { "name": "Jagan", "sex": "Male", "dob": "24-jul" }, { "name": "Satya", "sex": "Male", "dob": "24-apr" } ] } name: Jags Inc employees: [{name=Jagan, sex=Male, dob=24-jul}, {name=Satya, sex=Male, dob=24-apr}]

once upon a time there was a munich pizza baker who developed a technique to beam pizza out of bright sunshine. he can produce more than a thousand pizzas per second and needs a channel to sell this amount of pizza and decides to build an online shop. mario’s initial idea is to sell pizzas, but now he is thinking about introduction of new product lines like beverages, salads and pasta. before we take a look to the validation of mario´s idea, lets take a short look at the existing online shop. mario’s online shop is based on mongodb , apache wicket and spring . mongodb is a document-oriented nosql-database . mongodb stores records not in tables as a relational database but in bson documents, which is a binary version of json (java script object notation) and very similar to the object structure in mario’s application. the usage of mongodb makes his development easier and deployment faster. the figure shows a json document which is very similar to a java object: a json document property with the according value corresponds to the java object property with the appropriate value. you can add or remove properties in your java object and this will automatically change your database schema. so there is no need to put your java object model into a relational schema via hibernate. mario also decided to build his online shop only with open-source technologies like apache wicket and spring. wicket is a very common lightweight component-based web application framework and it is closely patterned after stateful gui frameworks such as javafx . the spring framework is an open source application framework and inversion of control container for the java platform and does not impose any specific programming model. spring has become popular in the java community as an alternative to, replacement for, or even addition to the enterprise javabean (ejb) model. because of this architecture mario is able to deploy its application in a lightweight application server like tomcat or jetty . this figure shows the system landscape of mario. mario has two major system on the lefthand site there is his online shop and on the righthand site there is ‘pas’ a famous billing system. in the middle is hadoop that connects both systems together. in the business world an application normally does not stand alone. in most cases an application must communicate with others. the lean architecture of marios online shop enables him to connect the billing system ‘pas’ to his online shop. spring for apache hadoop provides this integration between the two systems online shop and ‘pas’. hadoop supports data-intensive distributed applications and implements a computational paradigm named mapreduce, where the computation is divided into many small fragments, each of them may be executed or re-executed on any node in the cluster of commodity hardware. mario uses hadoop as an etl layer that enables him to transfer gigabytes of order information into the billing system. in this case hadoop makes it possible for a financial controller to verify if all orders were billed correctly. in addition to the online shop feature mario has a real-time sales dashboard that enables him to track his sales in real time. the dashboard displays daily and monthly sales statistics for each pizza and contains a map with the geographical overview of customer activity and competitor locations. here is a walkthrough of the shop : now lets talk about mario’s incredible new idea : mario wants to sell even more pizza! and other products as well. mario decides to use lean startup methods in order to test the possible introduction of new product lines and plans an experiment to validate his new idea using a scientific approach and pure facts instead of hunches. mario´s core assumption is that customers wants to buy other products than pizza – drinks, salads and pasta. furthermore he is worried about pricing. mario contacts all customers to complete a survey and provides an incentive for the participation, a free pizza to every customer who responds to the survey. the result of the survey validated mario’s assumption – customers want to buy beverages, salads and pasta. but he also found out that his customers are willing to pay higher prices for high-quality products and that they simply love his easy shopping flow. currently a pizza order can be completed with three clicks only, so there is new riskiest assumption to validate: will a more complex shopping flow affect his sales? the figures shows a validation board. a validation board is a deceptively simple tool for testing out product ideas. furthermore a validation board tracks pivots which follows from customer feedback. mario decides to introduce beverages, salads and pasta product lines and thinks about a possibility, how he can handle the extension of the product line without destroying the easy shopping flow. that’s why mario thinks a recommendation engine is the right way for him. panels for recommendations can be integrated in the online shop without changing the shopping flow. mario hired a statistician to help him implement a recommender system for his online shop for better cross-selling. he also defined new measurement points to validate his new idea . therefore he tracks the conversion rate of orders as well as cross-selling rates and every event in the online shop is already tracked in realtime. so mario can very easily perform further experiments in order to verify more assumptions. follow the blog to see how the story continues or come to mongodb usergroup meetup in munich , february 20, 2013 or mongodb days in berlin , february 26, 2013 to get a live presentation. our talk sheds light on how to build an online recommendation engine based on mongodb and apache mahout. we’ll show which recommenders must be built to reach mario’s goal and how these can be integrated in mario’s shop infrastructure.

today, i thought a lot about how to examine different databases. choosing a database is often a daunting task. there's a lot of confusion, a 'theorem', and more than all, the immortal proverb 'not one size fits all'. as if it helps. one of the first things that you realize, when examining nosql distributed databases (and how could you not)is that these days databases are like cars: they're all good. old fashioned sql databases can scale in and out, horizontally sharded over several machines to achieve high availability. nosql systems claim to be consistent. what difference then does it make what database would you choose? the availability and consistency that i mentioned comes, of course, from the misunderstood cap theorem , that - so people say - states that you can only choose 2 out of the 3 consistency: every read would get you the most recent write availability: every node (if not failed) always executes queries partition-tolerance: even if the connections between nodes are down, the other two (a & c) promises, are kept. usually its depicted in a nicely equilaterl triangle, as this one from ofirm : there's a nice proof and explanation of it in this 4 minute video here . but if we think about it, and also see some of brewer's (the theorem author) later remarks , we'll see that the 2 out of 3 is really 1 out of 2: it's really just a vs c! and this is simply because: availability is achieved by replicating the data across different machines consistency is achieved by updating several nodes before allowing further reads total partitioning, meaning failure of part of the system is rare. however, we could look at a delay, a latency, of the update between nodes, as a temporary partitioning . it will then cause a temporary decision between a and c: on systems that allow reads before updating all the nodes, we will get high availability on systems that lock all the nodes before allowing reads, we will get consistency that's it! and since this decision is temporary, it exists only for the duration of the delay, some may say that we are really contrasting latency (another word for availability) against consistency. by the way, there's no distributed system that wants to live with "paritioning" - if it does, it's not distributed. that is why putting sql in this triangle may lead to confusion.

Hi, I have recently created www.php.is-best.net out of an actual business necessity: The need to quickly group lines of code into php functions. What www.php.is-best.net allows you to do is to quickly extract methods out of your code. Just Grab and paste any php script inside the designated area and WHALLA! you're done. A new method will be generated out of your code.

imagine you have a csv file that you want to manipulate. here’s a sample file we can play with: lopez,charlie,2002,11,21 parker,ward,1995,04,08 henderson,russell,2007,10,01 our goal is to transform this into the following form by combining the last three columns: lopez,charlie,20021121 parker,ward,19950408 henderson,russell,20071001 in linux this would take all of two seconds (excuse the awkward awk command): shell$ awk -f"," '{ print $1","$2","$3$4$5 }' people.txt what if you wanted to quickly do the same in hdfs - and let’s assume you want to write the results back to hdfs. one approach would be to use the hdfs cli to stream the inputs into awk, and stream the awk output back into hdfs. you could do this with the hdfs cat and put - options (note that adding a hyphen after put instructs the put command to stream data from standard input to hdfs): shell$ hadoop fs -cat people.txt | awk -f"," '{ print $1","$2","$3$4$5 }' | hadoop fs -put - people-coalesed.txt btw, if your input and output files are lzop-compressed then this command would work: shell$ hadoop fs -cat people.txt.lzo | lzop -dc | awk -f"," '{ print $1","$2","$3$4$5 }' | \ lzop -c | hadoop fs -put - people-coalesed.txt.lzo this is great if your file isn’t too large, but if it’s multiple gigabytes in length then you probably want to harness the power of mapreduce to get this done in a jiffy! the words “in a jiffy” and “mapreduce” aren’t commonly used together, so what do we do? well you could crack open pig or hive and write some custom user-defined functions, but this means you end up in java which we want to avoid. hadoop streaming comes to the rescue in these situations. let’s first create our awk script which will be executed: shell$ cat people.awk #!/bin/awk -f begin { fs = "," } { print $1","$2","$3$4$5 } in linux, if you make this awk script executable, you could execute is as follows: shell$ ./people.awk people.txt in mapreduce-land we don’t need to join data in this particular example, so we don’t need to run any reducers. call your awk script from mappers via hadoop streaming with this command: shell$ hadoop_home=/usr/lib/hadoop shell$ ${hadoop_home}/bin/hadoop \ jar ${hadoop_home}/contrib/streaming/*.jar \ -d mapreduce.job.reduces=0 \ -d mapred.reduce.tasks=0 \ -input people.txt \ -output people-coalesed \ -mapper people.awk \ -file people.awk a few options in the hadoop streaming command are worth examining: finally - to get lzo into the picture you need to add -inputformat , -d mapred.output.compress and -d mapred.output.compression.codec arguments: shell$ hadoop_home=/usr/lib/hadoop shell$ ${hadoop_home}/bin/hadoop \ jar ${hadoop_home}/contrib/streaming/*.jar \ -d mapreduce.job.reduces=0 \ -d mapred.reduce.tasks=0 \ -d mapred.output.compress=true \ -d stream.map.input.ignorekey=true \ -d mapred.output.compression.codec=com.hadoop.compression.lzo.lzopcodec \ -inputformat com.hadoop.mapred.deprecatedlzotextinputformat \ -input people.txt.lzo \ -output people-coalesed \ -mapper people.awk \ -file people.awk

ActiveMq is a powerful open source messaging broker, and is very easy and straightforward to use with Spring as the below classes and XML will prove. The example below is the bar minimum needed to get up and running with transactions and message converters. On the sending side, the ActiveMq connection factory needs to be created with the url of the broker. This in turn is used to create the Spring JMS connection factory and as no session cache property is supplied the default cache is one. The template is then used in turn to create the Message Sender class: An example sending class is below. It uses the convertandSend method of the JmsTemplate class. As there is no destination arg, the message will be sent to the default destination which was set up in the XML file: import java.util.Map; import org.springframework.jms.core.JmsTemplate; public class MessageSender { private final JmsTemplate jmsTemplate; public MessageSender(final JmsTemplate jmsTemplate) { this.jmsTemplate = jmsTemplate; } public void send(final Map map) { jmsTemplate.convertAndSend(map); } } On the receiving side, there needs to be a listener container. The simplest example of this is the SimpleMessageListenerContainer. This requires a connection factory, a destination (or destination name) and a message listener. An example of the Spring configuration for the receiving messages is below: The listening/receiving class needs to extend javax.jms.MessageListener and implement the onMessage method: import javax.jms.MapMessage; import javax.jms.Message; import javax.jms.MessageListener; public class MessageReceiver implements MessageListener { public void onMessage(final Message message) { if (message instanceof MapMessage) { final MapMessage mapMessage = (MapMessage) message; // do something } } } To then send a message would be as simple as getting the sending bean from the bean factory as shown in the below code: MessageSender sender = (MessageSender) factory.getBean("messageSender"); Map map = new HashMap(); map.put("Name", "MYNAME"); sender.send(map); Will try to expand and build up JMS and Spring articles with examples of using transactions and other brokers like MQSeries.

Resin has supported caching, session replication (another form of caching), and http proxy caching in cluster environments for over ten years. When you use Resin caching, you are using the same platform that has the speed and scalability of custom services written in C like NginX with the usability of Java, and the industry platform Java EE. JCache JSR 107 is a distributed cache that has a similar interface to the HashMap that you know and love. To be more specific, the Cache object in JCache looks like a java.util.ConncurrentHashMap. In addition, JCache JSR 107 defines integration with CDI (as well as Spring and Guice). You can decorate services with interceptors that apply caching to the services just by defining annotations. Resin 4 has support for JCache, and JCache support is required for Java EE 7. Let's look at a small example to see how easy is to get started with JCache. package hello.world; import javax.cache.Cache; import javax.cache.CacheBuilder; import javax.cache.CacheManager; import javax.cache.Caching; ... @WebServlet("/HelloServlet") public class HelloServlet extends HttpServlet { Cache cache; public Cache cache() { if (cache == null) { //building a cache CacheManager manager = Caching.getCacheManager("cacheManagerHello"); CacheBuilder builder = manager.createCacheBuilder("a"); cache = builder.build(); } return cache; } protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setContentType("text/html"); response.getWriter().append(" "); String helloMessage = cache().get("hello message"); if (helloMessage == null) { helloMessage = new StringBuilder(20) .append("Hello World ! ") .append(System.currentTimeMillis()).toString(); cache().put("hello message", helloMessage); // <-------------- putting results in the cache } response.getWriter().append(helloMessage); response.getWriter().append(" "); } } The above works out fairly well, but what if we want to periodically change the helloMessage. Let's say we get 2,000 requests a second, but every 10 seconds or so we would like to regenerate the helloMessage. The message might be: Hello World ! 1358979745996 Later we would want it to change. If we wanted it to change every 10 seconds after it was last accessed, we would do this: cache = builder.setExpiry(ExpiryType.ACCESSED, new Duration(TimeUnit.SECONDS, 10)).build(); For this example, we want to change it every 10 seconds after is was last modified. We would set up the timeout on the creation as follows: cache = builder.setExpiry(ExpiryType.MODIFIED, new Duration(TimeUnit.SECONDS, 10)).build(); This would go right in the cache method we defined earlier. public Cache cache() { if (cache == null) { CacheManager manager = Caching.getCacheManager("cacheManagerHello"); CacheBuilder builder = manager.createCacheBuilder("b"); cache = builder.setExpiry(ExpiryType.MODIFIED, new Duration(TimeUnit.SECONDS, 10)).build(); } return cache; } Resin's JCache implementation is built on top Resin distributed cache architecture. You get replication, and data redundancy built in. Bill Digman is a Java EE / Servlet enthusiast and Open Source enthusiast who loves working with Caucho's Resin Servlet Container, a Java EE Web Profile Servlet Container. Caucho's Resin OpenSource Servlet Container Java EE Web Profile Servlet Container Caucho's Resin 4.0 JCache blog post