1. Overview This is the second of a series of posts about setting up a RESTful web service using Spring 3.1 with Java based configuration. The first post of the series focused on bootstrapping the web application; this post will focus on setting up REST in Spring, the Controller and HTTP response codes, configuration of payload marshalling and content negotiation. 2. Understanding REST in Spring The Spring framework supports 2 ways of creating RESTful services: using MVC with ModelAndView using HTTP message converters The ModelAndView approach is older and much better documented, but also more verbose and configuration heavy. It tries to shoehorn the REST paradigm into the old model, which is not without problems. The Spring team understood this and provided first-class REST support starting with Spring 3.0. The new approach, based on HttpMessageConverter and annotations, is much more lightweight and easy to implement. Configuration is minimal and it provides sensible defaults for what you would expect from a RESTful service. It is however newer and a a bit on the light side concerning documentation; what’s more, the Spring reference doesn’t go out of it’s way to make the distinction and the tradeoffs between the two approaches as clear as they should be. Nevertheless, this is the way RESTful services should be build after Spring 3.0. 3. The Java configuration @Configuration @EnableWebMvc public class WebConfig{ // } The new @EnableWebMvc annotation does a number of useful things – specifically, in the case of REST, it detect the existence of Jackson and JAXB 2 on the classpath and automatically creates and registers default JSON and XML converters. The functionality of the annotation is equivalent to the XML version: This is a shortcut, and though it may be useful in many situations, it’s not perfect. When more complex configuration is needed, remove the annotation and extend WebMvcConfigurationSupport directly. 4. Testing the Spring context Starting with Spring 3.1, we get first-class testing support for @Configuration classes: @RunWith( SpringJUnit4ClassRunner.class ) @ContextConfiguration( classes = { ApplicationConfig.class, PersistenceConfig.class },loader = AnnotationConfigContextLoader.class ) public class SpringTest{ @Test public void whenSpringContextIsInstantiated_thenNoExceptions(){ // When } } The Java configuration classes are simply specified with the @ContextConfiguration annotation and the new AnnotationConfigContextLoader loads the bean definitions from the @Configuration classes. Notice that the WebConfig configuration class was not included in the test because it needs to run in a servlet context, which is not provided. 5. The Controller The @Controller is the central artifact in the entire Web Tier of the RESTful API. For the purpose of this post, the controller is modeling a simple REST resource – Foo: @Controller class FooController{ @Autowired IFooService service; @RequestMapping( value = "foo",method = RequestMethod.GET ) @ResponseBody public List< Foo > getAll(){ return this.service.getAll(); } @RequestMapping( value = "foo/{id}",method = RequestMethod.GET ) @ResponseBody public Foo get( @PathVariable( "id" ) Long id ){ return RestPreconditions.checkNotNull( this.service.getById( id ) ); } @RequestMapping( value = "foo",method = RequestMethod.POST ) @ResponseStatus( HttpStatus.CREATED ) @ResponseBody public Long create( @RequestBody Foo entity ){ RestPreconditions.checkNotNullFromRequest( entity ); return this.service.create( entity ); } @RequestMapping( value = "foo",method = RequestMethod.PUT ) @ResponseStatus( HttpStatus.OK ) public void update( @RequestBody Foo entity ){ RestPreconditions.checkNotNullFromRequest( entity ); RestPreconditions.checkNotNull( this.service.getById( entity.getId() ) ); this.service.update( entity ); } @RequestMapping( value = "foo/{id}",method = RequestMethod.DELETE ) @ResponseStatus( HttpStatus.OK ) public void delete( @PathVariable( "id" ) Long id ){ this.service.deleteById( id ); } } The Controller implementation is non-public – this is because there is no need for it to be. Usually the controller is the last in the chain of dependencies – it receives HTTP requests from the Spring front controller (the DispathcerServlet) and simply delegate them forward to a service layer. If there is no use case where the controller has to be injected or manipulated through a direct reference, then I prefer not to declare it as public. The request mappings are straightforward – as with any Spring controller, the actual value of the mapping as well as the HTTP method are used to determine the target method for the request. @RequestBody will bind the parameters of the method to the body of the HTTP request, whereas @ResponseBody does the same for the response and return type. They also ensure that the resource will be marshalled and unmarshalled using the correct HTTP converter. Content negotiation will take place to choose which one of the active converters will be used, based mostly on the Accept header, although other HTTP headers may be used to determine the representation as well. 6. Mapping the HTTP response codes The status codes of the HTTP response are one of the most important parts of the REST service, and the subject can quickly become very complex. Getting these right can be what makes or breaks the service. 6.1. Unmapped requests If Spring MVC receives a request which doesn’t have a mapping, it considers the request not to be allowed and returns a 405 METHOD NOT ALLOWED back to the client. It is also good practice to include the Allow HTTP header when returning a 405 to the client, in order to specify which operations are allowed. This is the standard behavior of Spring MVC and does not require any additional configuration. 6.2. Valid, mapped requests For any request that does have a mapping, Spring MVC considers the request valid and responds with 200 OK if no other status code is specified otherwise. It is because of this that controller declares different @ResponseStatus for the create, update and delete actions but not for get, which should indeed return the default 200 OK. 6.3. Client error In case of a client error, custom exceptions are defined and mapped to the appropriate error codes. Simply throwing these exceptions from any of the layers of the web tier will ensure Spring maps the corresponding status code on the HTTP response. @ResponseStatus( value = HttpStatus.BAD_REQUEST ) public class BadRequestException extends RuntimeException{ // } @ResponseStatus( value = HttpStatus.NOT_FOUND ) public class ResourceNotFoundException extends RuntimeException{ // } These exceptions are part of the REST API and, as such, should only be used in the appropriate layers corresponding to REST; if for instance a DAO/DAL layer exist, it should not use the exceptions directly. Note also that these are not checked exceptions but runtime exceptions – in line with Spring practices and idioms. 6.4. Using @ExceptionHandler Another option to map custom exceptions on specific status codes is to use the @ExceptionHandler annotation in the controller. The problem with that approach is that the annotation only applies to the controller in which it is defined, not to the entire Spring Container, which means that it needs to be declared in each controller individually. This quickly becomes cumbersome, especially in more complex applications which many controllers. There are a few JIRA issues opened with Spring at this time to handle this and other related limitations: SPR-8124, SPR-7278, SPR-8406. 7. Additional Maven dependencies In addition to the pom.xml from the first post, two dependencies need to be added: org.codehaus.jackson jackson-mapper-asl ${jackson-mapper-asl.version} runtime javax.xml.bind jaxb-api ${jaxb-api.version} runtime 1.9.12 2.2.4 These are the libraries used to convert the representation of the REST resource to either JSON or XML. 8. Conclusion This post covered the configuration and implementation of a RESTful service using Spring 3.1 and Java based configuration, discussing HTTP response codes, basic content negotiation and marshaling. In the next articles of the series I will focus on discoverability of the API, advanced content negotiation and working with additional representations of a resource. In the meantime, check out the github project.

This is part one in a series. Part 2 can be found here. This tutorial will show you how to create a simple data structure for shapes on an HTML5 canvas and how to have them be selectable. The finished canvas will look like this: This text is displayed if your browser does not support HTML5 Canvas. You'll be able to click to drag boxes and double click to add new boxes. This article’s code is written primarily to be easy to understand. It isn’t optimized for performance, though a little bit of the drawing is set up so that more complex shapes can easily be added in the future. We’ll be going over a few things that are also essential to game-development (drawing loop, hit testing), and in later tutorials I will probably turn this example into a small game. The HTML5 Canvas A Canvas is made by using the tag in HTML: This text is displayed if your browser does not support HTML5 Canvas. A canvas isn’t smart: it’s just a place for drawing pixels. If you ask it to draw something it will do so and then immediately forget everything about what you have just done. Because of this we have to keep track ourselves of all the things we want to draw (and re-draw) each frame. So we’ll need to add: Code for keeping track of objects Code for initialization Code for drawing the objects as they are made and move around Code for mouse events Keeping track of what we draw To keep things simple for this example we will just make a rectangular object called Box. We’ll also make a method for creating Boxes a little easier. // holds all our rectangles var boxes = []; //Box object to hold data for all drawn rects function Box() { this.x = 0; this.y = 0; this.w = 1; // default width and height? this.h = 1; this.fill = '#444444'; } //Initialize a new Box, add it, and invalidate the canvas function addRect(x, y, w, h, fill) { var rect = new Box; rect.x = x; rect.y = y; rect.w = w rect.h = h; rect.fill = fill; boxes.push(rect); invalidate(); } Initialization I’m going to add a bunch of variables for keeping track of the drawing and mouse state. I already added boxes[] to keep track of each object, but we’ll also need a var for the canvas, the canvas’ 2d context (where wall drawing is done), whether the mouse is dragging, width/height of the canvas, and so on. We’ll also want to make a second canvas, for selection purposes, but I’ll talk about that later. var canvas; var ctx; var WIDTH; var HEIGHT; var INTERVAL = 20; // how often, in milliseconds, we check to see if a redraw is needed var isDrag = false; var mx, my; // mouse coordinates // when set to true, the canvas will redraw everything // invalidate() just sets this to false right now // we want to call invalidate() whenever we make a change var canvasValid = false; // The node (if any) being selected. // If in the future we want to select multiple objects, this will get turned into an array var mySel; // The selection color and width. Right now we have a red selection with a small width var mySelColor = '#CC0000'; var mySelWidth = 2; // we use a fake canvas to draw individual shapes for selection testing var ghostcanvas; var gctx; // fake canvas context // since we can drag from anywhere in a node // instead of just its x/y corner, we need to save // the offset of the mouse when we start dragging. var offsetx, offsety; // Padding and border style widths for mouse offsets var stylePaddingLeft, stylePaddingTop, styleBorderLeft, styleBorderTop; // initialize our canvas, add a ghost canvas, set draw loop // then add everything we want to intially exist on the canvas function init() { canvas = document.getElementById('canvas'); HEIGHT = canvas.height; WIDTH = canvas.width; ctx = canvas.getContext('2d'); ghostcanvas = document.createElement('canvas'); ghostcanvas.height = HEIGHT; ghostcanvas.width = WIDTH; gctx = ghostcanvas.getContext('2d'); //fixes a problem where double clicking causes text to get selected on the canvas canvas.onselectstart = function () { return false; } // fixes mouse co-ordinate problems when there's a border or padding // see getMouse for more detail if (document.defaultView && document.defaultView.getComputedStyle) { stylePaddingLeft = parseInt(document.defaultView.getComputedStyle(canvas, null)['paddingLeft'], 10) || 0; stylePaddingTop = parseInt(document.defaultView.getComputedStyle(canvas, null)['paddingTop'], 10) || 0; styleBorderLeft = parseInt(document.defaultView.getComputedStyle(canvas, null)['borderLeftWidth'], 10) || 0; styleBorderTop = parseInt(document.defaultView.getComputedStyle(canvas, null)['borderTopWidth'], 10) || 0; } // make draw() fire every INTERVAL milliseconds. setInterval(draw, INTERVAL); // add our events. Up and down are for dragging, // double click is for making new boxes canvas.onmousedown = myDown; canvas.onmouseup = myUp; canvas.ondblclick = myDblClick; // add custom initialization here: // add an orange rectangle addRect(200, 200, 40, 40, '#FFC02B'); // add a smaller blue rectangle addRect(25, 90, 25, 25, '#2BB8FF'); } Drawing Since our canvas is animated (boxes move over time), we have to set up a draw loop as I did in the init() function. We have to draw at a frame rate, maybe every 20 milliseconds or so. However, redrawing doesn’t just mean drawing the shapes over and over; we also have to clear the canvas before every redraw. If we don’t clear it, dragging will look like the box is making a solid line because none of the old box-positions will go away. Because of this, we clear the entire canvas before each Draw frame. This can get expensive, and we only want to draw if something has actually changed within our framework, so we will consider the canvas to be either valid or invalid. If everything just got drawn, the canvas is valid and there’s no need to draw again. However, if we do something like add a new Box or try to move a box by dragging it, the canvas will get invalidated and draw() will do a clear-redraw-validate. This isn’t the only way to optimize drawing, after all clearing and redrawing the entire canvas when one little box moves is excessive, but canvas invalidation is the only optimization we’re going to use for now. // While draw is called as often as the INTERVAL variable demands, // It only ever does something if the canvas gets invalidated by our code function draw() { if (canvasValid == false) { clear(ctx); // Add stuff you want drawn in the background all the time here // draw all boxes var l = boxes.length; for (var i = 0; i < l; i++) { drawshape(ctx, boxes[i], boxes[i].fill); } // draw selection // right now this is just a stroke along the edge of the selected box if (mySel != null) { ctx.strokeStyle = mySelColor; ctx.lineWidth = mySelWidth; ctx.strokeRect(mySel.x,mySel.y,mySel.w,mySel.h); } // Add stuff you want drawn on top all the time here canvasValid = true; } } As you can see, we go through all of boxes[] and draw each one, in order from first to last. This will give the nice appearance of later boxes looking as if they are on top of earlier boxes. After all the boxes are drawn, a selection handle (if there’s a selection) gets drawn around the box that mySel references. If you wanted a background (like a city) or a foreground (like clouds), one way to add them is to put them before or after the main two drawing bits. There are better ways though, like using multiple canvases, but we won’t go over that here. Mouse events Now we have objects, initialization, and a loop that will constantly re-draw when needed. All thats left is to make the mouse do things upon pressing, releasing, and double clicking. With our MouseDown event we need to see if there are any objects we could have clicked on. And we don’t want just any object; selections make the most sense when we only grab the top-most object. Now we could do something very easy and just check the bounds of each of our boxes – see if the mouse co-ordinates lie within the boxes width and height range – but that isn’t as extendable as I’d like. After all, What if later we want to select lines instead of boxes? Or select triangles? Or select text? So we’re going to do selection in a more general way: We will draw each shape, one at a time, onto a “ghost” canvas, and see if the mouse co-ordinates lie on a drawn pixel or not. A ghost canvas (or fake canvas, or temporary canvas) is a second canvas that we created in the same size and shape as our normal one. Only nothing from it will ever get seen, because we only created it in code and never added it to the page. Go back and look at ghostcanvas and its context (gctx) in the init() function to see how it was made. // Happens when the mouse is clicked in the canvas function myDown(e){ getMouse(e); clear(gctx); // clear the ghost canvas from its last use // run through all the boxes var l = boxes.length; for (var i = l-1; i >= 0; i--) { // draw shape onto ghost context drawshape(gctx, boxes[i], 'black'); // get image data at the mouse x,y pixel var imageData = gctx.getImageData(mx, my, 1, 1); var index = (mx + my * imageData.width) * 4; // if the mouse pixel exists, select and break if (imageData.data[3] > 0) { mySel = boxes[i]; offsetx = mx - mySel.x; offsety = my - mySel.y; mySel.x = mx - offsetx; mySel.y = my - offsety; isDrag = true; canvas.onmousemove = myMove; invalidate(); clear(gctx); return; } } // havent returned means we have selected nothing mySel = null; // clear the ghost canvas for next time clear(gctx); // invalidate because we might need the selection border to disappear invalidate(); } myMove and myUp are pretty self explanatory. the var isDrag becomes true if myDown found something to select, and it becomes false again when the mouse is released (myUp). // Happens when the mouse is moving inside the canvas function myMove(e){ if (isDrag){ getMouse(e); mySel.x = mx - offsetx; mySel.y = my - offsety; // something is changing position so we better invalidate the canvas! invalidate(); } } function myUp(){ isDrag = false; canvas.onmousemove = null; } There are a few little methods I added that are not shown, such as one to correctly get the mouse position in a canvas. You can see and download the full demo source here. Now that we have a basic structure down, it is easy to write code that handles more complex shapes, like paths or images or video. Rotation and scaling these things takes a bit more work, but is quite doable with the Canvas and our selection method is already set up to deal with them. If you would like to see this code enhanced in future posts (or have any fixes), let me know how in the comments. Part 2 of this tutorial is about resizing the shapes and can be found here. Source: http://simonsarris.com/blog/140-canvas-moving-selectable-shapes

In a classical model of recommendation system, there are "users" and "items". User has associated metadata (or content) such as age, gender, race and other demographic information. Items also has its metadata such as text description, price, weight ... etc. On top of that, there are interaction (or transaction) between user and items, such as userA download/purchase movieB, userX give a rating 5 to productY ... etc. Now given all the metadata of user and item, as well as their interaction over time, can we answer the following questions ... What is the probability that userX purchase itemY ? What rating will userX give to itemY ? What is the top k unseen items that should be recommended to userX ? Content-based Approach In this approach, we make use of the metadata to categorize user and item and then match them at the category level. One example is to recommend jobs to candidates, we can do a IR/text search to match the user's resume with the job descriptions. Another example is to recommend an item that is "similar" to the one that the user has purchased. Similarity is measured according to the item's metadata and various distance function can be used. The goal is to find k nearest neighbors of the item we know the user likes. Collaborative Filtering Approach In this approach, we look purely at the interactions between user and item, and use that to perform our recommendation. The interaction data can be represented as a matrix. Notice that each cell represents the interaction between user and item. For example, the cell can contain the rating that user gives to the item (in the case the cell is a numeric value), or the cell can be just a binary value indicating whether the interaction between user and item has happened. (e.g. a "1" if userX has purchased itemY, and "0" otherwise. The matrix is also extremely sparse, meaning that most of the cells are unfilled. We need to be careful about how we treat these unfilled cells, there are 2 common ways ... Treat these unknown cells as "0". Make them equivalent to user giving a rate "0". This may or may not be a good idea depends on your application scenarios. Guess what the missing value should be. For example, to guess what userX will rate itemA given we know his has rate on itemB, we can look at all users (or those who is in the same age group of userX) who has rate both itemA and itemB, then compute an average rating from them. Use the average rating of itemA and itemB to interpolate userX's rating on itemA given his rating on itemB. User-based Collaboration Filter In this model, we do the following Find a group of users that is “similar” to user X Find all movies liked by this group that hasn’t been seen by user X Rank these movies and recommend to user X This introduces the concept of user-to-user similarity, which is basically the similarity between 2 row vectors of the user/item matrix. To compute the K nearest neighbor of a particular users. A naive implementation is to compute the "similarity" for all other users and pick the top K. Different similarity functions can be used. Jaccard distance function is defined as the number of intersections of movies that both users has seen divided by the number of union of movies they both seen. Pearson similarity is first normalizing the user's rating and then compute the cosine distance. There are two problems with this approach Compare userX and userY is expensive as they have millions of attributes Find top k similar users to userX require computing all pairs of userX and userY Location Sensitive Hashing and Minhash To resolve problem 1, we approximate the similarity using a cheap estimation function, called minhash. The idea is to find a hash function h() such that the probability of h(userX) = h(userY) is proportion to the similarity of userX and userY. And if we can find 100 of h() function, we can just count the number of such function where h(userX) = h(userY) to determine how similar userX is to userY. The idea is depicted as follows ... It will be expensive to permute the rows if the number of rows is large. Remember that the purpose of h(c1) is to return row number of the first row that is 1. So we can scan each row of c1 to see if it is 1, if so we apply a function newRowNum = hash(rowNum) to simulate a permutation. Take the minimum of the newRowNum seen so far. As an optimization, instead of doing one column at a time, we can do it a row at the time, the algorithm is as follows To solve problem 2, we need to avoid computing all other users' similarity to userX. The idea is to hash users into buckets such that similar users will be fall into the same bucket. Therefore, instead of computing all users, we only compute the similarity of those users who is in the same bucket of userX. The idea is to horizontally partition the column into b bands, each with r rows. By pick the parameter b and r, we can control the likelihood (function of similarity) that they will fall into the same bucket in at least one band. Item-based Collaboration Filter If we transpose the user/item matrix and do the same thing, we can compute the item to item similarity. In this model, we do the following ... Find the set of movies that user X likes (from interaction data) Find a group of movies that is similar to these set of movies that we know user X likes Rank these movies and recommend to user X It turns out that computing item-based collaboration filter has more benefit than computing user to user similarity for the following reasons ... Number of items typically smaller than number of users While user's taste will change over time and hence the similarity matrix need to be updated more frequent, item to item similarity tends to be more stable and requires less update. Singular Value Decomposition If we look back at the matrix, we can see the matrix multiplication is equivalent to mapping an item from the item space to the user space. In other words, if we view each of the existing item as an axis in the user space (notice, each user is a vector of their rating on existing items), then multiplying a new item with the matrix gives the same vector like the user. So we can then compute a dot product with this projected new item with user to determine its similarity. It turns out that this is equivalent to map the user to the item space and compute a dot product there. In other words, multiply the matrix is equivalent to mapping between item space and user space. Now lets imagine there is a hidden concept space in between. Instead of jumping directly from user space to item space, we can think of jumping from user space to a concept space, and then to the item space. Notice that here we first map the user space to the concept space and also map the item space to the concept space. Then we match both user and item at the concept space. This is a generalization of our recommender. We can use SVD to factor the matrix into 2 parts. Let P be the m by n matrix (m rows and n columns). P = UDV where U is an m by m matrix, each column represents the eigenvectors of P*transpose(P). And V is an n by n matrix with each row represents the eigenvector of transpose(P)*P. D is a diagonal matrix containing eigenvalues of P*transpose(P), or transpose(P)*P. In other words, we can decompose P into U*squareroot(D) and squareroot(D)*V. Notice that D can be thought as the strength of each "concept" in the concept space. And the value is order in terms of their magnitude in decreasing order. If we remove some of the weakest concept by making them zero, we reduce the number of non-zero elements in D, which effective generalize the concept space (make them focus in the important concepts). Calculate SVD decomposition for matrix with large dimensions is expensive. Fortunately, if our goal is to compute an SVD approximation (with k diagonal non-zero value), we can use the random projection mechanism as describer here. Association Rule Based In this model, we use the market/basket association rule algorithm to discover rule like ... {item1, item2} => {item3, item4, item5} We represent each user as a basket and each viewing as an item (notice that we ignore the rating and use a binary value). After that we use association rule mining algorithm to detect frequent item set and the association rules. Then for each user, we match the user's previous viewing items to the set of rules to determine what other movies should we recommend. Evaluate the recommender After we have a recommender, how do we evaluate the performance of it ? The basic idea is to use separate the data into the training set and the test set. For the test set, we remove certain user-to-movies interaction (change certain cells from 1 to 0) and pretending the user hasn't seen the item. Then we use the training set to train a recommender and then fit the test set (with removed interaction) to the recommender. The performance is measured by how much overlap between the recommended items with the one that we have removed. In other words, a good recommender should be able to recover the set of items that we have removed from the test set. Leverage tagging information on items In some cases, items has explicit tags associated with them (we can considered the tags is a user-annotated concept space added to the items). Consider each item is described with a vector of tags. Now user can also be auto-tagged based on the items they have interacted. For example, if userX purchase itemY which is tagged with Z1, and Z2. Then user will increase her tag Z1 and Z2 in her existing tag vector. We can use a time decay mechanism to update the user's tag vector as follows ... current_user_tag = alpha * item_tag + (1 - alpha) * prev_user_tag To recommend an item to the user, we simply need to calculate the top k items by computing the dot product (ie: cosine distance) of the user tag vector and the item tag vector. Source: http://horicky.blogspot.com/2011/09/recommendation-engine.html

Perl was easy to use, wildly popular, and lots of fun. The Camel Book introduced many coders to a powerful new language (and the whimsically-covered O'Reilly series), and offered access to web programming via CGI. Plenty of people still develop in Perl ('the duct tape of the Internet'), although lately some criticism of Perl programmers has surfaced. No doubt about one thing, though: CGI is just too old. Sensing a need, Sebastian Ridel created Mojolicious to fill CGI's place, satisfying Perl programmers' desire for a more modern web framework Yesterday Sebastian showed off some of Mojolicious' simplicity and power: By now you've probably heard about WebSockets, and that they are the future of web development, but so far there are very little examples that really show how easy to use they actually are. So today we are going to explore the wonderful world of events in Mojolicious a bit and build a little application that forwards all framework log messages to a browser window. The script is short and sweet and, if you still love Perl, will warm your HTML5 heart. Check it out here.

i was buying a used book through amazon this evening. three resellers offered the book at essentially the same price. here were their ratings: 94% positive out of 85,193 reviews 98% positive out of 20,785 reviews 99% positive out of 840 reviews which reseller is likely to give the best service? before you assume it’s the seller with the highest percentage of positive reviews, consider the following simpler scenario. suppose one reseller has 90 positive reviews out of 100. the other reseller has two reviews, both positive. you could say one has 90% approval and the other has 100% approval, so the one with 100% approval is better. but this doesn’t take into consideration that there’s much more data on one than the other. you can have some confidence that 90% of the first reseller’s customers are satisfied. you don’t really know about the other because you have only two data points. xkcd.com a bayesian view of the problem naturally incorporates the amount of data as well as its average. let θ a be the probability of a customer being satisfied with company a’s service. let θ b be the corresponding probability for company b. suppose before we see any reviews we think all ratings are equally likely. that is, we start with a uniform prior distribution θ a and θ b . a uniform distribution is the same as a beta(1, 1) distribution. after observing 90 positive reviews and 10 negative reviews, our posterior estimate on θ a has a beta(91, 11) distribution. after observing 2 positive reviews, our posterior estimate on θ b has a beta(3, 1) distribution. the probability that a sample from θ a is bigger than a sample from θ b is 0.713. that is, there’s a good chance you’d get better service from the reseller with the lower average approval rating. now back to our original question. which of the three resellers is most likely to satisfy a customer? assume a uniform prior on θ x , θ y , and θ z , the probabilities of good service for each reseller. the posterior distributions on these variables have distributions beta(80082, 5113), beta(20370, 417), and beta(833, 9). these beta distributions have such large parameters that we can approximate them by normal distributions with the same mean and variance. (a beta( a , b ) random variable has mean a /( a + b ) and variance ab /(( a + b ) 2 ( a + b +1)).) the variable with the most variance, θ z , has standard deviation 0.003. the other variables have even smaller standard deviation. so the three distributions are highly concentrated at their mean values with practically non-overlapping support. and so a sample from θ x or θ y is unlikely to be higher than a sample from θ z . in general, going by averages alone works when you have a lot of customer reviews. but when you have a small number of reviews, going by averages alone could be misleading. thanks to charles mccreary for suggesting the xkcd comic. source: http://www.johndcook.com/blog/2011/09/27/bayesian-amazon/

Being a Spring framework user and enthusiast for many years I came across several misunderstandings and problems with this stack. Also there are places where abstractions leak terribly and to effectively and safely take advantage of all the features developers need to be aware of them. That is why I am starting a Spring pitfalls series. In the first part we will take a closer look at how proxying works. Bean proxying is an essential and one of the most important infrastructure features provided by Spring. It is so important and low-level that for most of the time we don't even realize that it exists. However transactions, aspect-oriented programming, advanced scoping, @Async support and various other domestic use-cases wouldn't be possible without it. So what is proxying? Here is an example: when you inject DAO into service, Spring takes DAO instances and injects it directly. That's it. However sometimes Spring needs to be aware of each and every call made by service (and any other bean) to DAO. For instance if DAO is marked transactional it needs to start a transaction before call and commit or rolls back afterwards. Of course you can do this manually, but this is tedious, error-prone and mixes concerns. That's why we use declarative transactions on the first place. So how does Spring implement this interception mechanism? There are three methods from simplest to most advanced ones. I won't discuss their advantages and disadvantages yet, we will see them soon on a concrete examples. Java dynamic proxies Simplest solution. If DAO implements any interface, Spring will create a Java dynamic proxy implementing that interface(s) and inject it instead of the real class. The real one still exists and the proxy has reference to it, but to the outside world – the proxy is the bean. Now every time you call methods on your DAO, Spring can intercept them, add some AOP magic and call the original method. CGLIB generated classes The downside of Java dynamic proxies is a requirement on the bean to implement at least one interface. CGLIB works around this limitation by dynamically subclassing the original bean and adding interception logic directly by overriding every possible method. Think of it as subclassing the original class and calling super version amongst other things: class DAO { def findBy(id: Int) = //... } class DAO$EnhancerByCGLIB extends DAO { override def findBy(id: Int) = { startTransaction try { val result = super.findBy(id) commitTransaction() result } catch { case e => rollbackTransaction() throw e } } } However, this pseudocode does not illustrate how it works in reality – which introduces yet another problem, stay tuned. BTW all examples will be in Scala, live with that and get used to it. AspectJ weaving This is the most invasive but also the most reliable and intuitive solution from the developer perspective. In this mode interception is applied directly to your class bytecode which means the class your JVM runs is not the same as the one you wrote. AspectJ weaver adds interception logic by directly modifying your bytecode of your class, either during build – compile time weaving (CTW) or when loading a class – load time weaving (LTW). If you are curious how AspectJ magic is implemented under the hood, here is a decompiled and simplified .class file compiled with AspectJ weaving beforehand: public void inInterfaceTransactional() { try { AnnotationTransactionAspect.aspectOf().ajc$before$1$2a73e96c(this, ajc$tjp_2); throwIfNotInTransaction(); } catch(Throwable throwable) { AnnotationTransactionAspect.aspectOf().ajc$afterThrowing$2$2a73e96c(this, throwable); throw throwable; } AnnotationTransactionAspect.aspectOf().ajc$afterReturning$3$2a73e96c(this); } With load time weaving the same transformation occurs at runtime, when the class is loaded. As you can see there is nothing disturbing here, in fact this is exactly how you would program the transactions manually. Side note: do you remember the times when viruses were appending their code into executable files or dynamically injecting themselves when executable was loaded by the operating system? Knowing proxy techniques is important to understand how proxying works and how it affects your code. Let us stick with declarative transaction demarcation example, here is our battlefield: trait FooService { def inInterfaceTransactional() def inInterfaceNotTransactional(); } @Service class DefaultFooService extends FooService { private def throwIfNotInTransaction() { assume(TransactionSynchronizationManager.isActualTransactionActive) } def publicNotInInterfaceAndNotTransactional() { inInterfaceTransactional() publicNotInInterfaceButTransactional() privateMethod(); } @Transactional def publicNotInInterfaceButTransactional() { throwIfNotInTransaction() } @Transactional private def privateMethod() { throwIfNotInTransaction() } @Transactional override def inInterfaceTransactional() { throwIfNotInTransaction() } override def inInterfaceNotTransactional() { inInterfaceTransactional() publicNotInInterfaceButTransactional() privateMethod(); } } Handy throwIfNotInTransaction() method... throws exception when not invoked within a transaction. Who would have thought? This method is called from various places and different configurations. If you examine carefully how methods are invoked – this should all work. However our developers' life tend to be brutal. First obstacle was unexpected: ScalaTest does not support Spring integration testing via dedicated runner. Luckily this can be easily ported with a simple trait (handles dependency injection to test cases and application context caching): trait SpringRule extends AbstractSuite { this: Suite => abstract override def run(testName: Option[String], reporter: Reporter, stopper: Stopper, filter: Filter, configMap: Map[String, Any], distributor: Option[Distributor], tracker: Tracker) { new TestContextManager(this.getClass).prepareTestInstance(this) super.run(testName, reporter, stopper, filter, configMap, distributor, tracker) } } Note that we are not starting and rolling back transactions like the original testing framework. Not only because it would interfere with our demo but also because I find transactional tests harmful – but more on that in the future. Back to our example, here is a smoke test. The complete source code can be downloaded here from proxy-problem branch. Don't complain about the lack of assertions – here we are only testing that exceptions are not thrown: @RunWith(classOf[JUnitRunner]) @ContextConfiguration class DefaultFooServiceTest extends FunSuite with ShouldMatchers with SpringRule{ @Resource private val fooService: FooService = null test("calling method from interface should apply transactional aspect") { fooService.inInterfaceTransactional() } test("calling non-transactional method from interface should start transaction for all called methods") { fooService.inInterfaceNotTransactional() } } Surprisingly, the test fails. Well, if you've been reading my articles for a while you shouldn't be surprised: Spring AOP riddle and Spring AOP riddle demystified. Actually, the Spring reference documentation explains this in great detail, also check out this SO question. In short – non transactional method calls transactional one but bypassing the transactional proxy. Even though it seems obvious that when inInterfaceNotTransactional() calls inInterfaceTransactional() the transaction should start – it does not. The abstraction leaks. By the way also check out fascinating Transaction strategies: Understanding transaction pitfalls article for more. Remember our example showing how CGLIB works? Also knowing how polymorphism works it seems like using class based proxies should help. inInterfaceNotTransactional() now calls inInterfaceTransactional() overriden by CGLIB/Spring, which in turns calls the original classes. Not a chance! This is the real implementation in pseudo-code: class DAO$EnhancerByCGLIB extends DAO { val target: DAO = ... override def findBy(id: Int) = { startTransaction try { val result = target.findBy(id) commitTransaction() result } catch { case e => rollbackTransaction() throw e } } } Instead of subclassing and instantiating subclassed bean Spring first creates the original bean and then creates a subclass which wraps the original one (somewhat Decorator pattern) in one of the post processors. This means that – again – the self call inside bean bypasses AOP proxy around our class. Of course using CGLIB changes how are bean behaves in few other ways. For instance we can now inject concrete class rather than an interface, in fact the interface is not even needed and CGLIB proxying is required in this circumstances. There are also drawbacks – constructor injection is no longer possible, see SPR-3150, which is a shame. So what about some more thorough tests? @RunWith(classOf[JUnitRunner]) @ContextConfiguration class DefaultFooServiceTest extends FunSuite with ShouldMatchers with SpringRule { @Resource private val fooService: DefaultFooService = null test("calling method from interface should apply transactional aspect") { fooService.inInterfaceTransactional() } test("calling non-transactional method from interface should start transaction for all called methods") { fooService.inInterfaceNotTransactional() } test("calling transactional method not belonging to interface should start transaction for all called methods") { fooService.publicNotInInterfaceButTransactional() } test("calling non-transactional method not belonging to interface should start transaction for all called methods") { fooService.publicNotInInterfaceAndNotTransactional() } } Please pick tests that will fail (pick exactly two). Can you explain why? Again common sense would suggest that everything should pass, but that's not the case. You can play around yourself, see class-based-proxy branch. We are not here to expose problems but to overcome them. Unfortunately our tangled service class can only be fixed using heavy artillery – true AspectJ weaving. Both compile- and load-time weaving makes the test pass. See aspectj-ctw and aspectj-ltw branches accordingly. You should now be asking yourself several question. Which approach should I take (or: do I really need to use AspectJ?) and why should I even bother? – amongst others. I would say – in most cases simple Spring proxying will suffice. But you absolutely have to be aware of how does the propagation work and when it doesn't. Otherwise bad things happen. Commits and rollbacks occurring in unexpected places, spanning unexpected amount of data, ORM dirty checking not working, invisible records – believe, this things happen on wild. And remember that topics we have covered here apply to all AOP aspects, not only transactions. From http://nurkiewicz.blogspot.com/2011/10/spring-pitfalls-proxying.html

Most people know that you can use @Autowired to tell Spring to inject one object into another when it loads your application context. A lesser known nugget of information is that you can also use the @Value annotation to inject values from a property file into a bean’s attributes. To demonstrate this requires a few bits and pieces, including a property file: jdbc.driverClassName=oracle.jdbc.OracleDriver jdbc.url=jdbc:oracle:thin:@on-the-beach:1521:mysid jdbc.username=john jdbc.password=lennon In this example, I’ve got a some simple datasource connection details for an Oracle database, which will be injected into a fake datasource class AutowiredFakaSource: @Component public class AutowiredFakaSource { @Value("${jdbc.driverClassName}") private String driverClassName; @Value("${jdbc.url}") private String url; @Value("${jdbc.username}") private String userName; @Value("${jdbc.password}") private String password; @Value("${java.io.tmpdir}") private String tmpDir; public AutowiredFakaSource() { } public String execute() { System.out.println("Execute FakaSource"); return "A Result"; } public String getDriverClassName() { return driverClassName; } public String getUrl() { return url; } public String getUserName() { return userName; } public String getPassword() { return password; } public String getTmpDir() { return tmpDir; } } In terms of this blog, AutowiredFakaSource doesn’t really need to do anything, it just has to be a bean with some attributes. The crux of the whole thing lies in the @Value annotations: @Value("${jdbc.username}") ...which shows that a value from a property file is referenced using its name and the ${} notation. The JUnit test below demonstrates that all this works okay. @Test public void testAutowiredPropertyPlaceHolder() { System.out.println("Autowired Property PlaceHolder Test."); ApplicationContext ctx = new ClassPathXmlApplicationContext("autowired_property_place_holder.xml"); AutowiredFakaSource fakeDataSource = ctx.getBean(AutowiredFakaSource.class); assertEquals("oracle.jdbc.OracleDriver", fakeDataSource.getDriverClassName()); assertEquals("jdbc:oracle:thin:@on-the-beach:1521:mysid", fakeDataSource.getUrl()); assertEquals("john", fakeDataSource.getUserName()); assertEquals("lennon", fakeDataSource.getPassword()); assertNotNull(fakeDataSource.getTmpDir()); String expected = System.getProperty("java.io.tmpdir"); assertEquals(expected, fakeDataSource.getTmpDir()); } On last thing to note is the XML configuration file. In the example below, you can see that the XML contains two entries. The first is the PropertyPlaceholderConfigurer class that loads the properties from the jdbc.properties file and second is the XML element that enables autowiring. jdbc.properties Finally, eagle eyed readers will have spotted that the @Value annotation also allows you to load system properties. In this example I’ve loaded the Java temp directory path using: @Value("${java.io.tmpdir}") …and then tested it using the final assert in the unit test: From http://www.captaindebug.com/2011/10/autowiring-property-values-into-spring_28.html

This is to remind me that I should never ever forget the cruical difference between build-time-only tags (i.e. having tag handlers only) and render-time tags that have corresponding components. The problem is that their lifespan is different and thus mixing them can easily lead to nasty surprises. Build time tags are used to modify the building of a component tree and have no effect during its rendering, where only the components participate. A typical mistake is the nesting of ui:include (build-time) inside ui:repeat (render-time) using the var that ui:repeat declares: This won’t work as intended because the var is only made available at the render time while ui:include is already evaluated at that point as it was invoked at the build time. This is why combining JSTL and JSF isn’t recommended in general. The complication with Facelets is that there is no clear distinct mark such as the namespace prefix that would distinguish build-time and render-time tags. In adition to JSTL also f.ex. f:actionListener, f:facet, ui:include, and any custom tag file are build-time while e.g. f:selectItems, ui:repeat,h:inputText, and any custom UIComponent are render time. An addition to that it seems that f:converter and f:validator are yet another special case [3] (though more like build-time tags). So make sure that you know which tags are build-time and which are render-time and when it is meaningful to mix them and when you should absolutely avoid it. References (highly recommended to read through): Andrew: Build time vs. render time (2008) – mainly about JSP but the last section relates it to Facelets, which have the same problem with its tag handlers as JSP with its (non-JSF) tags BalusC: Why @ViewScoped fails in tag handlers (2011, JSF 2.0) – the author lists render-time alternatives for build-time tags where available Roger Keays: c:forEach vs ui:repeat in Facelets (2007) From http://theholyjava.wordpress.com/2011/10/28/jsf-beware-the-difference-between-build-time-and-render-time-tags-in-facelets/

Developing with Grails and Groovy can be a blessing and and pain all at the same time. The development moves at a rapid rate but when you decide to include libraries that depend on other libraries, your pain starts to build up. For example, when you include the module “HttpBuilder”in your project you may run into issues with Xerces and xml-apis, especially when you attempt to deploy the WAR file under Tomcat. These libraries are included as part of Tomcat and so an older version of those classes may give you a heartburn. If your objective is to use some raw HTTP classes to create your requests and responses, then you can use the basic URL class to do most of the raw connection options. Although using HttpBuilder makes it a clean implementation, the URL class gives you very similar power without all the overhead of including the dependency classes. def urlConnect = new URL(url) def connection = urlConnect.openConnection() //Set all of your needed headers connection.setRequestProperty("X-Forwarded-For", "") if(connection.responseCode == 200){ responseText = connection.content.text } else{ println "An error occurred:" println connection.responseCode println connection.responseMessage } So the trick to the Groovy URL class is to use the “openConnection()” method and then gain access to some of the raw functionality. Cheers. From http://mythinkpond.wordpress.com/2011/10/24/grails-groovy-alternative-to-httpbuilder-adding-headers-to-your-http-request/

What is JIT Compiler? The Just In Time Compiler (JIT) concept and more generally adaptive optimization is well known concept in many languages besides Java (.Net, Lua, JRuby). In order to explain what is JIT Compiler I want to start with a definition of compiler concept. According to wikipedia compiler is "a computer program that transforms the source language into another computer language (the target language)". We are all familiar with static java compiler (javac) that compiles human readable .java files to a byte code that can be interpreted by JVM - .class files. Then what does JIT compile? The answer will given a moment later after explanation of what is "Just in Time". According to most researches, 80% of execution time is spent in executing 20% of code. That would be great if there was a way to determine those 20% of code and to optimize them. That's exactly what JIT does - during runtime it gathers statistics, finds the "hot" code compiles it from JVM interpreted bytecode (that is stored in .class files) to a native code that is executed directly by Operating System and heavily optimizes it. Smallest compilation unit is single method. Compilation and statistics gathering is done in parallel to program execution by special threads. During statistics gathering the compiler makes hypotheses about code function and as the time passes tries to prove or to disprove them. If the hypothesis is dis-proven the code is deoptimized and recompiled again. The name "Hotspot" of Sun (Oracle) JVM is chosen because of the ability of this Virtual Machine to find "hot" spots in code. What optimizations does JIT? Let's look closely at more optimizations done by JIT. Inline methods - instead of calling method on an instance of the object it copies the method to caller code. The hot methods should be located as close to the caller as possible to prevent any overhead. Eliminate locks if monitor is not reachable from other threads Replace interface with direct method calls for method implemented only once to eliminate calling of virtual functions overhead Join adjacent synchronized blocks on the same object Eliminate dead code Drop memory write for non-volatile variables Remove prechecking NullPointerException and IndexOutOfBoundsException Et cetera When the Java VM invokes a Java method, it uses an invoker method as specified in the method block of the loaded class object. The Java VM has several invoker methods, for example, a different invoker is used if the method is synchronized or if it is a native method. The JIT compiler uses its own invoker. Sun production releases check the method access bit for value ACC_MACHINE_COMPILED to notify the interpreter that the code for this method has already been compiled and stored in the loaded class. JIT compiler compiles the method block into native code for this method and stores that in the code block for that method. Once the code has been compiled the ACC_MACHINE_COMPILED bit, which is used on the Sun platform, is set. How do we know what JIT is doing in our program and how can it be controlled? First of all to disable JIT Djava.compiler=NONE parameter can be used. There are 2 types of JIT compilers in Hotspot - one is used for client program and one for server (-server option in VM parameters). Program, running on server enjoys usually from more resources than program running on client and to server program top throughput is usually more important. Hence JIT in server is more resource consuming and gathering statistics takes more time to make the statistics more accurate. For client program gathering statics for a method lasts 1500 method calls, for server 15000. These default values can be changed by -XX:CompileThreshold=XXX VM parameter. In order to find out whether default value is good for you try enabling "XX:+PrintCompilation" and "-XX:-CITime" parameters that print JIT statistics and time CPU spent by JIT. Benchmarks Most of the benchmarks show that JITed code runs 10 to 20 times faster than interpreted code. There are many benchmarks done. Below given result graphs of two of them: Its worth to mention that programs that run in JIT mode, but are still in "learning mode" run much slower than non JITed programs. Drawbacks of JIT JIT Increases level of unpredictability and complexity in Java program. It adds another layer that developers don't really understand. Example of possible bugs - 'happens before relations" in concurrency. JIT can easily reorder code if the change is safe for a program running in single thread. To solve this problem developers make hints to JIT using "synchronized" word or explicit locking. Increases non heap memory footprint - JITed code is stored in "Code Cache" generation. Advanced JIT JIT and garbage collection. For GC to occur program must reach safe points. For this purpose JIT injects yieldpoints at regular intervals in native code. In addition to scanning of stack to find root references, registers must be scanned as they may hold objects created by JIT Comments are appresiated. The article can be found also at: artiomg.blogspot.com/2011/10/just-in-time-compiler-jit-in-hotspot.html

Hibernate (and actually JPA) has collection mappings: @OneToMany, @ManyToMany, @ElementCollection. All of these are by default lazy. This means the collections are specific implementations of the List or Set interface that hold a reference to the persistent session and the values are loaded from the database only if the collection is accessed. That saves unnecessary database queries if you only occasionally use the collection. However, there’s a problem with that. The problem that manifests itself through the exception that in my observations is 2nd most commonly asked exception (after NullPointerException) – the LazyInitializationException. The problem is that the session is usually open for your service layer and is closed as soon as you return the entity to the view layer. And when you try to iterate the uninitialized collection in your view (jsp for example), the collection throws LazyInitializationException, because the session that they hold a reference to is already closed and they can’t fetch the items. How is this solved? The so called OpenSessionInView / OpenEntityManagerInView “patterns”. In short: you make a filter that opens the session when the request starts and closes it after the view has been rendered (and not after the service layer finishes). Some people call that an anti-pattern, because it leaks persistence handling into the view layer, and complicates the setup. I wouldn’t say it’s that bad: generally it solves the problem without introducing other problems. But in all recent project I’ve been involved, we aren’t using OpenSessionInView, and it works fine. It works fine because we aren’t using lazy collections. But then, you’ll rightly point, you will be fetching “the whole world” when you load a single entity. Well, no. There are two types of *ToMany mappings: value-type mappings where the collection logically does not hold more than a dozen elements. This is in most cases @ElementCollection, and also @*ToMany with items like “Category” or “Price” that are just more complex value objects, but that do not hold any other mappings themselves. Another common feature of these types of collections is that they are usually displayed in the UI together with their owning entity. It is most likely that you want to display the categories of an article, for example. For this type of collections EAGER is the better option. You’ll have to fetch them anyway, why not let hibernate (or any jpa implementation) think of some clever join? As I said – the collections are logically not bigger than a dozen or two, so fetching them won’t be a performance hit. And, logically, they won’t fetch a big object graph with them. mappings across the big, core entities. This can be “all orders made by the user” or “all users for the organization”, “all items of the supplier”, etc. You certainly don’t want to fetch them eagerly. Because if you fetch 2000 users for an organization, which in turn have 1000 orders each, and an order has 3 items on average which in turn have a collection of all people who have purchased it.. you’ll end up with your entire database in memory. Obviously you need lazy collections, right? Well, no. In that case you should not be using collection mappings at all. These types of relations are, in 99% of the cases, displayed in paged lists in the UI. Or in search results. They are never (and should never) be displayed all on one screen (or should rarely be returned in one API call, if your application provides something like a REST API). You have to make queries for them, and use query.setMaxResults and query.setFirstResult() (or limit them with some restrictive criteria). Furthermore having the collections mapped means someone will try to use them at some point, which may fail. And if the object is serialized (xml, json, etc.) the collection contents will be fetched. Something you almost certainly don’t want to happen. (A draft idea here: JPA could have a PagedList collection that would allow paged lazy fetching, thus eliminating the need for a query) So what did I just say – that you should never use lazy collections. Use eager collections for very simple, shallow mappings, and use paged queries for the bigger ones. Well, not exactly. Lazy collections are there and they have application, though it is rather limited. Or at least they are way less applicable than they are used. Here’s an example scenario where I found it applicable. In my side-project I have a Message entity, and it holds a collection of Picture entities. When a user uploads a picture, it is stored in that collection. A message can have no more than 10 pictures, so the collection could very well be eager. But then, Message is the most commonly used entity – it’s fetched virtually on every request. But only some messages have pictures (how many of the tweets on your stream have a a picture upload?). So I don’t want hibernate to make queries just to find out there are no pictures for a given message. Hence I store the number of pictures in a separate field, make the pictures collection lazy, and Hibernate.initialize(..) it manually only if the number of pictures is > 0. So there are scenarios, when the entity has optional collections that fall into the first category above (“small, shallow collections”). So if it is small, shallow and optional (say, used in less than 20% of the cases), then you should go with Lazy to save unnecessary queries. For everything else – having lazy collections will make your life harder. From http://techblog.bozho.net/?p=645

How often have we all heard that “batching” will increase latency? As someone with a passion for low-latency systems this surprises me. In my experience when batching is done correctly, not only does it increase throughput, it can also reduce average latency and keep it consistent. Well then, how can batching magically reduce latency? It comes down to what algorithm and data structures are employed. In a distributed environment we are often having to batch up messages/events into network packets to achieve greater throughput. We also employ similar techniques in buffering writes to storage to reduce the number of IOPS. That storage could be a block device backed file-system or a relational database. Most IO devices can only handle a modest number of IO operations per second, so it is best to fill those operations efficiently. Many approaches to batching involve waiting for a timeout to occur and this will by its very nature increase latency. The batch can also get filled before the timeout occurs making the latency even more unpredictable. Figure 1. Figure 1. above depicts decoupling the access to an IO device, and therefore the contention for access to it, by introducing a queue like structure to stage the messages/events to be sent and a thread doing the batching for writing to the device. The Algorithm An approach to batching uses the following algorithm in Java pseudo code: public final class NetworkBatcher implements Runnable { private final NetworkFacade network; private final Queue queue; private final ByteBuffer buffer; public NetworkBatcher(final NetworkFacade network, final int maxPacketSize, final Queue queue) { this.network = network; buffer = ByteBuffer.allocate(maxPacketSize); this.queue = queue; } @Override public void run() { while (!Thread.currentThread().isInterrupted()) { while (null == queue.peek()) { employWaitStrategy(); // block, spin, yield, etc. } Message msg; while (null != (msg = queue.poll())) { if (msg.size() > buffer.remaining()) { sendBuffer(); } buffer.put(msg.getBytes()); } sendBuffer(); } } private void sendBuffer() { buffer.flip(); network.send(buffer); buffer.clear(); } } Basically, wait for data to become available and as soon as it is, send it right away. While sending a previous message or waiting on new messages, a burst of traffic may arrive which can all be sent in a batch, up to the size of the buffer, to the underlying resource. This approach can use ConcurrentLinkedQueue which provides low-latency and avoid locks. However it has an issue in not creating back pressure to stall producing/publishing threads if they are outpacing the batcher whereby the queue could grow out of control because it is unbounded. I’ve often had to wrap ConcurrentLinkedQueue to track its size and thus create back pressure. This size tracking can add 50% to the processing cost of using this queue in my experience. This algorithm respects the single writer principle and can often be employed when writing to a network or storage device, and thus avoid lock contention in third party API libraries. By avoiding the contention we avoid the J-Curve latency profile normally associated with contention on resources, due to the queuing effect on locks. With this algorithm, as load increases, latency stays constant until the underlying device is saturated with traffic resulting in a more "bathtub" profile than the J-Curve. Let’s take a worked example of handling 10 messages that arrive as a burst of traffic. In most systems traffic comes in bursts and is seldom uniformly spaced out in time. One approach will assume no batching and the threads write to device API directly as in Figure 1. above. The other will use a lock free data structure to collect the messages plus a single thread consuming messages in a loop as per the algorithm above. For the example let’s assume it takes 100µs to write a single buffer to the network device as a synchronous operation and have it acknowledged. The buffer will ideally be less than the MTU of the network in size when latency is critical. Many network sub-systems are asynchronous and support pipelining but we will make the above assumption to clarify the example. If the network operation is using a protocol like HTTP under REST or Web Services then this assumption matches the underlying implementation. Best (µs) Average (µs) Worst (µs) Packets Sent Serial 100 500 1,000 10 Smart Batching 100 150 200 1-2 The absolute lowest latency will be achieved if a message is sent from the thread originating the data directly to the resource, if the resource is un-contended. The table above shows what happens when contention occurs and a queuing effect kicks in. With the serial approach 10 individual packets will have to be sent and these typically need to queue on a lock managing access to the resource, therefore they get processed sequentially. The above figures assume the locking strategy works perfectly with no perceivable overhead which is unlikely in a real application. For the batching solution it is likely all 10 packets will be picked up in first batch if the concurrent queue is efficient, thus giving the best case latency scenario. In the worst case only one message is sent in the first batch with the other nine following in the next. Therefore in the worst case scenario one message has a latency of 100µs and the following 9 have a latency of 200µs thus giving a worst case average of 190µs which is significantly better than the serial approach. This is one good example when the simplest solution is just a bit too simple because of the contention. The batching solution helps achieve consistent low-latency under burst conditions and is best for throughput. It also has a nice effect across the network on the receiving end in that the receiver has to process fewer packets and therefore makes the communication more efficient both ends. Most hardware handles data in buffers up to a fixed size for efficiency. For a storage device this will typically be a 4KB block. For networks this will be the MTU and is typically 1500 bytes for Ethernet. When batching, it is best to understand the underlying hardware and write batches down in ideal buffer size to be optimally efficient. However keep in mind that some devices need to envelope the data, e.g. the Ethernet and IP headers for network packets so the buffer needs to allow for this. There will always be an increased latency from a thread switch and the cost of exchange via the data structure. However there are a number of very good non-blocking structures available using lock-free techniques. For the Disruptor this type of exchange can be achieved in as little as 50-100ns thus making the choice of taking the smart batching approach a no brainer for low-latency or high-throughput distributed systems. This technique can be employed for many problems and not just IO. The core of the Disruptor uses this technique to help rebalance the system when the publishers burst and outpace the EventProcessors. The algorithm can be seen inside the BatchEventProcessor. Note: For this algorithm to work the queueing structure must handle the contention better than the underlying resource. Many queue implementations are extremely poor at managing contention. Use science and measure before coming to a conclusion. Batching with the Disruptor The code below shows the same algorithm in action using the Disruptor's EventHandler mechanism. In my experience, this is a very effective technique for handling any IO device efficiently and keeping latency low when dealing with load or burst traffic. public final class NetworkBatchHandler implements EventHander { private final NetworkFacade network; private final ByteBuffer buffer; public NetworkBatchHandler(final NetworkFacade network, final int maxPacketSize) { this.network = network; buffer = ByteBuffer.allocate(maxPacketSize); } public void onEvent(Message msg, long sequence, boolean endOfBatch) throws Exception { if (msg.size() > buffer.remaining()) { sendBuffer(); } buffer.put(msg.getBytes()); if (endOfBatch) { sendBuffer(); } } private void sendBuffer() { buffer.flip(); network.send(buffer); buffer.clear(); } } The endOfBatch parameter greatly simplifies the handling of the batch compared to the double loop in the algorithm above. I have simplified the examples to illustrate the algorithm. Clearly error handling and other edge conditions need to be considered. Separation of IO from Work Processing There is another very good reason to separate the IO from the threads doing the work processing. Handing off the IO to another thread means the worker thread, or threads, can continue processing without blocking in a nice cache friendly manner. I've found this to be critical in achieving high-performance throughput. If the underlying IO device or resource becomes briefly saturated then the messages can be queued for the batcher thread allowing the work processing threads to continue. The batching thread then feeds the messages to the IO device in the most efficient way possible allowing the data structure to handle the burst and if full apply the necessary back pressure, thus providing a good separation of concerns in the workflow. Conclusion So there you have it. Smart Batching can be employed in concert with the appropriate data structures to achieve consistent low-latency and maximum throughput. From http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html

There is more and more writing on DevOps lately, which is good and bad. There still remains a small core of thoughtful people that are worth listening to and learning from. There’s more and more marketing from vendors and consultants jumping on the Devops bandwagon. There’s some naïve silliness (“Hire wicked smart people and give them all access to root.”) which can probably be safely ignored. And then there’s stuff that is half-right and half-wrong, too dangerous to be ignored. Like this recent post on Rollbacks and Other Deployment Myths, in which John Vincent lists 5 “myths” about system deployment, which I want to take some time to respond to here. Change Is Change? The author tries to make the point that “Change is neither good or bad. It’s just change.” Therefore we do not need to be afraid of making changes. I don’t agree. This attitude to change ignores the fact that once a system is up and running and customers are relying on it to conduct their business, whatever change you are making to the system is almost never as important as making sure that the system keeps running properly. Unfortunately, changes often lead to problems. We know from Visible Ops that based on studies of hundreds of companies, 80% of operational failures are caused by mistakes made during changes. This is where heavyweight process control frameworks like ITIL and COBIT, and detective change control tools like Tripwire came from. To help companies get control over IT change, because people had to find some way to stop shit from breaking. Yes, I get the point that in IT we tend to over-compensate, and I agree that calling in the Release Police and trying to put up a wall around all changes isn’t sustainable. People don’t have to work this way. But trivializing change, pretending that changes don't lead to problems, is dangerous. Deploys Are Not Risky? You can be smart and careful and break changes down into small steps and try to automate code pushes and configuration changes, and plan ahead and stage and review and test all your changes, and after all of this you can still mess up the deploy. Even if you make frequent small changes and simplify the work and practice it a lot. For systems like Facebook and online games and a small number of other cases, maybe deployment really is a non-issue. I don’t care if Facebook deploys in the middle of the day – I can usually tell when they are doing a “zero downtime” deploy (or maybe they are “transparently” recovering from a failure) because data disappears temporarily or shows up in the wrong order, functions aren’t accessible for a while, forms don’t resolve properly, and other weird shit happens, and then things come back later or they don’t. As a customer, do I care? No. It’s an inconvenience, and it’s occasionally unsettling ("WTF just happened?"), but I get used to it and so do millions of others. That’s because most of us don’t use Facebook or systems like this for anything important. For business-critical systems handling thousands of transactions a second that are tied into hundreds of other company’s systems (the world that I work in) this doesn’t cut it. Maybe I spend too much time at this extreme, where even small problems with compatibility that only affect a small number of customers, or slight and temporary performance slow downs, are a big deal. But most people I work with and talk to in software development and maintenance and system operations agree that deployment is a big deal and needs to be done with care and attention, no matter how simple and small the changes are and no matter how clean and simple and automated the deployment process is. Rollbacks Are a Myth? Vincent wants us to “understand that it’s typically more risky to rollback than rolling forward. Always be rolling forward.” Not even the Continuous Deployment advocates (who are often some of the most radical – and I think some of the most irresponsible – voices in the Devops community) agree with this – they still roll back if they find problems with changes. “Rollbacks are a myth” is an echo of the “real men fail forward” crap I heard at Velocity last year and it is where I draw the line. It's one thing to state an extreme position for argument's sake or put up a straw man – but this is just plain wrong. If you're going to deploy, you have to anticipate roll back and think about it when you make changes and you have to test rolling back to make sure that it works. All of this is hard. But without a working roll back you have no choice other than to fail forward (whatever this means, because nobody who talks about it actually explains how to do it), and this is putting your customers and their business at unnecessary risk. It’s not another valid way of thinking. It’s irresponsible. James Hamilton wrote an excellent paper on Designing and Delivering Internet-Scale Services when he was at Microsoft (now he’s a an executive and Distinguished Engineer at Amazon). Hamilton’s paper remains one of the smartest things that anyone has written about how to deal with deployment and operational problems at scale. Everyone who designs, builds, maintains or operates an online system should read it. His position on roll back is simple and obvious and right: Reverting to the previous version is a rip cord that should always be available on any deployment. Everything Fails. Embrace Failure. I agree that everything can and will fail some day, that we can’t pretend that we can prevent failures in any system. But I don’t agree with embracing failure, at least in business-critical enterprise systems, where recovering from a failure means lost business and requires unraveling chains of transactions between different upstream and downstream systems and different companies, messing up other companies' businesses as well as your own and dealing the follow-on compliance problems. Failures in these kinds of systems, and a lot of other systems, are ugly and serious, and they should be treated seriously. We do whatever we can to make sure that failures are controlled and isolated, and we make sure that we can recover quickly if something goes wrong (which includes being able to roll back!). But we also do everything that we can to prevent failures. Embracing failure is fine for online consumer web site startups – let’s leave it to them. SLAs I wanted to respond to the points about SLAs, but it’s not clear to me what the author was trying to say. SLAs are not about servers. Umm, yes that’s right of course... SLAs are important to set business expectations with your customers (the people who are using the system) and with your partners and suppliers. So that your partners and suppliers know what you need from them and what you are paying them for, and so that you know if you can depend on them when you have to. So that your customers know what they are paying you for. And SLAs (not just high-level uptime SLAs, but SLAs for Recovery Time and Recovery Point goals and incident response and escalation) are important so that your team understands the constraints that they need to work under, what trade-offs to make in design and implementation and in operations. Under-Compensating Is Worse Than Over-Compensating I spent more time than I thought I would responding to this post, because some of what the author says is right – especially in the second part of his post, Deploy All the Things where he provides some good practical advice on how to reduce risk in deployment. He’s right that Operations main purpose isn’t to stop change – it can’t be. We have to be able to keep changing, and developers and operations have to work together to do this in safe and efficient ways. But trivializing the problems and risks of change and over-simplifying how to deal with these risks and how to deal with failures, isn’t the way to do this. There has to be a middle way between the ITIL and COBIT world of controls and paper and process, and cool Web startups failing forward, a way that can really work for the rest of us. Source: http://swreflections.blogspot.com/2011/10/rolling-forward-and-other-deployment.html

as a user interface fan, i could not miss the development with html 5. so the goal of this post is to walk through a graphic application that uses javascript and html 5. we will see through examples one way (among others) to develop this kind of project. application overview tools the html 5 page data gathering cards loading & cache handling cards display mouse management state storage animations handling multi-devices conclusion to go further application overview we will produce an application that will let us display a magic the gathering ©(courtesy of www.wizards.com/magic ) cards collection. users will be able to scroll and zoom using the mouse (like bing maps, for example). you can see the final result here: http://bolaslenses.catuhe.com the project source files can be downloaded here: http://www.catuhe.com/msdn/bolaslenses.zip cards are stored on windows azure storage and use the azure content distribution network ( cdn : a service that deploys data near the final users) in order to achieve maximum performances. an asp.net service is used to return cards list (using json format). tools to write our application, we will use visual studio 2010 sp1 with web standards update . this extension adds intellisense support in html 5 page (which is a really important thing ). so, our solution will contain an html 5 page side by side with .js files (these files will contain javascript scripts). about debug, it is possible to set a breakpoint directly in the .js files under visual studio. it is also possible to use the developer bar of internet explorer 9 (use f12 key to display it). debug with visual studio 2010 debug with internet explorer 9 (f12/developer bar) so, we have a modern developer environment with intellisense and debug support. therefore, we are ready to start and first of all, we will write the html 5 page. the html 5 page our page will be built around an html 5 canvas which will be used to draw the cards: cards scanned by mwshq team magic the gathering official site : http://www.wizards.com/magic bolas lenses your browser does not support html5 canvas. loading data... if we dissect this page, we can note that it is divided into two parts: the header part with the title, the logo and the special mentions the main part (section) holds the canvas and the tooltips that will display the status of the application. there is also a hidden image ( backimage ) used as source for not yet loaded cards. to build the layout of the page, a style sheet ( full.css ) is applied. style sheets are a mechanism used to change the tags styles (in html, a style defines the entire display options for a tag): html, body { height: 100%; } body { background-color: #888888; font-size: .85em; font-family: "segoe ui, trebuchet ms" , verdana, helvetica, sans-serif; margin: 0; padding: 0; color: #696969; } a:link { color: #034af3; text-decoration: underline; } a:visited { color: #505abc; } a:hover { color: #1d60ff; text-decoration: none; } a:active { color: #12eb87; } header, footer, nav, section { display: block; } table { width: 100%; } header, #header { position: relative; margin-bottom: 0px; color: #000; padding: 0; } #title { font-weight: bold; color: #fff; border: none; font-size: 60px !important; vertical-align: middle; margin-left: 70px } #legal { text-align: right; color: white; font-size: 14px; width: 50%; position: absolute; top: 15px; right: 10px } #leftheader { width: 50%; vertical-align: middle; } section { margin: 20px 20px 20px 20px; } #maincanvas{ border: 4px solid #000000; } #cardscount { font-weight: bolder; font-size: 1.1em; } .tooltip { position: absolute; bottom: 5px; color: black; background-color: white; margin-right: auto; margin-left: auto; left: 35%; right: 35%; padding: 5px; width: 30%; text-align: center; border-radius: 10px; -webkit-border-radius: 10px; -moz-border-radius: 10px; box-shadow: 2px 2px 2px #333333; } #bolaslogo { width: 64px; height: 64px; } #picturecell { float: left; width: 64px; margin: 5px 5px 5px 5px; vertical-align: middle; } thus, this sheet is responsible for setting up the following display: style sheets are powerful tools that allow an infinite number of displays. however, they are sometimes complicated to setup (for example if a tag is affected by a class, an identifier and its container). to simplify this setup, the development bar of internet explorer 9 is particularly useful because we can use it to see styles hierarchy that is applied to a tag. for example let’s take a look at the waittext tooltip with the development bar. to do this, you must press f12 in internet explorer and use the selector to choose the tooltip: once the selection is done, we can see the styles hierarchy: thus, we can see that our div received its styles from the body tag and the . tooltip entry of the style sheet. with this tool, it becomes possible to see the effect of each style (which can be disabled). it is also possible to add new style on the fly. another important point of this window is the ability to change the rendering mode of internet explorer 9. indeed, we can test how, for example, internet explorer 8 will handle the same page. to do this, go to the [ browser mode ] menu and select the engine of internet explorer 8. this change will especially impact our tooltip as it uses border-radius (rounded edge) and box-shadow that are features of css 3: internet explorer 9 internet explorer 8 our page provides a graceful degradation as it still works (with no annoying visual difference) when the browser does not support all the required technologies. now that our interface is ready, we will take a look at the data source to retrieve the cards to display. the server provides the cards list using json format on this url: http://bolaslenses.catuhe.com/ home/listofcards/?colorstring=0 it takes one parameter ( colorstring ) to select a specific color (0 = all). when developing with javascript, there is a good reflex to have (reflex also good in other languages too, but really important in javascript): one must ask whether what we want to develop has not been already done in an existing framework. indeed, there is a multitude of open source projects around javascript. one of them is jquery which provides a plethora of convenient services. thus, in our case to connect to the url of our server and get the cards list, we could go through a xmlhttprequest and have fun to parse the returned json. or we can use jquery . so we will use the getjson function which will take care of everything for us: function getlistofcards() { var url = "http://bolaslenses.catuhe.com/home/listofcards/?jsoncallback=?"; $.getjson(url, { colorstring: "0" }, function (data) { listofcards = data; $("#cardscount").text(listofcards.length + " cards displayed"); $("#waittext").slidetoggle("fast"); }); } as we can see, our function stores the cards list in the listofcards variable and calls two jquery functions: text that change the text of a tag slidetoggle that hides (or shows) a tag by animating its height the listofcards list contains objects whose format is: id : unique identifier of the card path : relative path of the card (without the extension) it should be noted that the url of the server is called with the “ ?jsoncallback=? ” suffix. indeed, ajax calls are constrained in terms of security to connect only to the same address as the calling script. however, there is a solution called jsonp that will allow us to make a concerted call to the server (which of course must be aware of the operation). and fortunately, jquery can handle it all alone by just adding the right suffix. once we have our cards list, we can set up the pictures loading and caching. cards loading & cache handling the main trick of our application is to draw only the cards effectively visible on the screen. the display window is defined by a zoom level and an offset (x, y) in the overall system. var visucontrol = { zoom : 0.25, offsetx : 0, offsety : 0 }; the overall system is defined by 14819 cards that are spread over 200 columns and 75 rows. also, we must be aware that each card is available in three versions: high definition: 480x680 without compression (.jpg suffix) medium definition: 240x340 with standard compression (.50.jpg suffix) low definition: 120x170 with strong compression (.25.jpg suffix) thus, depending on the zoom level, we will load the correct version to optimize networks transfer. to do this we will develop a function that will give an image for a given card. this function will be configured to download a certain level of quality. in addition it will be linked with lower quality level to return it if the card for the current level is not yet uploaded: function imagecache(substr, replacementcache) { var extension = substr; var backimage = document.getelementbyid("backimage"); this.load = function (card) { var localcache = this; if (this[card.id] != undefined) return; var img = new image(); localcache[card.id] = { image: img, isloaded: false }; currentdownloads++; img.onload = function () { localcache[card.id].isloaded = true; currentdownloads--; }; img.onerror = function() { currentdownloads--; }; img.src = "http://az30809.vo.msecnd.net/" + card.path + extension; }; this.getreplacementfromlowercache = function (card) { if (replacementcache == undefined) return backimage; return replacementcache.getimageforcard(card); }; this.getimageforcard = function(card) { var img; if (this[card.id] == undefined) { this.load(card); img = this.getreplacementfromlowercache(card); } else { if (this[card.id].isloaded) img = this[card.id].image; else img = this.getreplacementfromlowercache(card); } return img; }; } an imagecache is built by giving the associated suffix and the underlying cache. here you can see two important functions: load : this function will load the right picture and will store it in a cache (the msecnd.net url is the azure cdn address of the cards) getimageforcard : this function returns the card picture from the cache if already loaded. otherwise it requests the underlying cache to return its version (and so on) so to handle our 3 levels of caches, we have to declare three variables: var imagescache25 = new imagecache(".25.jpg"); var imagescache50 = new imagecache(".50.jpg", imagescache25); var imagescachefull = new imagecache(".jpg", imagescache50); selecting the right cover is only depending on zoom: function getcorrectimagecache() { if (visucontrol.zoom <= 0.25) return imagescache25; if (visucontrol.zoom <= 0.8) return imagescache50; return imagescachefull; } to give a feedback to the user, we will add a timer that will manage a tooltip that indicates the number of images currently loaded: function updatestats() { var stats = $("#stats"); stats.html(currentdownloads + " card(s) currently downloaded."); if (currentdownloads == 0 && statsvisible) { statsvisible = false; stats.slidetoggle("fast"); } else if (currentdownloads > 1 && !statsvisible) { statsvisible = true; stats.slidetoggle("fast"); } } setinterval(updatestats, 200); again we note the use of jquery to simplify animations. we will now discuss the display of cards. cards display to draw our cards, we need to actually fill the canvas using its 2d context (which exists only if the browser supports html 5 canvas): var maincanvas = document.getelementbyid("maincanvas"); var drawingcontext = maincanvas.getcontext('2d'); the drawing will be made by processlistofcards function (called 60 times per second): function processlistofcards() { if (listofcards == undefined) { drawwaitmessage(); return; } maincanvas.width = document.getelementbyid("center").clientwidth; maincanvas.height = document.getelementbyid("center").clientheight; totalcards = listofcards.length; var localcardwidth = cardwidth * visucontrol.zoom; var localcardheight = cardheight * visucontrol.zoom; var effectivetotalcardsinwidth = colscount * localcardwidth; var rowscount = math.ceil(totalcards / colscount); var effectivetotalcardsinheight = rowscount * localcardheight; initialx = (maincanvas.width - effectivetotalcardsinwidth) / 2.0 - localcardwidth / 2.0; initialy = (maincanvas.height - effectivetotalcardsinheight) / 2.0 - localcardheight / 2.0; // clear clearcanvas(); // computing of the viewing area var initialoffsetx = initialx + visucontrol.offsetx * visucontrol.zoom; var initialoffsety = initialy + visucontrol.offsety * visucontrol.zoom; var startx = math.max(math.floor(-initialoffsetx / localcardwidth) - 1, 0); var starty = math.max(math.floor(-initialoffsety / localcardheight) - 1, 0); var endx = math.min(startx + math.floor((maincanvas.width - initialoffsetx - startx * localcardwidth) / localcardwidth) + 1, colscount); var endy = math.min(starty + math.floor((maincanvas.height - initialoffsety - starty * localcardheight) / localcardheight) + 1, rowscount); // getting current cache var imagecache = getcorrectimagecache(); // render for (var y = starty; y < endy; y++) { for (var x = startx; x < endx; x++) { var localx = x * localcardwidth + initialoffsetx; var localy = y * localcardheight + initialoffsety; // clip if (localx > maincanvas.width) continue; if (localy > maincanvas.height) continue; if (localx + localcardwidth < 0) continue; if (localy + localcardheight < 0) continue; var card = listofcards[x + y * colscount]; if (card == undefined) continue; // get from cache var img = imagecache.getimageforcard(card); // render try { if (img != undefined) drawingcontext.drawimage(img, localx, localy, localcardwidth, localcardheight); } catch (e) { $.grep(listofcards, function (item) { return item.image != img; }); } } }; // scroll bars drawscrollbars(effectivetotalcardsinwidth, effectivetotalcardsinheight, initialoffsetx, initialoffsety); // fps computefps(); } this function is built around many key points: if the cards list is not yet loaded, we display a tooltip indicating that download is in progress: var pointcount = 0; function drawwaitmessage() { pointcount++; if (pointcount > 200) pointcount = 0; var points = ""; for (var index = 0; index < pointcount / 10; index++) points += "."; $("#waittext").html("loading...please wait" + points); subsequently, we define the position of the display window (in terms of cards and coordinates), then we proceed to clean the canvas: function clearcanvas() { maincanvas.width = document.body.clientwidth - 50; maincanvas.height = document.body.clientheight - 140; drawingcontext.fillstyle = "rgb(0, 0, 0)"; drawingcontext.fillrect(0, 0, maincanvas.width, maincanvas.height); } then we browse the cards list and call the drawimage function of the canvas context. the current image is provided by the active cache (depending on the zoom): // get from cache var img = imagecache.getimageforcard(card); // render try { if (img != undefined) drawingcontext.drawimage(img, localx, localy, localcardwidth, localcardheight); } catch (e) { $.grep(listofcards, function (item) { return item.image != img; }); we also have to draw the scroll bar with the roundedrectangle function that uses quadratic curves: function roundedrectangle(x, y, width, height, radius) { drawingcontext.beginpath(); drawingcontext.moveto(x + radius, y); drawingcontext.lineto(x + width - radius, y); drawingcontext.quadraticcurveto(x + width, y, x + width, y + radius); drawingcontext.lineto(x + width, y + height - radius); drawingcontext.quadraticcurveto(x + width, y + height, x + width - radius, y + height); drawingcontext.lineto(x + radius, y + height); drawingcontext.quadraticcurveto(x, y + height, x, y + height - radius); drawingcontext.lineto(x, y + radius); drawingcontext.quadraticcurveto(x, y, x + radius, y); drawingcontext.closepath(); drawingcontext.stroke(); drawingcontext.fill(); } function drawscrollbars(effectivetotalcardsinwidth, effectivetotalcardsinheight, initialoffsetx, initialoffsety) { drawingcontext.fillstyle = "rgba(255, 255, 255, 0.6)"; drawingcontext.linewidth = 2; // vertical var totalscrollheight = effectivetotalcardsinheight + maincanvas.height; var scaleheight = maincanvas.height - 20; var scrollheight = maincanvas.height / totalscrollheight; var scrollstarty = (-initialoffsety + maincanvas.height * 0.5) / totalscrollheight; roundedrectangle(maincanvas.width - 8, scrollstarty * scaleheight + 10, 5, scrollheight * scaleheight, 4); // horizontal var totalscrollwidth = effectivetotalcardsinwidth + maincanvas.width; var scalewidth = maincanvas.width - 20; var scrollwidth = maincanvas.width / totalscrollwidth; var scrollstartx = (-initialoffsetx + maincanvas.width * 0.5) / totalscrollwidth; roundedrectangle(scrollstartx * scalewidth + 10, maincanvas.height - 8, scrollwidth * scalewidth, 5, 4); } and finally, we need to compute the number of frames per second: function computefps() { if (previous.length > 60) { previous.splice(0, 1); } var start = (new date).gettime(); previous.push(start); var sum = 0; for (var id = 0; id < previous.length - 1; id++) { sum += previous[id + 1] - previous[id]; } var diff = 1000.0 / (sum / previous.length); $("#cardscount").text(diff.tofixed() + " fps. " + listofcards.length + " cards displayed"); } drawing cards relies heavily on the browser's ability to speed up canvas rendering. for the record, here are the performances on my machine with the minimum zoom level (0.05): browser fps internet explorer 9 30 firefox 5 30 chrome 12 17 ipad (with a zoom level of 0.8) 7 windows phone mango (with a zoom level of 0.8) 20 (!!) the site even works on mobile phones and tablets as long as they support html 5. here we can see the inner power of html 5 browsers that can handle a full screen of cards more than 30 times per second! mouse management to browse our cards collection, we have to manage the mouse (including its wheel). for the scrolling, we'll just handle the onmouvemove , onmouseup and onmousedown events. onmouseup and onmousedown events will be used to detect if the mouse is clicked or not: var mousedown = 0; document.body.onmousedown = function (e) { mousedown = 1; getmouseposition(e); previousx = posx; previousy = posy; }; document.body.onmouseup = function () { mousedown = 0; }; the onmousemove event is connected to the canvas and used to move the view: var previousx = 0; var previousy = 0; var posx = 0; var posy = 0; function getmouseposition(eventargs) { var e; if (!eventargs) e = window.event; else { e = eventargs; } if (e.offsetx || e.offsety) { posx = e.offsetx; posy = e.offsety; } else if (e.clientx || e.clienty) { posx = e.clientx; posy = e.clienty; } } function onmousemove(e) { if (!mousedown) return; getmouseposition(e); mousemovefunc(posx, posy, previousx, previousy); previousx = posx; previousy = posy; } this function (onmousemove) calculates the current position and provides also the previous value in order to move the offset of the display window: function move(posx, posy, previousx, previousy) { currentaddx = (posx - previousx) / visucontrol.zoom; currentaddy = (posy - previousy) / visucontrol.zoom; } mousehelper.registermousemove(maincanvas, move); note that jquery also provides tools to manage mouse events. for the management of the wheel, we will have to adapt to different browsers that do not behave the same way on this point: function wheel(event) { var delta = 0; if (event.wheeldelta) { delta = event.wheeldelta / 120; if (window.opera) delta = -delta; } else if (event.detail) { /** mozilla case. */ delta = -event.detail / 3; } if (delta) { wheelfunc(delta); } if (event.preventdefault) event.preventdefault(); event.returnvalue = false; } we can see that everyone does what he wants :). the function to register with this event is: mousehelper.registerwheel = function (func) { wheelfunc = func; if (window.addeventlistener) window.addeventlistener('dommousescroll', wheel, false); window.onmousewheel = document.onmousewheel = wheel; }; and we will use this function to change the zoom with the wheel: // mouse mousehelper.registerwheel(function (delta) { currentaddzoom += delta / 500.0; }); finally we will add a bit of inertia when moving the mouse (and the zoom) to give some kind of smoothness: // inertia var inertia = 0.92; var currentaddx = 0; var currentaddy = 0; var currentaddzoom = 0; function doinertia() { visucontrol.offsetx += currentaddx; visucontrol.offsety += currentaddy; visucontrol.zoom += currentaddzoom; var effectivetotalcardsinwidth = colscount * cardwidth; var rowscount = math.ceil(totalcards / colscount); var effectivetotalcardsinheight = rowscount * cardheight var maxoffsetx = effectivetotalcardsinwidth / 2.0; var maxoffsety = effectivetotalcardsinheight / 2.0; if (visucontrol.offsetx < -maxoffsetx + cardwidth) visucontrol.offsetx = -maxoffsetx + cardwidth; else if (visucontrol.offsetx > maxoffsetx) visucontrol.offsetx = maxoffsetx; if (visucontrol.offsety < -maxoffsety + cardheight) visucontrol.offsety = -maxoffsety + cardheight; else if (visucontrol.offsety > maxoffsety) visucontrol.offsety = maxoffsety; if (visucontrol.zoom < 0.05) visucontrol.zoom = 0.05; else if (visucontrol.zoom > 1) visucontrol.zoom = 1; processlistofcards(); currentaddx *= inertia; currentaddy *= inertia; currentaddzoom *= inertia; // epsilon if (math.abs(currentaddx) < 0.001) currentaddx = 0; if (math.abs(currentaddy) < 0.001) currentaddy = 0; } this kind of small function does not cost a lot to implement, but adds a lot to the quality of user experience. state storage also to provide a better user experience, we will save the display window’s position and zoom. to do this, we will use the service of localstorage (which saves pairs of keys / values for the long term (the data is retained after the browser is closed) and only accessible by the current window object): function saveconfig() { if (window.localstorage == undefined) return; // zoom window.localstorage["zoom"] = visucontrol.zoom; // offsets window.localstorage["offsetx"] = visucontrol.offsetx; window.localstorage["offsety"] = visucontrol.offsety; } // restore data if (window.localstorage != undefined) { var storedzoom = window.localstorage["zoom"]; if (storedzoom != undefined) visucontrol.zoom = parsefloat(storedzoom); var storedoffsetx = window.localstorage["offsetx"]; if (storedoffsetx != undefined) visucontrol.offsetx = parsefloat(storedoffsetx); var storedoffsety = window.localstorage["offsety"]; if (storedoffsety != undefined) visucontrol.offsety = parsefloat(storedoffsety); } animations to add even more dynamism to our application we will allow our users to double-click on a card to zoom and focus on it. our system should animate three values: the two offsets (x, y) and the zoom. to do this, we will use a function that will be responsible of animating a variable from a source value to a destination value with a given duration: var animationhelper = function (root, name) { var paramname = name; this.animate = function (current, to, duration) { var offset = (to - current); var ticks = math.floor(duration / 16); var offsetpart = offset / ticks; var tickscount = 0; var intervalid = setinterval(function () { current += offsetpart; root[paramname] = current; tickscount++; if (tickscount == ticks) { clearinterval(intervalid); root[paramname] = to; } }, 16); }; }; the use of this function is: // prepare animations parameters var zoomanimationhelper = new animationhelper(visucontrol, "zoom"); var offsetxanimationhelper = new animationhelper(visucontrol, "offsetx"); var offsetyanimationhelper = new animationhelper(visucontrol, "offsety"); var speed = 1.1 - visucontrol.zoom; zoomanimationhelper.animate(visucontrol.zoom, 1.0, 1000 * speed); offsetxanimationhelper.animate(visucontrol.offsetx, targetoffsetx, 1000 * speed); offsetyanimationhelper.animate(visucontrol.offsety, targetoffsety, 1000 * speed); the advantage of the animationhelper function is that it is able to animate as many parameters as you wish (and that only with the settimer function!) handling multi-devices finally we will ensure that our page can also be seen on tablets pc and even on phones. to do this, we will use a feature of css 3: the media-queries . with this technology, we can apply style sheets according to some queries such as a specific display size: here we see that if the screen width is less than 480 pixels, the following style sheet will be added: #legal { font-size: 8px; } #title { font-size: 30px !important; } #waittext { font-size: 12px; } #bolaslogo { width: 48px; height: 48px; } #picturecell { width: 48px; } finally we will ensure that our page can also be seen on tablets pc and even on phones. to do this, we will use a feature of css 3: #legal { font-size: 8px; } #title { font-size: 30px !important; } #waittext { font-size: 12px; } #bolaslogo { width: 48px; height: 48px; } #picturecell { width: 48px; } conclusion html 5 / css 3 / javascript and visual studio 2010 allow to develop portable and efficient solutions (within the limits of browsers that support html 5 of course) with some great features such as hardware accelerated rendering. this kind of development is also simplified by the use of frameworks like jquery. also, i am especially fan of javascript that turns out to be a very powerful dynamic language. of course, c# or vb.net developers have to change theirs reflexes but for the development of web pages it's worth. in conclusion, i think that the best to be convinced is to try! to go further internet explorer test drive: http://ie.microsoft.com/testdrive/ internet explorer 9 guide for developer : http://msdn.microsoft.com/en-us/ie/ff468705 w3c site for html 5 : http://dev.w3.org/html5/spec/overview.html internet explorer site : http://msdn.microsoft.com/en-us/ie/aa740469 about the author david catuhe is a developer evangelist for microsoft france in charge of user experience development tools (from xaml to directx/xna and html5). he defines himself as a geek and likes coding all that refer to graphics. before working for microsoft, he founded a company that developed a realtime 3d engine written with directx ( www.vertice.fr ). source: http://blogs.msdn.com/b/eternalcoding/archive/2011/07/25/feedback-of-a-graphic-development-using-html5-amp-javascript.aspx

After learning the basics of PHP's basic file system functions, the first thing you'll want to do is put it to use. One of the easiest and flashiest things you can create is a page counting script. I'll show you how to create a page hit script that is easy to create and even easier to implement. This is the entire script. Let's go through it line by line. $filename = "counter.txt"; This assigns the filename to a variable to be used throughout the rest of the script. In this case I used counter.txt. The first thing to do after implementing this script is to create counter.txt on your server in the same directory as this script and chmod is to 777 so that PHP may write to the file later on. Here is a great tutorial on how to chmod files: http://support.discusware.com/center/resources/howto/chmod.html $count = file_get_contents($filename); This line stores in entire contents of counter.txt into the variable $count. In this case there is only a number inside counter.txt so $count holds that number. if ($count == null) $count = 0; If the file is empty we need to make a case for that or problems will come up later. If $count does not contain anything than we go ahead and set $count to 0. echo $count; This will display the number inside the text file. $count++; This adds 1 to the current $count. $handle = fopen($filename, "w+"); The next thing to do is to open the file for writing and to truncate it (erase its contents), that's why I choose w+. Please see my article on file system basics for an explanation on other modes. fopen() requires that we assign a resource handle to be referenced in the future. fwrite($handle, $count); This line writes the variable $count to our file represented by $handle. fclose($handle); This is the final line of the script where we do some cleanup. This closes the file we opened earlier. It isn't essential to the script what its always nice to cover all bases. Now that you understand how the script works it's time to implement it. This assumes you name the script above counter.php. Wherever you put this in your page is where the script will echo the page hit count. It's as easy as that.

In the scenario of today, an if chooses to execute different behavior depending on the type of an object. We should define "type" very looosely; for example, it may be: the class of the object or one of the interfaces it implements (instanceof). The value of one of the object's fields (usually enumerative). The result of a query method, such as isXXX() or getTotalValue(). Each of these discriminants has the power to be used in dozens of identical conditionals throughout the codebase. Why eliminating this conditional? And by replacing it with what? We can replace easily this kind of conditionals with polymorphism: we move the relevant logic in the object whose type is queried. This object becomes an instance of a subclass of the original one: the different legs of the conditional become different subclasses. In case the subclasses already exist, it's even easier as it's just a movement of code at the method scale. Typically polymorphism allows to remove the conditional or to put it at the instantiation time of the new object. It's an expression of the Tell, Don't Ask principle, where instead of asking repeatedly (in different places of client code) the object what to do, you simply tell the object to do something and pass in some references if necessary. The end result is that the duplication of the conditional is eliminated as it is splitted into the various classes of the hierarchy. The addition or removal of a new type impacts just a subclass, which has to be created anew or thrown away. You don't have to grep all your application looking for if()s involving a field. Note that this refactoring works not only for ifs and elses, but also for selects. The missing step A prerequisite for this refactoring is having an existing inheritance structure where the dependency of client code is on the parent, but different instances of the child can be passed in. If there is not already a hierarchy, you have two choices as preliminary refactorings: inheritance: Replace Type code with Subclasses. The current class gains new children; it is a less invasive approach but also but less clear. Inheritance is a one shot strategy as you won't be able to use it for other axis of change. composition: Replace type code with State or Strategy. A new class is extracted, which gains the bew children. More flexible as you can extract many collaborators like this, and forces to name the new concept and its new hierarchy. The steps in the rest of the article presume you already have this hierarchy, so let these two refactorings guide you into extracting the classes (even if they come up empty initially). This refactoring then only focuses on breaking up the conditional code. The previous refactorings allow you to tackle the hierarchy and the instantiation code instead. Steps Extract a method containing the conditional. Move this method to the top of the inheritance hierarchy. Copy the original method into each subclass, and eliminate all machinery to leave just one leg. Some data of the original class may become protected instead of private. Remove the copied leg and repeat with the next subclass until the method in the superclass is empty and the overrides do not contain references to classes different from the current one. Example In the initial state of the example, the Renderer class has a kind of switch where it generates different HTML depending on the class of the object to render: Brand pages can have custom URLs, while user profile must follow the standard naming scheme and use their nick. assertEquals('giorgio', $renderer->__toString()); } public function testABrandPageShouldHaveACustomURL() { $renderer = new Renderer(new Brand('Coca Cola', 'coke')); $this->assertEquals('Coca Cola', $renderer->__toString()); } } class User { private $name; public function __construct($name) { $this->name = $name; } public function getName() { return $this->name; } } class Brand { private $name; private $url; public function __construct($name, $url) { $this->name = $name; $this->url = $url; } public function getName() { return $this->name; } public function getURL() { return $this->url; } } class Renderer { private $domainObject; public function __construct($domainObject) { $this->domainObject = $domainObject; } public function __toString() { if ($this->domainObject instanceof User) { return '' . $this->domainObject->getName() . ''; } if ($this->domainObject instanceof Brand) { return '' . $this->domainObject->getName() . ''; } } } There are two issues with this design: if we add a new domain object (e.g. Group, with its own page) we have to open up the hood and insert new code in an already existing class. It's generally easier to deal with a change by adding brand new classes, since we can't break existing code if we do not touch it. Not only Renderer contains these different cases, but probably many objects composing Brand and User. The first move is to isolate the variability into a single hierarchy of classes.Since there is no common ancestor, we create a common superclass in Brand and User. getURL() would be better named as getSlug() actually (an url-friendly label). abstract class Addressable { public function render($template) { if ($this instanceof User) { return sprintf($template, $this->getName(), $this->getName()); } if ($this instanceof Brand) { return sprintf($template, $this->getUrl(), $this->getName()); } } } We only hid the mess, not resolved it: the base class is still a single point that knows everything about the subclasses. At least there would be no duplication if someone wants to use name or URL in other HTML fragments. We copy the method into the various subclasses. In this case we augment duplication, but that's a temporary move since we will eliminate most of the code in the methods shortly. class User extends Addressable { private $name; public function __construct($name) { $this->name = $name; } public function getName() { return $this->name; } public function render($template) { if ($this instanceof User) { return sprintf($template, $this->getName(), $this->getName()); } if ($this instanceof Brand) { return sprintf($template, $this->getUrl(), $this->getName()); } } } class Brand extends Addressable { private $name; private $url; public function __construct($name, $url) { $this->name = $name; $this->url = $url; } public function getName() { return $this->name; } public function getURL() { return $this->url; } public function render($template) { if ($this instanceof User) { return sprintf($template, $this->getName(), $this->getName()); } if ($this instanceof Brand) { return sprintf($template, $this->getUrl(), $this->getName()); } } } In the next step, we make the original method abstract: since each concrete class has its own copy, it will never be executed. abstract class Addressable { public abstract function render($template); } We eliminate impossible cases: now dynamic dispatch is doing the work of choosing which method to execute (instead of a chain of ifs). Brand and User only refer to their own methods and do not know anything about each other's presence. abstract class Addressable { public abstract function render($template); } class User extends Addressable { private $name; public function __construct($name) { $this->name = $name; } public function getName() { return $this->name; } public function render($template) { return sprintf($template, $this->getName(), $this->getName()); } } class Brand extends Addressable { private $name; private $url; public function __construct($name, $url) { $this->name = $name; $this->url = $url; } public function getName() { return $this->name; } public function getURL() { return $this->url; } public function render($template) { return sprintf($template, $this->getUrl(), $this->getName()); } } The last step, although not strictly part of this refactoring, is to eliminate the getters since no one calls them from outside the class in this particular example. class User extends Addressable { private $name; public function __construct($name) { $this->name = $name; } public function render($template) { return sprintf($template, $this->name, $this->name); } } class Brand extends Addressable { private $name; private $url; public function __construct($name, $url) { $this->name = $name; $this->url = $url; } public function render($template) { return sprintf($template, $this->url, $this->name); } } We have now a solution where new classes can be added freely, and new Renderer can use all Addressable classes. It's a Bridge pattern, or just good factoring.

My previous blog post discussed the use of Neo4J as a RDF triple store. Michael Hunger however informed me that the neo-rdf-sail component is no longer under active development and advised me to have a look at Tinkerpop’s Sail implementation. As mentioned in my previous blog post, I recently got asked to implement a storage and querying platform for biological RDF (Resource Description Framework) data. Traditional RDF stores are not really an option as my solution should also provide the ability to calculate shortest paths between random subjects. Calculating shortest path is however one of the strong selling points of Graph Databases and more specifically Neo4J. Unfortunately, the neo-rdf-sail component, which suits my requirements perfectly, is no longer under active development. Tinkerpop’s Sail implementation however, fills the void with an even better alternative! 1. What is Tinkerpop? Tinkerpop is an open source project that provides an entire stack of technologies within the Graph Database space. At the core of this stack is the Blueprints framework. Blueprints can be considered as the JDBC of Graph Databases. By providing a collection of generic interfaces, it allows to develop graph-based applications, without introducing explicit dependencies on concrete Graph Database implementations. Additionally, Blueprints provides concrete bindings for the Neo4J, OrientDB and Dex Graph Databases. On top of Blueprints, the Tinkerpop team developed an entire range of graph technologies, including Gremlin, a powerful, domain-specific language designed for traversing graphs. Hence, once a Blueprints binding is available for a particular Graph Database, an entire range of technologies can be leveraged. 2. Tinkerpop and Sail Last time, I talked about exposing a Neo4J Graph Database (containing RDF triples) through the Sail interface, which is part of the openrdf.org project. By doing so, we can reuse an entire range of RDF utilities (parsers and query evaluators) that are part of the openrdf.org project. The Blueprints framework provides us with a similar ability: each Graph Database binding that implements the Tinkerpop TransactionalGraph and IndexableGraph interfaces can be exposed as a GraphSail, which is Tinkerpop’s implementation of the Sail interface. Once you have your Sail available, storing and querying RDF is analogous to the piece of code shown in my previous blog article. // Create the sail graph database graph = new MyNeo4jGraph("var/flights", 100000); graph.setTransactionMode(TransactionalGraph.Mode.MANUAL); sail = new GraphSail(graph); // Initialize the sail store sail.initialize(); // Get the sail repository connection connection = new SailRepository(sail).getConnection(); // Import the data connection.add(getResource("sneeair.rdf"), null, RDFFormat.RDFXML); // Execute SPARQL query TupleQuery durationquery = connection.prepareTupleQuery(QueryLanguage.SPARQL, "PREFIX io: " + "PREFIX fl: " + "SELECT ?number ?departure ?destination " + "WHERE { " + "?flight io:flight ?number . " + "?flight fl:flightFromCityName ?departure . " + "?flight fl:flightToCityName ?destination . " + "?flight io:duration \"1:35\" . " + "}"); TupleQueryResult result = durationquery.evaluate(); The two first lines of code require some more clarification. A TransactionalGraph can be run in MANUAL or AUTOMATIC transaction mode. In AUTOMATIC mode, transactions are basically ignored, in the sense that each item that gets created is immediately persisted in the underlying Graph Database. Although this fits my needs, AUTOMATIC mode is extremely slow in case of Neo4J because of the continuous IO access. MANUAL mode on the other hand is very fast; a new transaction is created at the moment the import of the RDF data file starts and is only committed to the Neo4J data store once all RDF triples are parsed and created. Unfortunately, MANUAL mode does not scale either in my specific situation; as some of my RDF data files contain over 50 million RDF triples, they can not fit into memory (i.e. Java heap space error). Requiring fast imports, I extended the default Neo4J Blueprints binding to support intermediate commits. I based my implementation on Neo4J’s best practices for big transactions. The idea is rather simple: you specify the maximum number of items that can be kept in memory, before they should be committed to the Neo4J data store. Once this number is reached, the current transaction is committed and a new one is automatically started. Simple, but very effective! public class MyNeo4jGraph extends Neo4jGraph { private long numberOfItems = 0; private long maxNumberOfItems = 1; public MyNeo4jGraph(final String directory, long maxNumberOfItems) { super(directory, null); this.maxNumberOfItems = maxNumberOfItems; } public MyNeo4jGraph(final String directory, final Map configuration, long maxNumberOfItems) { super(directory, configuration); this.maxNumberOfItems = maxNumberOfItems; } public Vertex addVertex(final Object id) { Vertex vertex = super.addVertex(id); commitIfRequired(); return vertex; } public Edge addEdge(final Object id, final Vertex outVertex, final Vertex inVertex, final String label) { Edge edge = super.addEdge(id, outVertex, inVertex, label); commitIfRequired(); return edge; } private void commitIfRequired() { // Check whether commit should be executed if (++numberOfItems % maxNumberOfItems == 0) { // Stop the transaction stopTransaction(Conclusion.SUCCESS); // Immediately start a new one startTransaction(); } } } 3. Shortest path calculation Although Blueprints allows you to abstract away the Neo4J implementation details, it still provides you with access to the raw Neo4J data store if needed. Hence, one can still use the graph algorithms provided in the neo4j-graph-algo component to calculate shortest paths between random subjects. The complete source code can be found on the Datablend public GitHub repository.

This tutorial will walk you throughout how to save and load an image from database (MySQL) using Hibernate. Requirements For this sampel project, we are going to use: Eclipse IDE (you can use your favorite IDE); MySQL (you can use any other database, make sure to change the column type if required); Hibernate jars and dependencies (you can download the sample project with all required jars); JUnit - for testing (jar also included in the sample project). PrintScreen When we finish implementing this sample projeto, it should look like this: Database Model Before we get started with the sample projet, we have to run this sql script into MySQL: DROP SCHEMA IF EXISTS `blog` ; CREATE SCHEMA IF NOT EXISTS `blog` DEFAULT CHARACTER SET latin1 COLLATE latin1_swedish_ci ; USE `blog` ; -- ----------------------------------------------------- -- Table `blog`.`BOOK` -- ----------------------------------------------------- DROP TABLE IF EXISTS `blog`.`BOOK` ; CREATE TABLE IF NOT EXISTS `blog`.`BOOK` ( `BOOK_ID` INT NOT NULL AUTO_INCREMENT , `BOOK_NAME` VARCHAR(45) NOT NULL , `BOOK_IMAGE` MEDIUMBLOB NOT NULL , PRIMARY KEY (`BOOK_ID`) ) ENGINE = InnoDB; This script will create a table BOOK, which we are going to use in this tutorial. Book POJO We are going to use a simple POJO in this project. A Book has an ID, a name and an image, which is represented by an array of bytes. As we are going to persist an image into the database, we have to use the BLOB type. MySQLhas some variations of BLOBs, you can check the difference between them here. In this example, we are going to use the Medium Blob, which can store L + 3 bytes, where L < 2^24. Make sure you do not forget to add the column definition on the Column annotation. package com.loiane.model; import javax.persistence.Column; import javax.persistence.Entity; import javax.persistence.GeneratedValue; import javax.persistence.Id; import javax.persistence.Lob; import javax.persistence.Table; @Entity @Table(name="BOOK") public class Book { @Id @GeneratedValue @Column(name="BOOK_ID") private long id; @Column(name="BOOK_NAME", nullable=false) private String name; @Lob @Column(name="BOOK_IMAGE", nullable=false, columnDefinition="mediumblob") private byte[] image; public long getId() { return id; } public void setId(long id) { this.id = id; } public String getName() { return name; } public void setName(String name) { this.name = name; } public byte[] getImage() { return image; } public void setImage(byte[] image) { this.image = image; } } Hibernate Config This configuration file contains the required info used to connect to the database. com.mysql.jdbc.Driver jdbc:mysql://localhost/blog root root org.hibernate.dialect.MySQLDialect 1 true Hibernate Util The HibernateUtil class helps in creating the SessionFactory from the Hibernate configuration file. package com.loiane.hibernate; import org.hibernate.SessionFactory; import org.hibernate.cfg.AnnotationConfiguration; import com.loiane.model.Book; public class HibernateUtil { private static final SessionFactory sessionFactory; static { try { sessionFactory = new AnnotationConfiguration() .configure() .addPackage("com.loiane.model") //the fully qualified package name .addAnnotatedClass(Book.class) .buildSessionFactory(); } catch (Throwable ex) { System.err.println("Initial SessionFactory creation failed." + ex); throw new ExceptionInInitializerError(ex); } } public static SessionFactory getSessionFactory() { return sessionFactory; } } DAO In this class, we created two methods: one to save a Book instance into the database and another one to load a Book instance from the database. package com.loiane.dao; import org.hibernate.HibernateException; import org.hibernate.Session; import org.hibernate.Transaction; import com.loiane.hibernate.HibernateUtil; import com.loiane.model.Book; public class BookDAOImpl { /** * Inserts a row in the BOOK table. * Do not need to pass the id, it will be generated. * @param book * @return an instance of the object Book */ public Book saveBook(Book book) { Session session = HibernateUtil.getSessionFactory().openSession(); Transaction transaction = null; try { transaction = session.beginTransaction(); session.save(book); transaction.commit(); } catch (HibernateException e) { transaction.rollback(); e.printStackTrace(); } finally { session.close(); } return book; } /** * Delete a book from database * @param bookId id of the book to be retrieved */ public Book getBook(Long bookId) { Session session = HibernateUtil.getSessionFactory().openSession(); try { Book book = (Book) session.get(Book.class, bookId); return book; } catch (HibernateException e) { e.printStackTrace(); } finally { session.close(); } return null; } } Test To test it, first we need to create a Book instance and set an image to the image attribute. To do so, we need to load an image from the hard drive, and we are going to use the one located in the images folder. Then we can call the DAO class and save into the database. Then we can try to load the image. Just to make sure it is the same image we loaded, we are going to save it in the hard drive. package com.loiane.test; import static org.junit.Assert.assertNotNull; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; import com.loiane.dao.BookDAOImpl; import com.loiane.model.Book; public class TestBookDAO { private static BookDAOImpl bookDAO; @BeforeClass public static void runBeforeClass() { bookDAO = new BookDAOImpl(); } @AfterClass public static void runAfterClass() { bookDAO = null; } /** * Test method for {@link com.loiane.dao.BookDAOImpl#saveBook()}. */ @Test public void testSaveBook() { //File file = new File("images\\extjsfirstlook.jpg"); //windows File file = new File("images/extjsfirstlook.jpg"); byte[] bFile = new byte[(int) file.length()]; try { FileInputStream fileInputStream = new FileInputStream(file); fileInputStream.read(bFile); fileInputStream.close(); } catch (Exception e) { e.printStackTrace(); } Book book = new Book(); book.setName("Ext JS 4 First Look"); book.setImage(bFile); bookDAO.saveBook(book); assertNotNull(book.getId()); } /** * Test method for {@link com.loiane.dao.BookDAOImpl#getBook()}. */ @Test public void testGetBook() { Book book = bookDAO.getBook((long) 1); assertNotNull(book); try{ //FileOutputStream fos = new FileOutputStream("images\\output.jpg"); //windows FileOutputStream fos = new FileOutputStream("images/output.jpg"); fos.write(book.getImage()); fos.close(); }catch(Exception e){ e.printStackTrace(); } } } To verify if it was really saved, let’s check the table Book: and if we right click… and choose to see the image we just saved, we will see it: Source Code Download You can download the complete source code (or fork/clone the project – git) from: Github: https://github.com/loiane/hibernate-image-example BitBucket: https://bitbucket.org/loiane/hibernate-image-example/downloads Happy Coding! From http://loianegroner.com/2011/10/how-to-load-or-save-image-using-hibernate-mysql/

Version 2.3 of the Java servlet spec introduced the concept of filters. According to the documentation from Oracle’s site: “A filter dynamically intercepts requests and responses to transform or use the information contained in the requests or responses”. Today I’ll show you how to build a simple filter to intercept the response HTTP response code using annotations introduced in the Servlet 3.0 specification. With NetBeans IDE 7 create a new Maven Java Web Application called: Intercept Delete the index.jsp file under the Web Pages folder. Right-click on the project and add a new servlet called: MainServlet Since we are using the new Servlet 3 annotations we don’t need to set a whole lot of properties. Maven generates a decent MainServlet.java file for us, I just removed the comments for the output. My file looks like this: package com.giantflyingsaucer.intercept; import java.io.IOException; import java.io.PrintWriter; import javax.servlet.ServletException; import javax.servlet.annotation.WebServlet; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; @WebServlet(name = "MainServlet", urlPatterns = {"/"}) public class MainServlet extends HttpServlet { protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setContentType("text/html;charset=UTF-8"); PrintWriter out = response.getWriter(); try { out.println(""); out.println(""); out.println(""); out.println(""); out.println(""); out.println("Servlet MainServlet"); out.println(""); out.println(""); } finally { out.close(); } } // /** * Handles the HTTP GET method. * @param request servlet request * @param response servlet response * @throws ServletException if a servlet-specific error occurs * @throws IOException if an I/O error occurs */ @Override protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { processRequest(request, response); } /** * Handles the HTTP POST method. * @param request servlet request * @param response servlet response * @throws ServletException if a servlet-specific error occurs * @throws IOException if an I/O error occurs */ @Override protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { processRequest(request, response); } /** * Returns a short description of the servlet. * @return a String containing servlet description */ @Override public String getServletInfo() { return "Short description"; }// } Right-click on the project and add a Filter called: InterceptFilter We will add the following two lines to the doFilter method. HttpServletResponse hsr = (HttpServletResponse) response; System.out.println("HTTP Status: " + hsr.getStatus()); My doFilter method looks like this: @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { if (debug) { log("InterceptFilter:doFilter()"); } doBeforeProcessing(request, response); HttpServletResponse hsr = (HttpServletResponse) response; System.out.println("HTTP Status: " + hsr.getStatus()); Throwable problem = null; try { chain.doFilter(request, response); } catch (Throwable t) { problem = t; t.printStackTrace(); } doAfterProcessing(request, response); if (problem != null) { if (problem instanceof ServletException) { throw (ServletException) problem; } if (problem instanceof IOException) { throw (IOException) problem; } sendProcessingError(problem, response); } } Clean and Build the project and deploy it to Apache Tomcat. Access the URL with a browser and take a look at your catalina.out file and you should see the HTTP response code. Note: You shouldn’t need to do any changes to the web.xml file for this project to work. From http://www.giantflyingsaucer.com/blog/?p=3279

Import allows one XML schema to reference elements and types from another XML schema.