Tests developed during TDD can be classified into several levels, depending on the size of the object graph they need to work with. End-to-end tests span the whole application graph, while unit tests usually target a single public class at a time. In the middle we find functional tests, which exercise a group of objects. A recurring problem is that of nearby classes C creeping into unit tests of unrelated classes; this situation transform what would be a unit test of the original class O into a functional tests of O and C together (possibly with multiple C classes involved). Functional tests are handy for specifying behavior at an higher level of abstraction than that of a single object, and sometimes for checking the wiring of a component of the application. However, if they are introduced involuntarily in place of unit tests they are prone to raise maintenance problems, since they will need to change every time the C class is updated. Moreover, they will fail along with the unit test of C, pointing to a problem into either O or C, which are not able to localize immediately. Consider this test, where the original class is DocumentsDeclarationNodeCommand and the collaborating one is InMemoryDocumentCopy: @Test public void shouldSendTheListOfDocumentsAndWaitForAcknowledgement() throws ConnectionClosedException { UpstreamConnection upstream = mock(UpstreamConnection.class); DownstreamConnection downstream = mock(DownstreamConnection.class); InMemoryDocumentCopy first = new InMemoryDocumentCopy("1.txt", "hello"); InMemoryDocumentCopy second = new InMemoryDocumentCopy("2.txt", "hello2"); DocumentsDeclarationNodeCommand command = DocumentsDeclarationNodeCommand.fromDocumentCopies( Arrays.asList(first, second), 10001); command.execute(upstream, downstream); InOrder inOrder = inOrder(upstream, downstream); inOrder.verify(upstream).command("DOCUMENTS|PORT=10001"); inOrder.verify(upstream).command("1.txt|5"); inOrder.verify(upstream).command("2.txt|6"); inOrder.verify(upstream).endCommandSection(); inOrder.verify(downstream, times(1)).readResponse(); } The two expectations on command() make this test a functional one: a change in the textual serialization format of InMemoryDocumentCopy (such as "1.txt|sha1_hash|5") will break this checks, even if DocumentsDeclarationNodeCommand still works. Yet we cannot avoid to verify that the documents are really sent to the server by this object. Functional tests can be transformed again into unit tests by testing O with a Test Double instead of C (a Stub, or a Mock.) The only remaining dependency will be the one of the interface of C, which can be even extracted into an independent entity (a first-class interface in language that support them such as Java, C# and PHP.) Pure functions What happens when you can't easily inject a Test Double to maintain the tests at the unit level? This issues exists in functional languages where functions call a tree of other functions. A analogue approach to dependency injection is to inject the function as a parameter, but doesn't probably scale to the level of injection we perform on objects: every function signature would have to receive all the collaborating ones as additional parameters. There are even mocking frameworks for functional languages like Marick's one which are able to isolate a function from its collaborators. Uncle Bob uses the Derived Expectation pattern instead: testing "update-all" (let [ o1 (make-object ...) o2 (make-object ...) o3 (make-object ...) os [o1 o2 o3] us (update-all os) ] (is (= (nth us 0) (reposition (accelerate (accumulate-forces os o1) (is (= (nth us 1) (reposition (accelerate (accumulate-forces os o2) (is (= (nth us 2) (reposition (accelerate (accumulate-forces os o3) ) ) The update-all function calls internally reposition, accelerate and accumulate-forces (or it calls other functions which in turn call these three). Instead of specifying unreadable literal expectations in the tests like (1.096, 4.128), this approach let the test specify update-all link to the other functions without introducing magic numbers. It is therefore a unit test for update-all, while the same test containing numbers would be a functional test. Note that this approach is safe for functional languages because the collaborating functions have no state, being pure; you can call reposition and accelerate how many times you want, and their result won't change. This is not necessarily true for collaborators in object-oriented languages: in principle, a method can return a different value for each call. Tests with derived expectations As long as the composed methods do not change their result, this approach would build real unit tests, whose success does not depend on the correctness of classes other than the one under test. Apart from corner cases like the composed methods throwing exceptions, a change in the collaborator's behavior would change only the collaborator's test. Value Objects are the ideal collaborator to stub out with derived expectations: they are immutable, so their methods always return the same result. Their code is usually self-contained and simple, so it's difficult for a method to throw an exception or to break internally once the Value Object has been correctly built. Being simple, final classes they do not implement an explicit interface; and they are not commonly substituted by Test Doubles. Their behavior is mixed in with the objects using them. The test becomes: @Test public void shouldSendTheListOfDocumentsAndWaitForAcknowledgement() throws ConnectionClosedException { UpstreamConnection upstream = mock(UpstreamConnection.class); DownstreamConnection downstream = mock(DownstreamConnection.class); InMemoryDocumentCopy first = new InMemoryDocumentCopy("1.txt", "hello"); InMemoryDocumentCopy second = new InMemoryDocumentCopy("2.txt", "hello2"); DocumentsDeclarationNodeCommand command = DocumentsDeclarationNodeCommand.fromDocumentCopies( Arrays.asList(first, second), 10001); command.execute(upstream, downstream); InOrder inOrder = inOrder(upstream, downstream); inOrder.verify(upstream).command("DOCUMENTS|PORT=10001"); inOrder.verify(upstream).command(first.toString()); inOrder.verify(upstream).command(second.toString()); inOrder.verify(upstream).endCommandSection(); inOrder.verify(downstream, times(1)).readResponse(); } Conclusion We saw that Test Doubles like Mocks and Stubs are not the only way to achieve isolated tests, which fail only where the class under test fail and not when a collaborator changes its implementation. In the Example, DocumentsDeclarationNodeCommand is tested by involving the real collaborator, but setting up Derived Expectation from it instead of literal ones. The result is this test is only tied to the method signatures of the collaborator instead of to the real behavior (the output format of toString()). This technique doesn't need to be used often: its purpose is to isolate from an immutable object, without introducing a Test Double.

In this tutorial we are making a HTML5 Slideshow (canvas) with its own animated transitioning effect. The main idea is: drawing images with higher contrast in the ends of transitions. Here are our demo and downloadable packages: Live Demo download in package Ok, download the source files and let's start coding ! Step 1. HTML This is the markup of our resulting slideshow page. index.html HTML5 Canvas Slideshow Back to original tutorial on Script Tutorials Step 2. CSS Here are all the stylesheets: css/main.css /* page layout styles */ *{ margin:0; padding:0; } body { background-color:#eee; color:#fff; font:14px/1.3 Arial,sans-serif; } header { background-color:#212121; box-shadow: 0 -1px 2px #111111; display:block; height:70px; position:relative; width:100%; z-index:100; } header h2{ font-size:22px; font-weight:normal; left:50%; margin-left:-400px; padding:22px 0; position:absolute; width:540px; } header a.stuts,a.stuts:visited{ border:none; text-decoration:none; color:#fcfcfc; font-size:14px; left:50%; line-height:31px; margin:23px 0 0 110px; position:absolute; top:0; } header .stuts span { font-size:22px; font-weight:bold; margin-left:5px; } .container { color: #000; margin: 20px auto; position: relative; width: 900px; } #slideshow { border:1px #000 solid; box-shadow:4px 6px 6px #444444; display:block; margin:0 auto; height:300px; width:900px; } .container .slides { display:none; } Step 3. JS js/pixastic.custom.js Pixastic – JavaScript Image Processing Library. I have used it to change the brightness and contrast of our canvas. You can download this library here. Or, you can find this library in our package too. js/script.js var canvas, ctx; var aImages = []; var iCurSlide = 0; var iCnt = 0; var iSmTimer = 0; var iContr = 0; var iEfIter = 50; $(function(){ // creating canvas objects canvas = document.getElementById('slideshow'); ctx = canvas.getContext('2d'); // collect all images $('.slides').children().each(function(i){ var oImg = new Image(); oImg.src = this.src; aImages.push(oImg); }); // draw first image ctx.drawImage(aImages[iCurSlide], 0, 0); var iTimer = setInterval(changeSlideTimer, 5000); // set inner timer }); function changeSlideTimer() { iCurSlide++; if (iCurSlide == $(aImages).length) { iCurSlide = 0; } clearInterval(iSmTimer); iSmTimer = setInterval(drawSwEffect, 40); // extra one timer } // draw switching effect function drawSwEffect() { iCnt++; if (iCnt <= iEfIter / 2) { iContr += 0.004; // change brightness and contrast Pixastic.process(canvas, 'brightness', { 'brightness': 2, 'contrast': 0.0 + iContr, 'leaveDOM': true }, function(img) { ctx.drawImage(img, 0, 0); } ); } if (iCnt > iEfIter / 2) { // change brightness Pixastic.process(canvas, 'brightness', { 'brightness': -2, 'contrast': 0, 'leaveDOM': true }, function(img) { ctx.drawImage(img, 0, 0); } ); } if (iCnt == iEfIter / 2) { // switch actual image iContr = 0; ctx.drawImage(aImages[iCurSlide], 0, 0); Pixastic.process(canvas, 'brightness', { 'brightness': iEfIter, 'contrast': 0, 'leaveDOM': true }, function(img) { ctx.drawImage(img, 0, 0); } ); } else if (iCnt == iEfIter) { // end of cycle, clear extra sub timer clearInterval(iSmTimer); iCnt = 0; iContr = 0; } } As you can see – the main functionality is easy. I have defined the main timer (which will change images), and one inner timer, which will change the brightness and contrast of our canvas. Live Demo download in package Conclusion I hope that today’s html5 slideshow lesson was interesting for you as always. We have made another nice html5 example. I will be glad to see your thanks and comments. Good luck! Source: http://www.script-tutorials.com/html5-canvas-slideshow/

In this post we will use a Monte Carlo method to approximate pi. The idea behind the method that we are going to see is the following: Draw the unit square and the unit circle. Consider only the part of the circle inside the square and pick uniformly a large number of points at random over the square. Now, the unit circle has pi/4 the area of the square. So, it should be apparent that of the total number of points that hit within the square, the number of points that hit the circle quadrant is proportional to the area of that part. This gives a way to approximate pi/4 as the ratio between the number of points inside circle and the total number of points and multiplying it by 4 we have pi. Let's see the python script that implements the method discussed above using the numpy's indexing facilities: from pylab import plot,show,axis from numpy import random,sqrt,pi # scattering n points over the unit square n = 1000000 p = random.rand(n,2) # counting the points inside the unit circle idx = sqrt(p[:,0]**2+p[:,1]**2) < 1 plot(p[idx,0],p[idx,1],'b.') # point inside plot(p[idx==False,0],p[idx==False,1],'r.') # point outside axis([-0.1,1.1,-0.1,1.1]) show() # estimation of pi print '%0.16f' % (sum(idx).astype('double')/n*4),'result' print '%0.16f' % pi,'real pi' The program will print the pi approximation on the standard out: 3.1457199999999998 result 3.1415926535897931 real pi and will show a graph with the generated points: Note that the lines of code used to estimate pi are just 3! Source: http://glowingpython.blogspot.com/2012/01/monte-carlo-estimate-for-pi-with-numpy.html

I love Sonar. It is a wonderful way to collect some metrics for your Java projects - hassle-free and wrapped in a sweet-looking UI. For Maven-based projects Sonar literally works out of the box. Just start up your Sonar instance (assuming you are using the default settings running on localhost) and then you simply fire it off using: $ mvn sonar:sonar A few moments later you should have the metrics available at: http://localhost:9000/ Well, the past few days I was setting up a multi-module Gradle project for Sonar. Let me start by stating that Gradle is awesome. Having the ability to declare dependencies as one-liners and also being able to customize your scripts easily, yet having sensible defaults, is very nice. Kind of the best of both worlds. Setting up Sonar for a multi module project, though, is unfortunately a bit more complicated, compared to what I am used to in the Maven world. It is not awfully complicated, but it took me a while to collect all the pieces of information. Getting your Sonar plugin to just doing something is fairly simple. Just follow the basic steps outlined in the plugin documentation at: http://gradle.org/docs/current/userguide/sonar_plugin.html One differentiator between the Sonar Plugin for Gradle and Maven is, that the Gradle version does not automatically run code coverage analysis. This needs to be manually setup. This is where the official doc just vaguely refers to Cobertura. First, I tried using Cobertura for code coverage, but I seemed to run into difficulties for my multi-module projects. The Cobertura Plugin is here: https://github.com/valkolovos/gradle_cobertura/wiki By chance I realized that the Sonar guys have a GitHub repository with samples on how to setup sonars for various build systems, including Gradle: https://github.com/SonarSource/sonar-examples/ In their examples, they are using JaCoCo, which is not mentioned in the original Gradle docs and maybe I could have continued with Cobertura but it seemed that Sonar was preferring JaCoCo and thus I continued with that. Some Gradle Sonar Plugin Limitations The Gradle Sonar Plugin has an annoying limitation, where I can run it for the ROOT project OR for the sub-projects individually. See the following Gradle Jira ticket for details: http://issues.gradle.org/browse/GRADLE-1813 Furthermore, I hit the minor issue that I cannot set the links in the Sonar dashboard. This seems to be related to the following Sonar Jira issue: https://jira.codehaus.org/browse/SONAR-2749 The JaCoCo code coverage plugin is "slightly less" supported by the Gradle Sonar Plugin, e.g. the Gradle Plugin does not have an explicit setter for the JacocoReportPath and it assumes the "target" folder as the build directory by default. Therefore you must set explicitly: props["sonar.jacoco.reportPath"] = "${buildDirName}/jacoco.exec" Lastly, I deviated a bit from the SonarSource Gradle example, and instead of System properties, I wanted to use Gradle properties to allow for users to provide non-default Sonar configuration settings (databasem url, jdbc parameters etc.). Well, while setting that up I ran into yet another Gradle Jira issue: http://issues.gradle.org/browse/GRADLE-1826 But at the the end, I am happily able to run a multi-module Gradle project with Sonar and collecting Code Coverage statistics. Here is the relavant code from my build.gradle file: apply plugin: 'sonar' sonar { if (rootProject.hasProperty('sonarHostUrl')) { server.url = rootProject.sonarHostUrl } database { if (rootProject.hasProperty('sonarJdbcUrl')) { url = rootProject.sonarJdbcUrl } if (rootProject.hasProperty('sonarJdbcDriver')) { driverClassName = rootProject.sonarJdbcDriver } if (rootProject.hasProperty('sonarJdbcUsername')) { username = rootProject.sonarJdbcUsername } if (rootProject.hasProperty('sonarJdbcPassword')) { password = rootProject.sonarJdbcPassword } } project { dynamicAnalysis = "reuseReports" withProjectProperties { props -> props["sonar.core.codeCoveragePlugin"] = "jacoco" props["sonar.jacoco.reportPath"] = "${buildDirName}/jacoco.exec" } } println("Sonar parameters used: server.url='${server.url}'; database.url='${database.url}'; database.driverClassName='${database.driverClassName}'; database.username='${database.username}'") } subprojects { subproject -> ... // See http://www.gradle.org/docs/current/userguide/dependency_management.html#sub:configurations // and http://www.gradle.org/docs/current/dsl/org.gradle.api.artifacts.ConfigurationContainer.html configurations { jacoco //Configuration Group used by Sonar to provide Code Coverage using JaCoCo } // dependencies that are common across all java projects dependencies { ... jacoco group: "org.jacoco", name: "org.jacoco.agent", version: "0.5.3.201107060350", classifier: "runtime" ... } test { jvmArgs "-javaagent:${configurations.jacoco.asPath}=destfile=${buildDir}/jacoco.exec,includes=org.your.project.*" } ... } I hope this is useful information for all Gradle users out there. From http://hillert.blogspot.com/2012/01/sonar-and-gradle-multi-module-projects.html

The dependency:tree goal of the Maven plugin dependency supports various graphical outputs from the version 2.4 up. This is how you would create a diagram showing all dependencies in the com.example group in the dot format: mvn dependency:tree -Dincludes=com.example-DappendOutput=true -DoutputType=dot -DappendOutput=true -DoutputFile=/path/to/output.dot To actually produce an image from .dot you can use one of .dot renderers, f.ex. this online dot renderer (paste into the right text box, press enter). You could also generate the output f.ex. in the graphml format & visualize it in Eclipse. From http://theholyjava.wordpress.com/2012/01/13/visualize-maven-project-dependencies-with-dependencytree-and-dot-diagram-output/

Two variants of run-length encoding are the diagram encoding and the pattern substitution algorithms. The diagram encoding is actually a very simple algorithm. Unlike run-length encoding, where the input stream must consists of many repeating elements, “aaaaaaaa” for instance, which are very rare in a natural language, there are many so-called “diagrams” in almost any natural language. In plain English there are some diagrams such as “the”, “and”, “ing” (in the word “waiting” for example), “ a”, “ t”, “ e” and many doubled letters. Actually we can extend those diagrams by adding surrounding spaces. Thus we can encode not only “the”, but “ the “, which are 5 characters (2 spaces and 3 letters) with something shorter. On the other hand, as I said, in plain English there are too many doubled letters, which unfortunately aren’t something special for run-length encoding and the compression ratio will be small. Even worse the encoded text may happen to be longer than the input message. Let’s see some examples. Let’s say we’ve to encode the message “successfully accomplished”, which consists of four doubled letters. However to compress it with run-length encoding we’ll need at least 8 characters, which doesn’t help us a lot. // 8 chars replaced by 8 chars!? input: "successfully accomplished" output: "su2ce2sfu2ly a2complished" The problem is that if the input text contains numbers, “2” in particular, we’ve to chose an escape symbol (“@” for example), which we’ll use to mark where the encoded run begins. Thus if the input message is “2 successfully accomplished tasks”, it will be encoded as “2 su@2ce@2sfu@2ly a@2complished tasks”. Now the output message is longer!!! than the input string. // the compressed message is longer!!! input: "2 successfully accomplished" output: "2 su@2ce@2sfu@2ly a@2complished tasks" Again if the input stream contains the escape symbol, we have to find another one, and the problem is that it is often too difficult to find short escape symbol that doesn’t appear in the input text, without a full scan of the text. That is why run-length encoding isn’t a good solution when compressing plain text, where long runs rarely appear. Well, of course, there are exceptions. For example such an exception is the lossy text compression with run-length encoding. It is intuitively clear that compressing text with loss is rarely useful, especially when you’ve to decompress exactly the same text. However there are some cases that lossy compression may be useful. Such case can be removing spaces. Indeed the text “successfully accomplished” brings us exactly the same information as “successfully accomplished”. In this case we can simply remove those spaces. Indeed we can use a marker to indicate the long run of spaces like “successfully@6 accomplished” in order to decompress the input string with absolutely no loss, but we can also throw those symbols away. This desision depends on the goal. Exactly with the same goal in mind we can remove new lines and tabs, only if we’re sure that the sense of the text is preserved. Yet again, a problem is that such long runs don’t happen to occur in random texts. That is why it’s better to use diagram encoding for plain text compression instead of run-length encoding. A Few Questions After understanding the principles of the diagram encoding, let’s see some examples. In the example above it is better to replace doubled letters with something shorter. Let’s say # for “cc”, @ for “ss” and % for “ll”. Thus the input text will be compressed as “su#e@fu%y a#omplished”, which is shorter. But yet again what will happen if the input message contains one of the substitutions? Also we can’t say if there are many doubled letters and enough reasonable substitutions for them. A better approach is to replace patterns. Run-length encoding isn't a good approach for text compression, because long runs rarely appear in a natural language. Pattern Substitution The pattern substitution algorithm is a variant of the diagram encoding. As I said above in plain English a very commonly used pattern can be “ the “, which is five characters long. We can now replace it with something like “$%” for example. In this case the message “I send the message” will become “I send$%message”. However there are some obstacles to overcome. The first problem is that we need to know the language and somehow to define commonly used patterns in a dictionary. What would happen with a message written in some language we don’t know nothing about. Let’s say – Latin like the example bellow. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Cras venenatis, sapien eget suscipit placerat, justo quam blandit mauris, quis tempor ante sapien sodales augue. Praesent ut mauris quam. Phasellus scelerisque, ante quis consequat tristique, metus turpis consectetur leo, vitae facilisis sapien mi eu sapien. Praesent vitae ligula elit, et faucibus augue. Sed rhoncus sodales dolor ut gravida. In quis augue ac nulla auctor mattis sed sed libero. Donec eget purus eget enim tempor porta vitae eget diam. Mauris aliquet malesuada ipsum, non pulvinar urna vestibulum ac. Donec feugiat velit vitae nunc cursus imperdiet. Donec accumsan faucibus dictum. Phasellus sed mauris sapien. Maecenas mi metus, tincidunt sed rhoncus nec, sodales non sapien. Clearly without knowing Latin it isn’t easy to define which are those commonly used patterns. The thing is that it’s better to use pattern substitution if you know in advance the set of words and characters. The second problem is related to decompression. It is obvious that we need to define a dictionary and this dictionary must be used when decoding the message. It will be great also if we find more patterns longer than three characters. If not, the compression ratio will be low. Unfortunately such patterns aren’t very common in any natural language. Diagram encoding and pattern substitution are far more suitable for text compression than run-length encoding. In fact, pattern substitution is very effective on compressing programming languages. Application It is interesting to answer the question, how to use diagram encoding or patter substitution to compress text in natural language, especially when we don’t know the language in detail? The answer hides in the question. We wont compress natural languages, but machine language. Exactly machine (programming) languages are limited to a smaller sets of words and symbols. Isn’t it true for any programing language? Like PHP, where words like “function”, “while”, “for”, “break”, “switch”, “foreach” happen to be often in use, or HTML with its defined set of tags. Perhaps the best example is CSS, where only the values of the properties can vary. CSS files also tend to have multiple new lines, tabs and spaces, which only humans read. The question here is why should we compress those file types. It’s clear that after the compression they will be completely useless, both for humans and machines. Yes, that is true, but what if we have to store versions of those files into a DB. Kind of a backup. Imagine you’re working for a web hosting company that has to store daily versions of the sites it’s hosting. Thus the volume of stored information even for small companies hosting only few sites can be enormous. The problem is that compressing those files with some conventional compressing tool isn’t a good idea. Thus we’ve to save a copy of the entire site every day, but as we know the difference between daily versions of a site can be small. A version control system is another solution, but then you’ve to store the plain text of the files. Perhaps a better approach is to compress the text using pattern substitution and then saving only differences – kind of version control, which can be done with “relative encoding”. Using the above method we can save lots of disk space and in the same time we can compress/decompress easily. Another good thing is that you can save only changes to the initial files, like version control, which can also be compressed. Implementation The implementation of this algorithm is again on PHP and tries only to describe the main principles of compression. In this case I tried to compress a CSS file using the compression above. Although this example is quite primitive we can see some interesting facts. First of all you only need encoding and decoding dictionaries. Practically the encoding and decoding processes are equal, so you don’t need to implement two different functions. Here in this example a native PHP function is used – str_replace, because the purpose of this algorithm is not to describe pattern substitution techniques, but pattern substitution. It assumes that today’s programming languages have string manipulation functions for the purposes of this task. $str = file_get_contents('large_style_file.css'); $encoding_dict = array( "\n" => '$0', 'text' => '$1', 'color' => '$2', 'display' => '$3', 'font' => '$4', 'width' => '$5', 'height' => '$6', ' ' => '', ); function replace_patterns($input, $dict) { foreach ($dict as $pattern => $replace) { $input = str_replace($pattern, $replace, $input); } return $input; } $result = replace_patterns($str, $encoding_dict); By only replacing few CSS properties I achieved almost 40% of compression ratio (as shown the diagram bellow). The initial file is 202 KB, while compressed it’s only 131 KB. Of course, it all depends on the CSS file, but how about replacing all property names with shorter ones. Perhaps then the compression will be even better. Source: http://www.stoimen.com/blog/2012/01/23/computer-algorithms-data-compression-with-diagram-encoding-and-pattern-substitution/

Not too long ago I tweeted what I felt was a small triumph on my latest project, streaming files from MongoDB GridFS for downloads (rather than pulling the whole file into memory and then serving it up). I promised to blog about this but unfortunately my specific usage was a little coupled to the domain on my project so I couldn’t just show it off as is. So I’ve put together an example node.js+GridFS application and shared it on github and will use this post to explain how I accomplished it. GridFS Module First off, special props go to tjholowaychuk who responded in the #node.js irc channel when I asked if anyone has had luck with using GridFS from mongoose. A lot of my resulting code is derived from an gist he shared with me. Anyway, to the code. I’ll describe how I’m using gridfs and after setting the ground work illustrate how simple it is to stream files from GridFS. I created a gridfs module that basically accesses GridStore through mongoose (which I use throughout my application) that can also share the db connection created when connecting mongoose to the mongodb server. mongoose = require "mongoose" request = require "request" GridStore = mongoose.mongo.GridStore Grid = mongoose.mongo.Grid ObjectID = mongoose.mongo.BSONPure.ObjectID We can’t get files from mongodb if we cannot put anything into it, so let’s create a putFile operation. exports.putFile = (path, name, options..., fn) -> db = mongoose.connection.db options = parse(options) options.metadata.filename = name new GridStore(db, name, "w", options).open (err, file) -> return fn(err) if err file.writeFile path, fn parse = (options) -> opts = {} if options.length > 0 opts = options[0] if !opts.metadata opts.metadata = {} opts This really just delegates to the putFile operation that exists in GridStore as part of the mongodb module. I also have a little logic in place to parse options, providing defaults if none were provided. One interesting feature to note is that I store the filename in the metadata because at the time I ran into a funny issue where files retrieved from gridFS had the id as the filename (even though a look in mongo reveals that the filename is in fact in the database). Now the get operation. The original implementation of this simply passed the contents as a buffer to the provided callback by calling store.readBuffer(), but this is now changed to pass the resulting store object to the callback. The value in this is that the caller can use the store object to access metadata, contentType, and other details. The user can also determine how they want to read the file (either into memory or using a ReadableStream). exports.get = (id, fn) -> db = mongoose.connection.db id = new ObjectID(id) store = new GridStore(db, id, "r", root: "fs" ) store.open (err, store) -> return fn(err) if err # band-aid if "#{store.filename}" == "#{store.fileId}" and store.metadata and store.metadata.filename store.filename = store.metadata.filename fn null, store This code just has a small blight in that it checks to see if the filename and fileId are equal. If they are, it then checks to see if metadata.filename is set and sets store.filename to the value found there. I’ve tabled the issue to investigate further later. The Model In my specific instance, I wanted to attach files to a model. In this example, let’s pretend that we have an Application for something (job, a loan application, etc) that we can attach any number of files to. Think of tax receipts, a completed application, other scanned documents. ApplicationSchema = new mongoose.Schema( name: String files: [ mongoose.Schema.Mixed ] ) ApplicationSchema.methods.addFile = (file, options, fn) -> gridfs.putFile file.path, file.filename, options, (err, result) => @files.push result @save fn Here I define files as an array of Mixed object types (meaning they can be anything) and a method addFile which basically takes an object that at least contains a path and filename attribute. It uses this to save the file to gridfs and stores the resulting gridstore file object in the files array (this contains stuff like an id, uploadDate, contentType, name, size, etc). Handling Requests This all plugs in to the request handler to handle form submissions to /new. All this entails is creating an Application model instance, adding the uploaded file from the request (in this case we named the file field “file”, hence req.files.file) and saving it. app.post "/new", (req, res) -> application = new Application() application.name = req.body.name opts = content_type: req.files.file.type application.addFile req.files.file, opts, (err, result) -> res.redirect "/" Now the sum of all this work allows us to reap the rewards by making it super simple to download a requested file from gridFS. app.get "/file/:id", (req, res) -> gridfs.get req.params.id, (err, file) -> res.header "Content-Type", file.type res.header "Content-Disposition", "attachment; filename=#{file.filename}" file.stream(true).pipe(res) Here we simply look up a file by id and use the resulting file object to set Content-Type and Content-Disposition fields and finally make use of ReadableStream::pipe to write the file out to the response object (which is an instance of WritableStream). This is the piece of magic that streams data from MongoDB to the client side. Ideas This is just a humble beginning. Other ideas include completely encapsulating gridfs within the model. Taking things further we could even turn the gridfs model into a mongoose plugin to allow completely blackboxed usage of gridfs. Feel free to check the project out and let me know if you have ideas to take it even further. Fork away! Source: http://blog.james-carr.org/2012/01/09/streaming-files-from-mongodb-gridfs/

Software Engineering is all about reuse. We programmers therefore love to split applications up into smaller components so that each of them can be reused or extended in an independent manner. A keyword here is "loose coupling". Slightly simplified, this means, each component should have as few dependencies to other components as possible. Most important, if I have a component B which relies on component A, I don't want that A needs to know about B. The component A should just provide a clean interface which could be used and extended by B. In Java there are many frameworks which provide this exact functionality: JavaEE, Spring, OSGI. However, each of those frameworks come with their own way to do things and provide lots and lots of additional functionality - whether you want it or not! Since we here at scireum love modularity (we build 4 products out of a set of about 10 independet modules) we built our own little framework. I factored out the most important parts and now have a single class with less than 250 lines of code+comments! I call this a microkernel approach, since it nicely compares to the situation we have with operating systems: There are monolithic kernels like the one of Linux with about 11,430,712 lines of code. And there is a concept called a microkernel, like to one of Minix with about 6,000 lines of executable kernel code. There is still an ongoing discussion which of the two solituons is better. A monolithic kernel is faster, a microkernel has way less critical code (critical code means: a bug there will crash the complete system. If you haven't already, you should read more about mikrokernels on Wikipedia. However one might think about operating systems - when it comes to Java I prefer less dependencies and if possible no black magic I don't understand. Especially if this magic involves complex ClassLoader structures. Therefore, here comes Nucleus... How does this work? The framework (Nucleus) solves two problems of modular applications: I want provide a service to other components - but I only want to show an interface and they should be provided with my implementation at runtime without knowning(referencing) it. I want to provide a service or callback for other components. I provide an interface, and I want to know all classes implementig it, so I can invoke them. Ok, we probably need examples for this. Say we want to implement a simple timer service. It provides an interface: public interface EveryMinute { void runTimer() throws Exception; } All classes implementing this interface should be invoked every minute. Additionally we provide some infos - namely, when was the timer executed last. public interface TimerInfo { String getLastOneMinuteExecution(); } Ok, next we need a client for our services: @Register(classes = EveryMinute.class) public class ExampleNucleus implements EveryMinute { private static Part timerInfo = Part.of(TimerInfo.class); public static void main(String[] args) throws Exception { Nucleus.init(); while (true) { Thread.sleep(10000); System.out.println("Last invocation: " + timerInfo.get().getLastOneMinuteExecution()); } } @Override public void runTimer() throws Exception { System.out.println("The time is: " + DateFormat.getTimeInstance().format(new Date())); } } The static field "Part timerInfo" is a simple helper class which fetches the registered instance from Nucleus on the first call and loads it into a private field. So accessing this part has almost no overhead to a normal field access - yet we only reference an interface, not an implementation. The main method first initializes Nucleus (this performs the classpath scan etc.) and then simply goes into an infinite loop, printing the last execution of our timer every ten seconds. Since our class wears a @Register annotation, it will be discovered by a special ClassLoadAction (not by Nucleus itself) instantiated and registered for the EveryMinute interface. Its method runTimer will then be invoced by our timer service every minute. Ok, but how would our TimerService look like? @Register(classes = { TimerInfo.class }) public class TimerService implements TimerInfo { @InjectList(EveryMinute.class) private List everyMinute; private long lastOneMinuteExecution = 0; private Timer timer; public TimerService() { start(); } public void start() { timer = new Timer(true); // Schedule the task to wait 60 seconds and then invoke // every 60 seconds. timer.schedule(new InnerTimerTask(), 1000 * 60, 1000 * 60); } private class InnerTimerTask extends TimerTask { @Override public void run() { // Iterate over all instances registered for // EveryMinute and invoke its runTimer method. for (EveryMinute task : everyMinute) { task.runTimer(); } // Update lastOneMinuteExecution lastOneMinuteExecution = System.currentTimeMillis(); } } @Override public String getLastOneMinuteExecution() { if (lastOneMinuteExecution == 0) { return "-"; } return DateFormat.getDateTimeInstance().format( new Date(lastOneMinuteExecution)); } } This class also wears a @Register annotation so that it will also be loaded by the ClassLoadAction named above (the ServiceLoadAction actually). As above it will be instantiated and put into Nucleus (as implementation of TimerInfo). Additionally it wears an @InjectList annotation on the everyMinute field. This will be processed by another class named Factory which performs simple dependency injection. Since its constructur starts a Java Timer for the InnerTimerTask, from that point on all instances registered for EveryMinute will be invoced by this timer - as the name says - every minute. How is it implemented? The good thing about Nucleus is, that it is powerful on the one hand, but very simple and small on the other hand. As you could see, there is no inner part for special or privileged services. Everything is built around the kernel - the class Nuclues. Here is what it does: It scans the classpath and looks for files called "component.properties". Those need to be in the root folder of a JAR or in the /src folder of each Eclipse project respectively. For each identified JAR / project / classpath element, it then collects all contained class files and loads them using Class.forName. For each class, it checks if it implements ClassLoadAction, if yes, it is put into a special list. Each ClassLoadAction is instanciated and each previously seen class is sent to it using: void handle(Class clazz) Finally each ClassLoadAction is notified, that nucleus is complete so that final steps (like annotation based dependency injection) could be performed. That's it. The only other thing Nucleus provides is a registry which can be used to register and retrieve objects for a class. (An in-depth description of the process above, can be found here: http://andreas.haufler.info/2012/01/iterating-over-all-classes-with.html). Now to make this framework useable as shown above, there is a set of classes around Nucleus. Most important is the class ServiceLoadAction, which will instantiate each class which wears a @Register annoation, runs Factory.inject (our mini DI tool) on it, and throws it into Nucleus for the listed classes. Whats important: The ServiceLoadActions has no specific rights or privileges, you can easily write your implementation which does smarter stuff. Next to some annotations, there are three other handy classes when it comes to retrieving instances from Nucleus: Factory, Part and Parts. As noted above, the Factory is a simple dependency injector. Currently only the ServiceLoadAction autmatically uses the Factory, as all classes wearing the @Register annotation are scanned for required injections. You can however use this factory to run injections on your own classes or other ClassLoadActions to do the same as ServiceLoadAction. If you can't or don't want to rely in annotation based dependency magic, you can use the two helper classes Part and Parts. Those are used like normal fields (see ExampleNucleus.timerInfo above) and fetch the appropriate object or list of objects automatically. Since the result is cached, repeated invocations have almost no overhead compared to a normal field. Nucleus and the example shown above is open source (MIT-License) and available here: https://github.com/andyHa/scireumOpen/blob/master/src/examples/ExampleNucleus.java https://github.com/andyHa/scireumOpen/tree/master/src/com/scireum/open/nucleus If you're interested in using Nucleus, I could put the relevant souces into a separater repository and also provide a release jar - just write a comment below an let me know. This post is the fourth part of the my series "Enterprisy Java" - We share our hints and tricks how to overcome the obstacles when trying to build several multi tenant web applications out of a set of common modules.

To test mobile versions of websites, it is useful to be able to connect to a web server on your local machine from a web browser on an Android emulator without having to expose the web server to the Internet. You can’t use the normal loop-back IP address of 127.0.0.1 because that refers to the emulated Android device itself. Instead you have to use 10.0.2.2 to connect to the host machine. That’s fine if your local web server is serving a single site, but if you are using name-based virtual hosting to serve different sites depending on the host name of the request (with aliases for localhost defined in your machine’s hosts file), then you need to be making requests from the browser using the correct host name, not the IP address. The Android emulator does not use the host machine’s hosts file for name resolution so attempting to access http://myvirtualhost in the emulator’s browser will not work. This is because the emulated Android device has it’s own hosts file, so you have to update this to map the virtual host names to the local machine. The first step is to start the AVD with an increased partition size otherwise you may get an out of memory error when you try to save the modified hosts file: emulator -avd MyAVD -partition-size 128 You then have to remount the system partition so that it is writeable: adb remount Then copy the hosts file from the emulated device to the host machine: adb pull /etc/hosts Edit the hosts file so that it includes mappings for all relevant virtual host names: 127.0.0.1 localhost 10.0.2.2 myvirtualhost1 myvirtualhost2 Then copy the updated file back to the emulated device: adb push hosts /etc/hosts You should then be able to visit http://myvirtualhost1 in the emulator’s browser and see the correct site. From http://blog.uncommons.org/2012/01/12/accessing-local-name-based-virtual-hosts-from-the-android-emulator/

If you remember from my previous blogs, I’m covering different approaches to parsing XML messages using the outrageously corny scenario of Pete’s Perfect Pizza, the pizza company with big ideas. In this story, you are an employee of Pete’s and have been asked to implement a system for sending orders from the front desk to the kitchen and you came up with the idea of using XML. You’ve just got your SAX Parser working, but Pete’s going global, opening kitchens around the world taking orders using the Internet. But, hang on a minute... didn’t I say this in my last blog? Déjà vu? Today’s blog is the alternative reality version of my JAXB blog as the scenario remains the same, but the solution changes. Instead of demonstrating JAXB, I’ll be investigating XMLBeans. So, Pete’s hired some consultants who’ve come up with a plan for extending your cosy XML message and they’ve specified it using a schema. They’ve also enhanced your message by adding in one of there own customer schemas. The result is that the following XSD files land in your inbox and you need to get busy... A wrapper around the customer and the pizza order This is a list of pizzas ordered by the customer The type of pizza on the menu type of base quantity of pizzas Plain and Simple Garlic Pizza... Ham and Musheroom with an egg thin base traditional Thick base The date is in the Common Era (minus sign in years is not permitted) The time zone although not included UTC is implied Generic Customer Definition The Customer's first name The Customer's surname The house number A line of an address You realise that with this level of complexity, you’ll be messing around with SAX for a long time, and you could also make a few mistakes. There must be a better way right? After-all XML has been around for some time, so there most be a few frameworks around that could be useful. After a bit more Googling you come across XMLBeans and realise that there are... XMLBeans uses a special compiler to convert an XML schema into a bunch of related Java classes that define the types required to access the XML elements, attributes and other content in a type-safe way. This blog isn’t a tutorial covering the ins and outs of XMLBeans, that can be found here, from Apache, except to say the the key idea for parsing, or unmarshalling, XML is that you compile your Java classes using XMLBeans and then use those classes in your application. In using any XML schema to Java class compiler, the neatest approach is to put all your schemas and the compiler in a separate JAR file. You can mix them in with your application’s source code, but that usually clouds the code base making maintenance more difficult. In creating a XMLBeans JAR file, you may come up with a POM file that looks something like this: 4.0.0 com.captaindebug xml-tips-xmlbeans jar 1.0-SNAPSHOT XML Beans for Pete's Perfect Pizza org.apache.xmlbeans xmlbeans 2.4.0 org.codehaus.mojo xmlbeans-maven-plugin xmlbeans true src/main/resources org.apache.maven.plugins maven-compiler-plugin 2.3.2 1.6 1.6 ...which is very straight forward. So, getting back to Pete’s Perfect Pizza, you’ve created your XMLBeans JAR file and all that’s left to do is to explore how it works, as demonstrated in the JUnit tests below: public class PizzaXmlBeansTest { private PizzaOrderDocument instance; @Test public void testLoadPizzaOrderXml() throws IOException, XmlException { String xml = loadResource("/pizza-order1.xml"); instance = PizzaOrderDocument.Factory.parse(xml); PizzaOrder order = instance.getPizzaOrder(); String orderId = order.getOrderID(); assertEquals("123w3454r5", orderId); // Check the customer details... CustomerType customerType = order.getCustomer(); NameType nameType = customerType.getName(); String firstName = nameType.getFirstName(); assertEquals("John", firstName); String lastName = nameType.getLastName(); assertEquals("Miggins", lastName); AddressType address = customerType.getAddress(); assertEquals(new BigInteger("15"), address.getHouseNumber()); assertEquals("Credability Street", address.getStreet()); assertEquals("Any Town", address.getTown()); assertEquals("Any Where", address.getArea()); assertEquals("AW12 3WS", address.getPostCode()); Pizzas pizzas = order.getPizzas(); PizzaType[] pizzasOrdered = pizzas.getPizzaArray(); assertEquals(3, pizzasOrdered.length); // Check the pizza order... for (PizzaType pizza : pizzasOrdered) { PizzaNameType.Enum pizzaName = pizza.getName(); if ((PizzaNameType.CAPRICCIOSA == pizzaName) || (PizzaNameType.MARINARA == pizzaName)) { assertEquals(BaseType.THICK, pizza.getBase()); assertEquals(new BigInteger("1"), pizza.getQuantity()); } else if (PizzaNameType.PROSCIUTTO_E_FUNGHI == pizzaName) { assertEquals(BaseType.THIN, pizza.getBase()); assertEquals(new BigInteger("2"), pizza.getQuantity()); } else { fail("Whoops, can't find pizza type"); } } } private String loadResource(String filename) throws IOException { InputStream is = getClass().getResourceAsStream(filename); if (is == null) { throw new IOException("Can't find the file: " + filename); } return toString(is); } private String toString(InputStream is) throws IOException { ByteArrayOutputStream bos = new ByteArrayOutputStream(); copyStreams(is, bos); return bos.toString(); } private void copyStreams(InputStream is, OutputStream os) throws IOException { byte[] buf = new byte[1024]; int c; while ((c = is.read(buf, 0, 1024)) != -1) { os.write(buf, 0, c); os.flush(); } } } The code above may look long and complex, but it really only comprises of three steps: firstly, turn the test file into a suitable type such as a String or InputStream (XMLBeans can handle several different input types). Then use the nested Factory class to process your XML source turning into a document object. Finally, use the returned document object to test that your results are what you'd expected them to be (this is by far the largest step). Once you’re happy with the largely boilerplate usage of XMLBeans, you add it into your Pete's Perfect Pizza kitchen XML parser code and distribute it around the world to Pete’s many pizza kitchens. One of the strong points of using a framework like XMLBeans is that if there’s ever a change to the schema, then all that’s required to incorporate those changes is to recompile, fixing up your client code accordingly. This may seem a bit of a headache, but it’s a much smaller headache than trying re-work a SAX parser. On the downside, XMLBeans has been criticised for being slow, but I’ve never had too many problems. It will theoretically use more memory that SAX - this may or may not be true - it does build sets of classes, but then again so do some SAX ContentHandler derived classes. Finally, it should be noted that there hasn’t been a release of XMLBeans since 2009, which may or may not be a good thing depending upon your viewpoint, though I should emphasized that this code certainly isn’t redundant as, to my certain knowledge, it is still widely used on a number of large scale projects. The source code is available from GitHub at: git://github.com/roghughe/captaindebug.git From http://www.captaindebug.com/2012/01/approaches-to-xml-part-4-xmlbeans.html

package singz.test; import java.util.Date; import java.util.GregorianCalendar; import javax.xml.datatype.DatatypeConfigurationException; import javax.xml.datatype.DatatypeFactory; import javax.xml.datatype.XMLGregorianCalendar; /** * A utility class for converting objects between java.util.Date and * XMLGregorianCalendar types * */ public class XMLGregorianCalendarConversionUtil { // DatatypeFactory creates new javax.xml.datatype Objects that map XML // to/from Java Objects. private static DatatypeFactory df = null; static { try { df = DatatypeFactory.newInstance(); } catch(DatatypeConfigurationException e) { throw new IllegalStateException( "Error while trying to obtain a new instance of DatatypeFactory", e); } } // Converts a java.util.Date into an instance of XMLGregorianCalendar public static XMLGregorianCalendar asXMLGregorianCalendar(java.util.Date date) { if(date == null) { return null; } else { GregorianCalendar gc = new GregorianCalendar(); gc.setTimeInMillis(date.getTime()); return df.newXMLGregorianCalendar(gc); } } // Converts an XMLGregorianCalendar to an instance of java.util.Date public static java.util.Date asDate(XMLGregorianCalendar xmlGC) { if(xmlGC == null) { return null; } else { return xmlGC.toGregorianCalendar().getTime(); } } public static void main(String[] args) { Date currentDate = new Date(); // Current date // java.util.Date to XMLGregorianCalendar XMLGregorianCalendar xmlGC = XMLGregorianCalendarConversionUtil.asXMLGregorianCalendar( currentDate); System.out.println( "Current date in XMLGregorianCalendar format: " + xmlGC.toString()); // XMLGregorianCalendar to java.util.Date System.out.println( "Current date in java.util.Date format: " + XMLGregorianCalendarConversionUtil.asDate(xmlGC).toString()); } } Why do we need XMLGregorianCalendar? Java Architecture for XML Binding (JAXB) allows Java developers to map Java classes to XML representations. JAXB provides two main features: the ability to marshal Java objects into XML and the inverse, i.e. to unmarshal XML back into Java objects. In the default data type bindings i.e. mappings of XML Schema (XSD) data types to Java data types in JAXB, the following types in XML schema (mostly used in web services definition) – xsd:dateTime, xsd:time, xsd:date and so on map to javax.xml.datatype.XMLGregorianCalendar Java type. From http://singztechmusings.in/datatype-conversion-in-java-xmlgregoriancalendar-to-java-util-date-java-util-date-to-xmlgregoriancalendar/

The goal of object detection is to find an object of a pre-defined class in an image. In this post we will see how to use the Haar Classifier implemented in OpenCV in order to detect faces and eyes in a single image. (Note: this article is part of a series (,2) on object detection with OpenCV in Python. --Ed.) We are going to use two trained classifiers stored in two XML files: haarcascade_frontalface_default.xml - that you can find in the directory /data/haarcascades/ of your OpenCV installation haarcascade_eye.xml - that you can download from this website. The first one is able to detect faces and the second one eyes. To use a trained classifier stored in a XML file we need to load it into memory using the function cv.Load() and call the function cv.HaarDetectObjects() to detect the objects. Let's see the snippet: imcolor = cv.LoadImage('detectionimg.jpg') # input image # loading the classifiers haarFace = cv.Load('haarcascade_frontalface_default.xml') haarEyes = cv.Load('haarcascade_eye.xml') # running the classifiers storage = cv.CreateMemStorage() detectedFace = cv.HaarDetectObjects(imcolor, haarFace, storage) detectedEyes = cv.HaarDetectObjects(imcolor, haarEyes, storage) # draw a green rectangle where the face is detected if detectedFace: for face in detectedFace: cv.Rectangle(imcolor,(face[0][0],face[0][1]), (face[0][0]+face[0][2],face[0][1]+face[0][3]), cv.RGB(155, 255, 25),2) # draw a purple rectangle where the eye is detected if detectedEyes: for face in detectedEyes: cv.Rectangle(imcolor,(face[0][0],face[0][1]), (face[0][0]+face[0][2],face[0][1]+face[0][3]), cv.RGB(155, 55, 200),2) cv.NamedWindow('Face Detection', cv.CV_WINDOW_AUTOSIZE) cv.ShowImage('Face Detection', imcolor) cv.WaitKey() These images are produced running the script with two different inputs. The first one is obtained from an image that contains two faces and four eyes: And the second one is obtained from an image that contains one face and two eyes (the shakira.jpg we used in the post about PCA):

This is the forth of a series of articles about Persistence with Spring. This article will focus on the configuration and implementation of the persistence layer with Spring 3.1, JPA and Spring Data. For a step by step introduction about setting up the Spring context using Java based configuration and the basic Maven pom for the project, see this article. The Persistence with Spring series: Part 1 – The Persistence Layer with Spring 3.1 and Hibernate Part 3 – The Persistence Layer with Spring 3.1 and JPA Part 5 – Transaction configuration with JPA and Spring 3.1 No More DAO implementations As I discussed in a previous post, the DAO layer usually consists of a lot of boilerplate code that can and should be simplified. The advantages of such a simplification are many fold: a decrease in the number of artifacts that need to be defined and maintained, simplification and consistency of data access patterns and consistency of configuration. Spring Data takes this simplification one step forward and makes it possible to remove the DAO implementations entirely – the interface of the DAO is now the only artifact that need to be explicitly defined. The Spring Data managed DAO In order to start leveraging the Spring Data programming model with JPA, a DAO interface needs to extend the JPA specific Repository interface - JpaRepository – in Spring’s interface hierarchy. This will enable Spring Data to find this interface and automatically create an implementation for it. Also, by extending the interface we get most if not all relevant CRUD generic methods for standard data access available in the DAO. Defining custom access method and queries As discussed, by implementing one of the Repository interfaces, the DAO will already have some basic CRUD methods (and queries) defined and implemented. To define more specific access methods, Spring JPA supports quite a few options – you can either simply define a new method in the interface, or you can provide the actual JPQ query by using the @Query annotation. A third option to define custom queries is to make use of JPA Named Queries, but this has the disadvantage that it either involves XML or burdening the domain class with the queries. In addition to these, Spring Data introduces a more flexible and convenient API, similar to the JPA Criteria API, only more readable and reusable. The advantages of this API will become more pronounced when dealing with a large number of fixed queries that could potentially be more concisely expressed through a smaller number of reusable blocks that keep occurring in different combinations. Automatic Custom Queries When Spring Data creates a new Repository implementation, it analyzes all the methods defined by the interfaces and tries to automatically generate queries from the method name. While this has limitations, it is a very powerful and elegant way of defining new custom access methods with very little effort. For example, if the managed entity has a name field (and the Java Bean standard getter and setter for that field), defining the findByName method in the DAO interface will automatically generate the correct query: public interface IFooDAO extends JpaRepository< Foo, Long >{ Foo findByName( final String name ); } This is a relatively simple example; a much larger set of keywords is supported by query creation mechanism. In the case that the parser cannot match the property with the domain object field, the following exception is thrown: java.lang.IllegalArgumentException: No property nam found for type class org.rest.model.Foo Manual Custom Queries In addition to deriving the query from the method name, a custom query can be manually specified with the method level @Query annotation. For even more fine grained control over the creation of queries, such as using named parameters or modifying existing queries, the reference is a good place to start. Spring Data transaction configuration The actual implementation of the Spring Data managed DAO – SimpleJpaRepository – uses annotations to define and configure transactions. A read only @Transactional annotation is used at the class level, which is then overridden for the non read-only methods. The rest of the transaction semantics are default, but these can be easily overridden manually per method. Exception Translation without the template One of the responsibilities of Spring ORM templates (JpaTemplate, HibernateTemplate) is exception translation – translating JPA exceptions – which tie the API to JPA – to Spring’s DataAccessException hierarchy. Without the template to do that, exception translation can still be enabled by annotating the DAOs with the @Repository annotation. That, coupled with a Spring bean postprocessor will advice all @Repository beans with all the implementations of PersistenceExceptionTranslator found in the Container – to provide exception translation without using the template. The fact that exception translation is indeed active can easily be verified with an integration test: @Test( expected = DataAccessException.class ) public void whenAUniqueConstraintIsBroken_thenSpringSpecificExceptionIsThrown(){ String name = "randomName"; this.service.save( new Foo( name ) ); this.service.save( new Foo( name ) ); } Exception translation is done through proxies; in order for Spring to be able to create proxies around the DAO classes, these must not be declared final. Spring Data Configuration To activate the Spring JPA repository support, the jpa namespace is defined and used to specify the package where to DAO interfaces are located: At this point, there is no equivalent Java based configuration – support for it is however in the works. The Spring Java or XML configuration The JPA configuration with Spring 3.1 has already been carefully discussed in the previous article of this series. Spring Data also takes advantage of the Spring support for the JPA @PersistenceContext annotation which it uses to wire the EntityManager into the Spring factory bean responsible with creating the actual DAO implementations – JpaRepositoryFactoryBean. In addition to the already discussed configuration, there is one last missing piece – including the Spring Data XML configuration in the overall persistence configuration: @Configuration @EnableTransactionManagement @ImportResource( "classpath*:*springDataConfig.xml" ) public class PersistenceJPAConfig{ ... } The Maven configuration In addition to the Maven configuration for JPA defined in a previous article, the spring-data-jpa dependency is addeed: org.springframework.data spring-data-jpa 1.0.2.RELEASE Conclusion This article covered the configuration and implementation of the persistence layer with Spring 3.1, JPA 2 and Spring JPA (part of the Spring Data umbrella project), using both XML and Java based configuration. The various method of defining more advanced custom queries are discussed, as well as configuration with the new jpa namespace and transactional semantics. The final result is a new and elegant take on data access with Spring, with almost no actual implementation work. You can check out the full implementation in the github project. From the originalThe Persistence Layer with Spring Data JPA of the Persistence with Spring series

Data exchanges between companies are increasing a lot. The number of applications that must be integrated is increasing, too. The interfaces use different technologies, protocols and data formats. Nevertheless, the integration of these applications must be modeled in a standardized way, realized efficiently and supported by automatic tests. Three integration frameworks are available in the JVM environment, which fulfil these requirements: Spring Integration, Mule ESB and Apache Camel. They implement the well-known Enteprise Integration Patterns (EIP, http://www.eaipatterns.com) and therefore offer a standardized, domain-specific language to integrate applications. These integration frameworks can be used in almost every integration project within the JVM environment – no matter which technologies, transport protocols or data formats are used. All integration projects can be realized in a consistent way without redundant boilerplate code. This article compares all three alternatives and discusses their pros and cons. If you want to know, when to use a more powerful Enterprise Service Bus (ESB) instead of one of these lightweight integration frameworks, then you should read this blog post: http://www.kai-waehner.de/blog/2011/06/02/when-to-use-apache-camel/ (it explains when to use Apache Camel, but the title could also be „When to use a lightweight integration framework“). Comparison Criteria Several criteria can be used to compare these three integration frameworks: Open source Basic concepts / architecture Testability Deployment Popularity Commercial support IDE-Support Errorhandling Monitoring Enterprise readiness Domain specific language (DSL) Number of components for interfaces, technologies and protocols Expandability Similarities All three frameworks have many similarities. Therefore, many of the above comparison criteria are even! All implement the EIPs and offer a consistent model and messaging architecture to integrate several technologies. No matter which technologies you have to use, you always do it the same way, i.e. same syntax, same API, same automatic tests. The only difference is the the configuration of each endpoint (e.g. JMS needs a queue name while JDBC needs a database connection url). IMO, this is the most significant feature. Each framework uses different names, but the idea is the same. For instance, „Camel routes“ are equivalent to „Mule flows“, „Camel components“ are called „adapters“ in Spring Integration. Besides, several other similarities exists, which differ from heavyweight ESBs. You just have to add some libraries to your classpath. Therefore, you can use each framework everywhere in the JVM environment. No matter if your project is a Java SE standalone application, or if you want to deploy it to a web container (e.g. Tomcat), JEE application server (e.g. Glassfish), OSGi container or even to the cloud. Just add the libraries, do some simple configuration, and you are done. Then you can start implementing your integration stuff (routing, transformation, and so on). All three frameworks are open source and offer familiar, public features such as source code, forums, mailing lists, issue tracking and voting for new features. Good communities write documentation, blogs and tutorials (IMO Apache Camel has the most noticeable community). Only the number of released books could be better for all three. Commercial support is available via different vendors: Spring Integration: SpringSource (http://www.springsource.com) Mule ESB: MuleSoft (http://www.mulesoft.org) Apache Camel: FuseSource (http://fusesource.com) and Talend (http://www.talend.com) IDE support is very good, even visual designers are available for all three alternatives to model integration problems (and let them generate the code). Each of the frameworks is enterprise ready, because all offer required features such as error handling, automatic testing, transactions, multithreading, scalability and monitoring. Differences If you know one of these frameworks, you can learn the others very easily due to their same concepts and many other similarities. Next, let’s discuss their differences to be able to decide when to use which one. The two most important differences are the number of supported technologies and the used DSL(s). Thus, I will concentrate especially on these two criteria in the following. I will use code snippets implementing the well-known EIP „Content-based Router“ in all examples. Judge for yourself, which one you prefer. Spring Integration Spring Integration is based on the well-known Spring project and extends the programming model with integration support. You can use Spring features such as dependency injection, transactions or security as you do in other Spring projects. Spring Integration is awesome, if you already have got a Spring project and need to add some integration stuff. It is almost no effort to learn Spring Integration if you know Spring itself. Nevertheless, Spring Integration only offers very rudimenary support for technologies – just „basic stuff“ such as File, FTP, JMS, TCP, HTTP or Web Services. Mule and Apache Camel offer many, many further components! Integrations are implemented by writing a lot of XML code (without a real DSL), as you can see in the following code snippet: You can also use Java code and annotations for some stuff, but in the end, you need a lot of XML. Honestly, I do not like too much XML declaration. It is fine for configuration (such as JMS connection factories), but not for complex integration logic. At least, it should be a DSL with better readability, but more complex Spring Integration examples are really tough to read. Besides, the visual designer for Eclipse (called integration graph) is ok, but not as good and intuitive as its competitors. Therefore, I would only use Spring Integration if I already have got an existing Spring project and must just add some integration logic requiring only „basic technologies“ such as File, FTP, JMS or JDBC. Mule ESB Mule ESB is – as the name suggests – a full ESB including several additional features instead of just an integration framework (you can compare it to Apache ServiceMix which is an ESB based on Apache Camel). Nevertheless, Mule can be use as lightweight integration framework, too – by just not adding and using any additional features besides the EIP integration stuff. As Spring Integration, Mule only offers a XML DSL. At least, it is much easier to read than Spring Integration, in my opinion. Mule Studio offers a very good and intuitive visual designer. Compare the following code snippet to the Spring integration code from above. It is more like a DSL than Spring Integration. This matters if the integration logic is more complex. The major advantage of Mule is some very interesting connectors to important proprietary interfaces such as SAP, Tibco Rendevous, Oracle Siebel CRM, Paypal or IBM’s CICS Transaction Gateway. If your integration project requires some of these connectors, then I would probably choose Mule! A disadvantage for some projects might be that Mule says no to OSGi: http://blogs.mulesoft.org/osgi-no-thanks/ Apache Camel Apache Camel is almost identical to Mule. It offers many, many components (even more than Mule) for almost every technology you could think of. If there is no component available, you can create your own component very easily starting with a Maven archetype! If you are a Spring guy: Camel has awesome Spring integration, too. As the other two, it offers a XML DSL: ${in.header.type} is ‘com.kw.DvdOrder’ ${in.header.type} is ‘com.kw.VideogameOrder’ Readability is better than Spring Integration and almost identical to Mule. Besides, a very good (but commercial) visual designer called Fuse IDE is available by FuseSource – generating XML DSL code. Nevertheless, it is a lot of XML, no matter if you use a visual designer or just your xml editor. Personally, I do not like this. Therefore, let’s show you another awesome feature: Apache Camel also offers DSLs for Java, Groovy and Scala. You do not have to write so much ugly XML. Personally, I prefer using one of these fluent DSLs instead XML for integration logic. I only do configuration stuff such as JMS connection factories or JDBC properties using XML. Here you can see the same example using a Java DSL code snippet: from(“file:incomingOrders “) .choice() .when(body().isInstanceOf(com.kw.DvdOrder.class)) .to(“file:incoming/dvdOrders”) .when(body().isInstanceOf(com.kw.VideogameOrder.class)) .to(“jms:videogameOrdersQueue “) .otherwise() .to(“mock:OtherOrders “); The fluent programming DSLs are very easy to read (even in more complex examples). Besides, these programming DSLs have better IDE support than XML (code completion, refactoring, etc.). Due to these awesome fluent DSLs, I would always use Apache Camel, if I do not need some of Mule’s excellent connectors to proprietary products. Due to its very good integration to Spring, I would even prefer Apache Camel to Spring Integration in most use cases. By the way: Talend offers a visual designer generating Java DSL code, but it generates a lot of boilerplate code and does not allow vice-versa editing (i.e. you cannot edit the generated code). This is a no-go criteria and has to be fixed soon (hopefully)! And the winner is… … all three integration frameworks, because they are all lightweight and easy to use – even for complex integration projects. It is awesome to integrate several different technologies by always using the same syntax and concepts – including very good testing support. My personal favorite is Apache Camel due to its awesome Java, Groovy and Scala DSLs, combined with many supported technologies. I would only use Mule if I need some of its unique connectors to proprietary products. I would only use Spring Integration in an existing Spring project and if I only need to integrate „basic technologies“ such as FTP or JMS. Nevertheless: No matter which of these lightweight integration frameworks you choose, you will have much fun realizing complex integration projects easily with low efforts. Remember: Often, a fat ESB has too much functionality, and therefore too much, unnecessary complexity and efforts. Use the right tool for the right job! Best regards, Kai Wähner (Twitter: @KaiWaehner) http://www.kai-waehner.de/blog/2012/01/10/spoilt-for-choice-which-integration-framework-to-use-spring-integration-mule-esb-or-apache-camel/

Memory Management in the Java HotSpot Virtual Machine View more documents from white paper Serial vs Parallel With serial collection, only one thing happens at a time. For example, even when multiple CPUs are available, only one is utilized to perform the collection. When parallel collection is used, the task of garbage collection is split into parts and those subparts are executed simultaneously, on different CPUs. The simultaneous operation enables the collection to be done more quickly, at the expense of some additional complexity and potential fragmentation. Concurrent versus Stop-the-world When stop-the-world garbage collection is performed, execution of the application is completely suspended during the collection. Alternatively, one or more garbage collection tasks can be executed concurrently, that is, simultaneously, with the application. Typically, a concurrent garbage collector does most of its work concurrently, but may also occasionally have to do a few short stop-the-world pauses. Stop-the-world garbage collection is simpler than concurrent collection, since the heap is frozen and objects are not changing during the collection. Its disadvantage is that it may be undesirable for some applications to be paused. Correspondingly, the pause times are shorter when garbage collection is done concurrently, but the collector must take extra care, as it is operating over objects that might be updated at the same time by the application. This adds some overhead to concurrent collectors that affects performance and requires a larger heap size. Compacting versus Non-compacting versus Copying After a garbage collector has determined which objects in memory are live and which are garbage, it can compact the memory, moving all the live objects together and completely reclaiming the remaining memory. After compaction, it is easy and fast to allocate a new object at the first free location. A simple pointer can be utilized to keep track of the next location available for object allocation. In contrast with a compacting collector, a non-compacting collector releases the space utilized by garbage objects in-place, i.e., it does not move all live objects to create a large reclaimed region in the same way a compacting collector does. The benefit is faster completion of garbage collection, but the drawback is potential fragmentation. In general, it is more expensive to allocate from a heap with in-place deallocation than from a compacted heap. It may be necessary to search the heap for a contiguous area of memory sufficiently large to accommodate the new object. A third alternative is a copying collector, which copies (or evacuates) live objects to a different memory area. The benefit is that the source area can then be considered empty and available for fast and easy subsequent allocations, but the drawback is the additional time required for copying and the extra space that may be required. Performance Metrics Several metrics are utilized to evaluate garbage collector performance, including: Throughput—the percentage of total time not spent in garbage collection, considered over long periods of time. Garbage collection overhead—the inverse of throughput, that is, the percentage of total time spent in garbage collection. Pause time—the length of time during which application execution is stopped while garbage collection is occurring. Frequency of collection—how often collection occurs, relative to application execution. Footprint—a measure of size, such as heap size. Promptness—the time between when an object becomes garbage and when the memory becomes available. If you’d like to explore more on this and in general about Java’s garbage collection / memory management, have a look at these slides: Java Garbage Collection, Monitoring, and Tuning View more presentations from Carol McDonald Related articles Practical Garbage Collection – Part 1: Introduction (worldmodscode.wordpress.com) Reducing memory churn when processing large data set (stackoverflow.com) The Top Java Memory Problems – Part 2 (dynatrace.com) imabonehead: Performance Tuning the JVM for Running Apache Tomcat | TomcatExpert (tomcatexpert.com) When Does the Garbage Collector Run in JVM ? (javacircles.wordpress.com) Why Garbage Collection Paranoia is Still (sometimes) Justified (prog21.dadgum.com) Adventures in Java Garbage Collection Tuning (rapleaf.com) From http://singztechmusings.in/java-garbage-collection-algorithm-design-choices-and-metrics-to-evaluate-garbage-collector-performance/

When I’m asked by web developers how they can get started with HTML5 Video, I ask them, “Why? What are you trying to solve?” Almost every time, I hear, “I just want my video to work on mobile devices.” Easy. I’ll show you how to get started. In most cases, the video content already exists in one format or another. A year and a half ago, I wrote about HTML5 video codecs and why I think H.264 is the clear leader. Nothing’s really changed. You still need to support a couple of codecs to be compatible with the full suite of modern desktop and mobile browsers, but as content creators, you get to decide how you want to encode your video content. Check out the IE Test Drive Video Format support page for some examples of how codecs work across different browsers. In reality, desktop browsers and web developers are happy to leave existing solutions in place to play existing video/audio content using plugins. That’s cool. Just supplement this with HTML5 Video and Audio tags if the browser is able to play your preferred codec natively. In my experience, the most popular mobile platforms—H.264, AAC, and MP3—are well supported using HTML5 Video and Audio Tags, which are already supported by what most people are already using. Ready to Go? Save Time, Development Cost, and Nerves. Start by learning about the Microsoft Media Platform (MMP), a frameworks are the glues together individual pieces of the Microsoft end-to-end media solution. The MMP: Player Framework (licensed for use under the Microsoft Public License Ms-PL) has recently added a preview of support for HTML5 (API Documentation) that lets you complement the Silverlight player framework with a HTML5 video experience and reach additional mobile platforms. Trust me, I have worked on a number of large-scale projects based on MMP (like the video platform behind the Rugby World Cup 2011). Two good commercial solutions that do all the work for you are JW Player™ (licensed for commercial use) and SublimeVideo® (Player as a Service). What If You Want to Roll Your Own Player? It’s surprisingly easy to roll your own video solution using default browser controls and codecs supported by the browser. The markup below shows what you need to play a video in HTML5 with a “Fall Back” to an unlisted video on YouTube. This WebMatrix is a lightweight IDE for building HTML5 mark-up. Use it—I find it handy. Demo Common Gotchas! 1. Video MIME types Set these on the server Azure Storage Explorer also allows you to do this on individual files. Update application/octet-stream to one of the following: .mp4 - “video/mp4″ .m4v - “video/m4v” .webm - “video/webm” .ogg - “application/ogg” .ogv - “video/ogg” Set these in the web.config 2. Fall-Back Fall-back content (like the YouTube example above) is only displayed by browsers that do not support the tag. If the browser supports the video tag but cannot play any of the media types you have requested, the fall-back code won’t fire. You’ll have to use JavaScript to detect this scenario using the canPlayType() method and provide fall-back content (shown below). Demo 3. Byte Range Requests (seeking) Content should be served from an HTTP 1.1-compatible web server to enable seek ahead to the end of the video. If your server is not HTTP 1.1-compatible (e.g. Azure Storage), you must encode the video with key index frames in the file and *not* at the end so that seek-ahead still works. The “H.264 YouTube HD” profile in Expression Encoder 4 Pro does this. NOTE: If the video file is gzipped, seeking won’t work. Since, with most codecs, the video/audio data is already compressed, gzip/deflate won't save you much bandwidth anyway. IIS also supports Bit Rate Throttling to save you bandwidth on the server side when delivering video content. Real-World Example I presented a session last week at Tech·Ed New Zealand, about a new video analysis system my buddies at NV Interactive, Gus and Zach, are creating for New Zealand Cricket. The solution uses video in wmv format and displays in a browser using Windows Media Plugin. This solution isn’t really supported cross platform, and it definitely doesn’t work on mobile devices. Gus and Zach are using H.264 and MediaElement.js to extend their video experience across a greater number of users and devices. Like the other commercial players, MediaElement.js uses the same HTML/CSS for all players. That means the HTML5 and Flash player experience looks the same for all users. Watch the video of the solution they’re working on: Where Does HTML5 Video Need to Go? There are currently a few key areas not addressed by the current W3C Video Standard (full screen support, live streaming, real-time communication, content protection, metadata, and accessibility). Recently, the W3C Web and TV Workshop covered these areas and offered some early thinking on how they may be adopted as web standards in the future. Live and adaptive streaming are the big topics for me. Currently, there are three proprietary solutions that support live and adaptive streaming we should pay attention to: Microsoft Smooth Streaming Apple HTTP Live Streaming Adobe HTTP Dynamic Streaming Dynamic Adaptive Streaming over HTTP (DASH) is currently in Draft International Standard. It looks likely that it will get W3C support if it is offered royalty free. DASH supports: Live, on-demand, and time-shifted content delivery and trick modes Splicing and ad insertion Byte-range requests Content descriptors for protection, accessibility, and rating What About Real-Time Communications? On HTML5Labs, you can find a Media Capture Audio Prototype that implements the audio portion of this W3C specification. HTML5 Labs is the site where Microsoft prototypes early and unstable specifications from web standards bodies such as W3C. Sharing these prototypes helps Microsoft have informed discussions with developer communities to provide better feedback on draft specifications based on this implementation experience. Their next prototype will support speech recognition and will implement the Microsoft proposal available on the W3C website. After that, the labs team will deliver another update to the Media Capture prototype that will add video capture capabilities. In Conclusion If you are hosting progressive download video and audio on the web, you should be looking to support HTML5 video and audio today to extend the reach of your content. More Resources 5 Things You Need to Know to Start Using and Today (my MIX conference presentation in Las Vegas) HTML5 Guide for Developers - video and audio Elements Simon Pieters - Everything you need to know about HTML5 video and audio Dive Into HTML5 - Video On The Web About the author Nigel Parker is an evangelist in the Developer and Platform Group at Microsoft New Zealand. He works with New Zealand software vendors helping them with the early adoption of web technologies. Nigel graduated from the University of Auckland with an honors degree in technology and philosophy. He has an entrepreneurial background and has numerous successful “start-ups” under his belt. He took some risks and broke some new ground during the first “.com” bubble and was one of the few that, despite losing money, didn’t lose heart for the unrealized potential of the tech industry. Source: http://msdn.microsoft.com/en-us/hh549238

Oracle Linux builds have a list of public yum repositories on the Oracle website, and they don't come configured with the builds, so if you're trying to install Puppet, you'll want to avoid this pitfall, along with a few others. We’ve been spending some time trying to setup our developer environment on a Oracle Linux 5.7 build and one of the first steps was to install Puppet as we’ve already created scripts which automate the installation of most things. Unfortunately Oracle Linux builds don’t come with any yum repos configured so when you run the following command… ls -alh /etc/yum.repos.d/ …you don’t see anything We eventually realised that there are a list of public yum repositories on the Oracle website, of which we needed to download the definition for Oracle Linux 5 like so: cd /etc/yum.repos.d wget http://public-yum.oracle.com/public-yum-el5.repo We then need to edit that file to enable the appropriate repository. In this case we want to enable ol5_u7_base: [ol5_u7_base] name=Oracle Linux $releasever - U7 - $basearch - base baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL5/7/base/$basearch/ gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5 gpgcheck=1 enabled=1 I made the mistake of enabling ol5_u5_base which led to us getting some really weird problems whereby yum got confused as to which version of libselinux we had installed and was therefore unable to install libselinux-ruby as its dependencies weren’t being properly satisfied. Calling ‘yum list installed’ suggested that we had libselinux 1.33.4.5-7 installed but if we ran ‘yum install libselinux’ then it suggested we already had 1.33.4.5-5 installed. Very confusing! After trying to uninstall and downgrade libselinux and pretty much destroying the installation in the process, another colleague spotted my mistake. We also found that we had to add the epel repo which gave us access to some other packages that we needed: rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm After all that was done we were able to run the command to install puppet: yum install puppet That installs puppet 2.6.12 as that’s the latest version in that repo. The latest stable version is 2.7.9 but I think we’ll need to hook up a puppet specific repo to get that working. Source: http://www.markhneedham.com/blog/2012/01/18/installing-puppet-on-oracle-linux

I love ELMAH – this is one those libraries which is both beautiful in its simplicity yet powerful in what it allows you to do. Combine the power of ELMAH with the convenience of NuGet and you can be up and running with absolutely invaluable error logging and handling in literally a couple of minutes. Yet, as the old adage goes, with great power comes great responsibility and if you’re not responsible with how you implement ELMAH, you’re also only a couple of minutes away from making session hijacking of your ASP.NET app – and many other exploits – very, very easy. What’s more, vulnerable apps are only a simple Google search away. Let me demonstrate. Update: I want to make it clear right up front that the out of the box ELMAH configuration does not make any of what you’re about to read possible. It’s only when ELMAH is configured to expose logs remotely and not properly secured that things go wrong. The ELMAH value proposition Some background first; ELMAH is the Error Logging Modules and Handlers written by the very clever Atif Aziz and it’s very popular. How popular? It’s presently the 14th most downloaded package on NuGet: What this means is that at the time of writing, there have been 42,429 downloads of the library. I suspect the popularity has a lot to do with how simple it is to implement. First of all, you can add ELMAH to your ASP.NET app without a recompile; it’s simply a matter of some web.config entries and including the ELMAH library. Secondly, it’s very easy to log errors to a number of different persistent storage mechanisms including the default in-memory store and to SQL Server (among others) for a bit more longevity. Once the web.config is set up, it just happens automagically. Thirdly, it’s very simple to configure ELMAH to fire you off an email when something goes wrong. There’s nothing like being able to actually contact a user with a “Hey, I see you were having a problem” message five minutes after they’ve had issues. People love that sort of proactivity! Fourthly, it’s very easy to retrieve log entries, you just head off to the /elmah.axd path of the app which invokes a nice little handler to pull recent messages out of whatever repository you’re using. And finally, ELMAH logs have really, really useful stuff in them for helping you actually fix the problem. But as it turns out, they also have really, really useful stuff in them for helping the bad guys break the application which brings us neatly to the purpose of today’s post. Hacker-friendly info in ELMAH Let’s start delving into the detail of what ELMAH gives us, or in this case today, what it gives the hacker. Here’s an example of what comes out of that elmah.axd handler: Actually, you can see this for yourself over at http://isnot.asafaweb.com/elmah.axd This particular site is one I use as a test bed for ASafaWeb and its deliberately insecure (more on that shortly). What you’re seeing in the image above is a request for the path “/blah.aspx” which doesn’t exist hence it’s causing an HTTP 404 “NOT FOUND” which is captured by ELMAH. So far, so good. Drilling down into that error, we’ll see a stack trace which begins to disclose the internal implementation of the code where the error occurred. Keep in mind this is totally independent of the custom errors configuration of the app; you can turn them on and provide a default redirect to an error page and ELMAH will still log what you see below – that’s the beauty of it! Now let’s scroll down a bit and get to the interesting section – server variables: I’ve highlighted two sections here and I want to refer to them from the bottom up: The “AUTH_USER” variable is set to “admin”. This is the username of the authenticated user when the error occurred. In other words, we know it was the admin logged in at the time. The “.ASPXAUTH” cookie. In case this isn’t already familiar, take a look through my post about OWASP Top 10 for .NET developers part 9: Insufficient Transport Layer Protection Read the post? Right, so now you’ll have a good idea of where this post is heading – the .ASPXAUTH cookie is used to persist the authenticated state of a user when the website uses the ASP.NET membership provider for authentication. Identifying a target The crux of this exploit centres around the fact that many people are not properly protecting ELMAH logs on their site and that it’s easily discoverable. In fact it’s so easy to discover, it’s just a matter of a simple Google search for inurl:elmahtr.axd ASPXAUTH Oh boy, that’s a lot of results – 11,000 unprotected ELMAH log pages with authentication cookie info! Substitute the “ASPXAUTH” criteria for “Error Log for” which appears at the top of every elmah.axd resource and the result is presently 192,000. Ouch! Sure, some of these results are for the same site and some are simply pages about ELMAH but whichever way you cut it, there are a huge number of sites putting their privates on public display! This is your classic Googledork or in other words, a carefully crafted search which turns up results related to careless configuration. Keep in mind also that this is obviously only the publicly accessible results which Google has indexed; how many more sites are out there exposing their ELMAH logs that simply haven’t been indexed? After all, elmah.axd is normally not a publicised resource; Google has to know it’s there and explicitly request the resource for indexing. Exploiting the website Now for the interesting bit – leveraging the info above to actually exploit the website. Let me paint a scenario which puts the ease and practicality of this into context: Wearing my evil hacker hat, I’ve just done the Google search above and identified the site I want to exploit. I can see from the logs that the ASPAUTH cookie is being captured so I know that forms authentication is being used or in other words, the site has something that it wants to protect. From this search alone there’s a good change I could find a valid ASPXAUTH token to use in my hijacking exercise. But what if the log is just using the default in-memory storage and has recently been flushed? Or there are no recent errors with an ASPAUTH cookie which is still valid? No problem, just a little bit of social engineering is required to help generate a new error message. Finding contact details for the site owner is usually simple, let’s try a message like this (I’m assuming @asafaweb is the target): The inclusion of the “<” character in the URL will cause request validation to fire – everything else in the message is just intended to build a sense of urgency (the message) and legitimacy (the domain of the URL). Now of course the user needs to be authenticated in order to get a new ASPAUTH cookie, but chances are they’re either already logged in or are using a long-lasting timeout to save them logging back in. Even if they’re not, a similarly crafted message with a link to an admin page could easily take care of this. Now it’s just a matter of the attacker watching ELMAH until a new log entry appears. The auth cookie will look something like this: .ASPXAUTH=3C886BA2344099338361C921C846EAF4E02F2A88E5E7EDE6838705928F7BB7C6FF469D35FEB1532C44B81DB38F200DEE08B6ED0E6121B945C659E932D8CE8B69FFF09E7B59DBE4820873DBD7891DD6B6BC4A486F35A2F99849017A6C72D9C6A44517D9AFDC731B3A3C55596E79732806F7DDDF9F With our hacker hat on, let’s now take this value and create a new cookie with the name and value from above. This becomes very simple with a browser extension like Edit This Cookie: You can see the “Log In” text behind the cookie window so this browser is definitely not authenticated before adding the cookie. But if the hacker submits the cookie and refreshes the page: And there you have it – the hacker is now logged in as an admin! This doesn’t give them the admin password, but it does them all the rights of the admin user. Depending on the system, this may give them the rights to view or create other accounts, manage permissions, view financial data, etc. etc. Use your imagination. But wait – there’s more A publicly exposed ELMAH log is pretty serious business vulnerability wise. There’s not just the risk of session hijacking as explained here, for example there’s the risk of disclosing internal database structure just by searching for SqlException: Or how about a bunch of SQL statements – this is just what you want to get a big head start on an injection attack: Or how about simply searching for “password” – you don’t even need to look beyond the search window on some of these: Want to find sites using a particular library in which a zero-day has just been discovered? And this is absolutely, positively only an example – I’m just picking a popular library which wouldn’t normally be discoverable by just browsing the site: But of course it’s not just about searching for existing errors which might be of interest, once an attacker knows a site is exposing ELMAH logs they can then go and attempt a whole range of other attacks and get immediate feedback about what’s going on internally! How convenient is that?! But there’s also a whole raft of other little snippets which might come in valuable to the evil doers; referring URLs, physical path of the site on the server, physical path of the site on the machine it was compiled on (often the developer’s machine), the names and values of all the cookies, the IP address of visitors to the site and so on and so forth. ELMAH logs are a treasure trove of information. Protecting against this attack This is really just a simple case of access controls or in OWASP speak, Failure to Restrict URL Access. This is not – and let me really emphasise this – is not a vulnerability in ELMAH. In a case where the membership and role providers are being used, the fix is nothing more complex than a simple authorisation entry in the web.config: It’s important to note that the ELMAH handler has been moved into the “admin” path; failure to do this may allow elmah.axd to be mapped to other paths such as “/foo/elmah.axd”. This would mean the logs could be accessed even if the path “elmah.axd” were to be secured as the authorisation rule would only apply to the handler in the root of the site. Update 11 Jan: The guidance above has been slightly modified based on the discussion in the comments below. It seems that the original guidance on the ELMAH website which I reproduced here allowed the same elmah.axd handler to be requested from other paths hence bypassing the authorisation. Bugger! That’s it, nothing more. Those 192,000 sites from the Googledork search do not have this! Each of those sites is easily vulnerable to the attack above. They’re also exposing internal stack traces and server variables which may lead to attacks of other natures. This is a serious, serious configuration vulnerability. Oh, and just in case it’s not already obvious, transport layer protection does absolutely nothing to mitigate this risk, it just means an attacker can load your sensitive log data over an encrypted connection :) A helping hand from ASafaWeb In fairness to those with publicly facing ELMAH logs, this is really easy to get wrong. The ease of implementation ELMAH offers makes it both powerful and potentially vulnerable at the same time. In fact that’s often the story with .NET in general; features like custom errors and stack traces can very easily be exposed entirely by accident. Last month I launched ASafaWeb with the intention of providing a free tool to easily check for ASP.NET configuration related vulnerabilities. Today I’m happy to also add a scan for the public accessibility of ELMAH. I’m very careful with any scans I add to ASafaWeb. Usually this means an additional HTTP request (as is the case with the ELMAH scan), and these are very costly little exercises in terms of the duration it adds to a scan. But in the case of ELMAH, the prevalence of the library combined with the enormous number of insecure sites and the ease of implementation makes it a good candidate to add. Here’s how it works; the ASafaWeb website allows you to either plug in a URL (this can be any publicly accessible URL) or alternatively, run a scan against the sample site I referred to above and have highlighted below: When the scan runs, there are a series of HTTP requests made in order to test various aspects of the site security. ASafaWeb shows you what the specific requests were then the status of each individual scan. Sometimes more than one HTTP request is used for a scan or a single request can be reused across multiple scans: Clicking on the failing “ELMAH log” scan then jumps us down the page to the details of the scan including the path with the vulnerability and how to fix it (essentially the information in the post above): You can deep link directly into the scan of the test site if you’d like to see it in action now. Summary In case I didn’t make it perfectly clear the first few times, this is not a flaw in ELMAH, in fact I think it’s a fantastic tool and I use it extensively in ASafaWeb: https://asafaweb.com/Admin/elmah.axd Whoops, you can’t access that though, can you?! And that’s really the point I’m making – ELMAH can be implemented securely and everything above is no way a recommendation not to use it. But please, please, apply a little due diligence and lock it down properly. If you do discover your ELMAH logs were publicly visible then decide to lock them down, there’s still the real risk they’ve already been indexed and cached versions are still available, in fact I saw this several times when researching for this post. If you’re in this camp, you want to take a good look at what’s in your (now secure) ELMAH logs and consider what information may have been exposed and is now searchable via the various search engines (remember, it’s not just Google). Source: http://www.troyhunt.com/2012/01/aspnet-session-hijacking-with-google.html

In my previous post we saw how to compress data consisting of very long runs of repeating elements. This type of compression is known as “run-length encoding” and can be very handy when transferring data with no loss. The problem is that the data must follow a specific format. Thus the string “aaaaaaaabbbbbbbb” can be compressed as “a8b8”. Now a string with length 16 can be compressed as a string with length 4, which is 25% of its initial length without loosing any information. There will be a problem in case the characters (elements) were dispersed in a different way. What would happen if the characters are the same, but they don’t form long runs? What if the string was “abababababababab”? The same length, the same characters, but we cannot use run-length encoding! Indeed using this algorithm we’ll get at best the same string. In this case, however, we can see another fact. The string consists of too many repeating elements, although not arranged one after another. We can compress this string with a bitmap. This means that we can save the positions of the occurrences of a given element with a sequence of bits, which can be easily converted into a decimal value. In the example above the string “abababababababab” can be compressed as “1010101010101010”, which is 43690 in decimals, and even better AAAA in hexadecimal. Thus the long string can be compressed. When decompressing (decoding) the message we can convert again from decimal/hexadecimal into binary and match the occurrences of the characters. Well, the example above is too simple, but let’s say only one of the characters is repeating and the rest of the string consists of different characters like this: “abacadaeafagahai”. Then we can use bitmap only for the character “a” – “1010101010101010” and compress it as “AAAA bcdefghi”. As you can see all the example strings are exactly 16 characters and that is a limitation. To use bitmaps with variable length of the data is a bit tricky and it is not always easy (if possible) to decompress it. Basically bitmap compression saves the positions of an element that is repeated very often in the message! In the other hand bitmap compression is not only applicable on strings. We can compress also arrays, objects or any kind of data. The example from my previous post is very suitable. Then we had to transfer a large array from a server to the client (browser) using JSON. The data then was very suitable for “run-length encoding”. Now let’s assume we have the same data – a set of different years, which this time are dispersed in a different way. $data = array( 0 => 1991, 1 => 1992, 2 => 1993, 3 => 1994, 4 => 1991, 5 => 1992, 6 => 1993, 7 => 1992, 8 => 1991, 9 => 1991, 10 => 1991, 11 => 1992, 12 => 1992, 13 => 1991, 14 => 1991, 15 => 1992, ... ); The JSON will encoded message will be the following (a simple but yet very large javascript array). [1991,1992,1993,1994,1991,1992,1993,1992,1991,1991,1991,1992,1992,1991,1991,1992, ...] However if we use bitmap compression we’ll get a “shorter” array. $data = array( 0 => array(1991, '1000100011100110'), 1 => array(1992, '0100010100011001'), 2 => array(1993, '0010001000000000'), 3 => array(1994, '0001000000000000'), ); Now the JSON is: [[1991,"1000100011100110"],[1992,"0100010100011001"],[1993,"0010001000000000"],[1994,"0001000000000000"]] It is obvious that the compression ratio is getting better and better as the uncompressed data grows. In fact, most of us know bitmap compression from images, because this algorithm is largely used for image compression. We can imagine how successful it can be when compressing black and white images (as black and white can be represented as 0 and 1s). Actually it is used for more than two colors (256 for instance) and again the level of compression is very high. Implementation The following implementation on PHP aims only to illustrate the bitmap compressing algorithm. As we know this algorithm can be applicable for any kind of data structures. // too many repeating "a" characters $msg = 'aazahalavaatalawacamaahakafaaaqaaaiauaacaaxaauaxaaaaaapaayatagaaoafaawayazavaaaazaaabararaaaaakakaaqaarazacajaazavanazaaaeanaaoajauaaaaaxalaraaapabataaavaaab'; function bitmap($message) { $i = 0; $bits = $rest = ''; while ($v = $message[$i]) { if ($v == 'a') { $bits .= '1'; } else { $bits .= '0'; $rest .= $v; } $i++; } return number_format(bindec($bits), 0, '.', '') . $rest;; } echo bitmap($msg); // uncompressed: acaaaaadaaaabalaaeaaaaganaaxakaavawamaasavajawaaaayaauaaadalanagaeaeamaarafalaazaaaiasaanaahaaazaraxaalaahaaawaaajasamahaajaakarapanaakaoakaanawalaacamauaamaal // compressed: 152299251941730035874325065523548237677352452096zhlvtlwcmhkfqiucxuxpytgofwyzvzbrrkkqrzcjzvnzenojuxlrpbtvb Application This algorithm is very useful when there is an element in our data that repeats very often, so you need to investigate the nature of the data you want to compress. Actually because of this fact this algorithm is used for image compression as PNG8 or GIF. Source: http://www.stoimen.com/blog/2012/01/16/computer-algorithms-data-compression-with-bitmaps/

One of the methods of exchanging data between processes with the multiprocessing module is directly shared memory via multiprocessing.Value. As any method that’s very general, it can sometimes be tricky to use. I’ve seen a variation of this question asked a couple of times on StackOverflow: I have some processes that do work, and I want them to increment some shared counter because [... some irrelevant reason ...] – how can this be done? The wrong way And surprisingly enough, some answers given to this question are wrong, since they use multiprocessing.Value incorrectly, as follows: import time from multiprocessing import Process, Value def func(val): for i in range(50): time.sleep(0.01) val.value += 1 if __name__ == '__main__': v = Value('i', 0) procs = [Process(target=func, args=(v,)) for i in range(10)] for p in procs: p.start() for p in procs: p.join() print v.value This code is a demonstration of the problem, distilling only the usage of the shared counter. A "pool" of 10 processes is created to run the func function. All processes share a Value and increment it 50 times. You would expect this code to eventually print 500, but in all likeness it won’t. Here’s some output taken from 10 runs of that code: > for i in {1..10}; do python sync_nolock_wrong.py; done 435 464 484 448 491 481 490 471 497 494 Why does this happen? I must admit that the documentation of multiprocessing.Value can be a bit confusing here, especially for beginners. It states that by default, a lock is created to synchronize access to the value, so one may be falsely led to believe that it would be OK to modify this value in any way imaginable from multiple processes. But it’s not. Explanation – the default locking done by Value This section is advanced and isn’t strictly required for the overall flow of the post. If you just want to understand how to synchronize the counter correctly, feel free to skip it. The locking done by multiprocessing.Value is very fine-grained. Value is a wrapper around a ctypes object, which has an underlying value attribute representing the actual object in memory. All Value does is ensure that only a single process or thread may read or write this value attribute simultaneously. This is important, since (for some types, on some architectures) writes and reads may not be atomic. I.e. to actually fill up the object’s memory, the CPU may need several instructions, and another process reading the same (shared) memory at the same time could see some intermediate, invalid state. The built-in lock of Value prevents this from happening. However, when we do this: val.value +=1 What Python actually performs is the following (disassembled bytecode with the dis module). I’ve annotated the locking done by Value in #<-- comments: 0 LOAD_FAST 0 (val) 3 DUP_TOP #<--- Value lock acquired 4 LOAD_ATTR 0 (value) #<--- Value lock released 7 LOAD_CONST 1 (1) 10 INPLACE_ADD 11 ROT_TWO #<--- Value lock acquired 12 STORE_ATTR 0 (value) #<--- Value lock released So it’s obvious that while process #1 is now at instruction 7 (LOAD_CONST), nothing prevents process #2 from also loading the (old) value attribute and be on instruction 7 too. Both processes will proceed incrementing their private copy and writing it back. The result: the actual value got incremented only once, not twice. The right way Fortunately, this problem is very easy to fix. A separate Lock is needed to guarantee the atomicity of modifications to the Value: import time from multiprocessing import Process, Value, Lock def func(val, lock): for i in range(50): time.sleep(0.01) with lock: val.value += 1 if __name__ == '__main__': v = Value('i', 0) lock = Lock() procs = [Process(target=func, args=(v, lock)) for i in range(10)] for p in procs: p.start() for p in procs: p.join() print v.value Now we get the expected result: > for i in {1..10}; do python sync_lock_right.py; done 500 500 500 500 500 500 500 500 500 500 A value and a lock may appear like too much baggage to carry around at all times. So, we can create a simple "synchronized shared counter" object to encapsulate this functionality: import time from multiprocessing import Process, Value, Lock class Counter(object): def __init__(self, initval=0): self.val = Value('i', initval) self.lock = Lock() def increment(self): with self.lock: self.val.value += 1 def value(self): with self.lock: return self.val.value def func(counter): for i in range(50): time.sleep(0.01) counter.increment() if __name__ == '__main__': counter = Counter(0) procs = [Process(target=func, args=(counter,)) for i in range(10)] for p in procs: p.start() for p in procs: p.join() print counter.value() Bonus: since we’ve now placed a more coarse-grained lock on the modification of the value, we may throw away Value with its fine-grained lock altogether, and just use multiprocessing.RawValue, that simply wraps a shared object without any locking. Source: http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing/