Oracle Linux builds have a list of public yum repositories on the Oracle website, and they don't come configured with the builds, so if you're trying to install Puppet, you'll want to avoid this pitfall, along with a few others. We’ve been spending some time trying to setup our developer environment on a Oracle Linux 5.7 build and one of the first steps was to install Puppet as we’ve already created scripts which automate the installation of most things. Unfortunately Oracle Linux builds don’t come with any yum repos configured so when you run the following command… ls -alh /etc/yum.repos.d/ …you don’t see anything We eventually realised that there are a list of public yum repositories on the Oracle website, of which we needed to download the definition for Oracle Linux 5 like so: cd /etc/yum.repos.d wget http://public-yum.oracle.com/public-yum-el5.repo We then need to edit that file to enable the appropriate repository. In this case we want to enable ol5_u7_base: [ol5_u7_base] name=Oracle Linux $releasever - U7 - $basearch - base baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL5/7/base/$basearch/ gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5 gpgcheck=1 enabled=1 I made the mistake of enabling ol5_u5_base which led to us getting some really weird problems whereby yum got confused as to which version of libselinux we had installed and was therefore unable to install libselinux-ruby as its dependencies weren’t being properly satisfied. Calling ‘yum list installed’ suggested that we had libselinux 1.33.4.5-7 installed but if we ran ‘yum install libselinux’ then it suggested we already had 1.33.4.5-5 installed. Very confusing! After trying to uninstall and downgrade libselinux and pretty much destroying the installation in the process, another colleague spotted my mistake. We also found that we had to add the epel repo which gave us access to some other packages that we needed: rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm After all that was done we were able to run the command to install puppet: yum install puppet That installs puppet 2.6.12 as that’s the latest version in that repo. The latest stable version is 2.7.9 but I think we’ll need to hook up a puppet specific repo to get that working. Source: http://www.markhneedham.com/blog/2012/01/18/installing-puppet-on-oracle-linux

I love ELMAH – this is one those libraries which is both beautiful in its simplicity yet powerful in what it allows you to do. Combine the power of ELMAH with the convenience of NuGet and you can be up and running with absolutely invaluable error logging and handling in literally a couple of minutes. Yet, as the old adage goes, with great power comes great responsibility and if you’re not responsible with how you implement ELMAH, you’re also only a couple of minutes away from making session hijacking of your ASP.NET app – and many other exploits – very, very easy. What’s more, vulnerable apps are only a simple Google search away. Let me demonstrate. Update: I want to make it clear right up front that the out of the box ELMAH configuration does not make any of what you’re about to read possible. It’s only when ELMAH is configured to expose logs remotely and not properly secured that things go wrong. The ELMAH value proposition Some background first; ELMAH is the Error Logging Modules and Handlers written by the very clever Atif Aziz and it’s very popular. How popular? It’s presently the 14th most downloaded package on NuGet: What this means is that at the time of writing, there have been 42,429 downloads of the library. I suspect the popularity has a lot to do with how simple it is to implement. First of all, you can add ELMAH to your ASP.NET app without a recompile; it’s simply a matter of some web.config entries and including the ELMAH library. Secondly, it’s very easy to log errors to a number of different persistent storage mechanisms including the default in-memory store and to SQL Server (among others) for a bit more longevity. Once the web.config is set up, it just happens automagically. Thirdly, it’s very simple to configure ELMAH to fire you off an email when something goes wrong. There’s nothing like being able to actually contact a user with a “Hey, I see you were having a problem” message five minutes after they’ve had issues. People love that sort of proactivity! Fourthly, it’s very easy to retrieve log entries, you just head off to the /elmah.axd path of the app which invokes a nice little handler to pull recent messages out of whatever repository you’re using. And finally, ELMAH logs have really, really useful stuff in them for helping you actually fix the problem. But as it turns out, they also have really, really useful stuff in them for helping the bad guys break the application which brings us neatly to the purpose of today’s post. Hacker-friendly info in ELMAH Let’s start delving into the detail of what ELMAH gives us, or in this case today, what it gives the hacker. Here’s an example of what comes out of that elmah.axd handler: Actually, you can see this for yourself over at http://isnot.asafaweb.com/elmah.axd This particular site is one I use as a test bed for ASafaWeb and its deliberately insecure (more on that shortly). What you’re seeing in the image above is a request for the path “/blah.aspx” which doesn’t exist hence it’s causing an HTTP 404 “NOT FOUND” which is captured by ELMAH. So far, so good. Drilling down into that error, we’ll see a stack trace which begins to disclose the internal implementation of the code where the error occurred. Keep in mind this is totally independent of the custom errors configuration of the app; you can turn them on and provide a default redirect to an error page and ELMAH will still log what you see below – that’s the beauty of it! Now let’s scroll down a bit and get to the interesting section – server variables: I’ve highlighted two sections here and I want to refer to them from the bottom up: The “AUTH_USER” variable is set to “admin”. This is the username of the authenticated user when the error occurred. In other words, we know it was the admin logged in at the time. The “.ASPXAUTH” cookie. In case this isn’t already familiar, take a look through my post about OWASP Top 10 for .NET developers part 9: Insufficient Transport Layer Protection Read the post? Right, so now you’ll have a good idea of where this post is heading – the .ASPXAUTH cookie is used to persist the authenticated state of a user when the website uses the ASP.NET membership provider for authentication. Identifying a target The crux of this exploit centres around the fact that many people are not properly protecting ELMAH logs on their site and that it’s easily discoverable. In fact it’s so easy to discover, it’s just a matter of a simple Google search for inurl:elmahtr.axd ASPXAUTH Oh boy, that’s a lot of results – 11,000 unprotected ELMAH log pages with authentication cookie info! Substitute the “ASPXAUTH” criteria for “Error Log for” which appears at the top of every elmah.axd resource and the result is presently 192,000. Ouch! Sure, some of these results are for the same site and some are simply pages about ELMAH but whichever way you cut it, there are a huge number of sites putting their privates on public display! This is your classic Googledork or in other words, a carefully crafted search which turns up results related to careless configuration. Keep in mind also that this is obviously only the publicly accessible results which Google has indexed; how many more sites are out there exposing their ELMAH logs that simply haven’t been indexed? After all, elmah.axd is normally not a publicised resource; Google has to know it’s there and explicitly request the resource for indexing. Exploiting the website Now for the interesting bit – leveraging the info above to actually exploit the website. Let me paint a scenario which puts the ease and practicality of this into context: Wearing my evil hacker hat, I’ve just done the Google search above and identified the site I want to exploit. I can see from the logs that the ASPAUTH cookie is being captured so I know that forms authentication is being used or in other words, the site has something that it wants to protect. From this search alone there’s a good change I could find a valid ASPXAUTH token to use in my hijacking exercise. But what if the log is just using the default in-memory storage and has recently been flushed? Or there are no recent errors with an ASPAUTH cookie which is still valid? No problem, just a little bit of social engineering is required to help generate a new error message. Finding contact details for the site owner is usually simple, let’s try a message like this (I’m assuming @asafaweb is the target): The inclusion of the “<” character in the URL will cause request validation to fire – everything else in the message is just intended to build a sense of urgency (the message) and legitimacy (the domain of the URL). Now of course the user needs to be authenticated in order to get a new ASPAUTH cookie, but chances are they’re either already logged in or are using a long-lasting timeout to save them logging back in. Even if they’re not, a similarly crafted message with a link to an admin page could easily take care of this. Now it’s just a matter of the attacker watching ELMAH until a new log entry appears. The auth cookie will look something like this: .ASPXAUTH=3C886BA2344099338361C921C846EAF4E02F2A88E5E7EDE6838705928F7BB7C6FF469D35FEB1532C44B81DB38F200DEE08B6ED0E6121B945C659E932D8CE8B69FFF09E7B59DBE4820873DBD7891DD6B6BC4A486F35A2F99849017A6C72D9C6A44517D9AFDC731B3A3C55596E79732806F7DDDF9F With our hacker hat on, let’s now take this value and create a new cookie with the name and value from above. This becomes very simple with a browser extension like Edit This Cookie: You can see the “Log In” text behind the cookie window so this browser is definitely not authenticated before adding the cookie. But if the hacker submits the cookie and refreshes the page: And there you have it – the hacker is now logged in as an admin! This doesn’t give them the admin password, but it does them all the rights of the admin user. Depending on the system, this may give them the rights to view or create other accounts, manage permissions, view financial data, etc. etc. Use your imagination. But wait – there’s more A publicly exposed ELMAH log is pretty serious business vulnerability wise. There’s not just the risk of session hijacking as explained here, for example there’s the risk of disclosing internal database structure just by searching for SqlException: Or how about a bunch of SQL statements – this is just what you want to get a big head start on an injection attack: Or how about simply searching for “password” – you don’t even need to look beyond the search window on some of these: Want to find sites using a particular library in which a zero-day has just been discovered? And this is absolutely, positively only an example – I’m just picking a popular library which wouldn’t normally be discoverable by just browsing the site: But of course it’s not just about searching for existing errors which might be of interest, once an attacker knows a site is exposing ELMAH logs they can then go and attempt a whole range of other attacks and get immediate feedback about what’s going on internally! How convenient is that?! But there’s also a whole raft of other little snippets which might come in valuable to the evil doers; referring URLs, physical path of the site on the server, physical path of the site on the machine it was compiled on (often the developer’s machine), the names and values of all the cookies, the IP address of visitors to the site and so on and so forth. ELMAH logs are a treasure trove of information. Protecting against this attack This is really just a simple case of access controls or in OWASP speak, Failure to Restrict URL Access. This is not – and let me really emphasise this – is not a vulnerability in ELMAH. In a case where the membership and role providers are being used, the fix is nothing more complex than a simple authorisation entry in the web.config: It’s important to note that the ELMAH handler has been moved into the “admin” path; failure to do this may allow elmah.axd to be mapped to other paths such as “/foo/elmah.axd”. This would mean the logs could be accessed even if the path “elmah.axd” were to be secured as the authorisation rule would only apply to the handler in the root of the site. Update 11 Jan: The guidance above has been slightly modified based on the discussion in the comments below. It seems that the original guidance on the ELMAH website which I reproduced here allowed the same elmah.axd handler to be requested from other paths hence bypassing the authorisation. Bugger! That’s it, nothing more. Those 192,000 sites from the Googledork search do not have this! Each of those sites is easily vulnerable to the attack above. They’re also exposing internal stack traces and server variables which may lead to attacks of other natures. This is a serious, serious configuration vulnerability. Oh, and just in case it’s not already obvious, transport layer protection does absolutely nothing to mitigate this risk, it just means an attacker can load your sensitive log data over an encrypted connection :) A helping hand from ASafaWeb In fairness to those with publicly facing ELMAH logs, this is really easy to get wrong. The ease of implementation ELMAH offers makes it both powerful and potentially vulnerable at the same time. In fact that’s often the story with .NET in general; features like custom errors and stack traces can very easily be exposed entirely by accident. Last month I launched ASafaWeb with the intention of providing a free tool to easily check for ASP.NET configuration related vulnerabilities. Today I’m happy to also add a scan for the public accessibility of ELMAH. I’m very careful with any scans I add to ASafaWeb. Usually this means an additional HTTP request (as is the case with the ELMAH scan), and these are very costly little exercises in terms of the duration it adds to a scan. But in the case of ELMAH, the prevalence of the library combined with the enormous number of insecure sites and the ease of implementation makes it a good candidate to add. Here’s how it works; the ASafaWeb website allows you to either plug in a URL (this can be any publicly accessible URL) or alternatively, run a scan against the sample site I referred to above and have highlighted below: When the scan runs, there are a series of HTTP requests made in order to test various aspects of the site security. ASafaWeb shows you what the specific requests were then the status of each individual scan. Sometimes more than one HTTP request is used for a scan or a single request can be reused across multiple scans: Clicking on the failing “ELMAH log” scan then jumps us down the page to the details of the scan including the path with the vulnerability and how to fix it (essentially the information in the post above): You can deep link directly into the scan of the test site if you’d like to see it in action now. Summary In case I didn’t make it perfectly clear the first few times, this is not a flaw in ELMAH, in fact I think it’s a fantastic tool and I use it extensively in ASafaWeb: https://asafaweb.com/Admin/elmah.axd Whoops, you can’t access that though, can you?! And that’s really the point I’m making – ELMAH can be implemented securely and everything above is no way a recommendation not to use it. But please, please, apply a little due diligence and lock it down properly. If you do discover your ELMAH logs were publicly visible then decide to lock them down, there’s still the real risk they’ve already been indexed and cached versions are still available, in fact I saw this several times when researching for this post. If you’re in this camp, you want to take a good look at what’s in your (now secure) ELMAH logs and consider what information may have been exposed and is now searchable via the various search engines (remember, it’s not just Google). Source: http://www.troyhunt.com/2012/01/aspnet-session-hijacking-with-google.html

In my previous post we saw how to compress data consisting of very long runs of repeating elements. This type of compression is known as “run-length encoding” and can be very handy when transferring data with no loss. The problem is that the data must follow a specific format. Thus the string “aaaaaaaabbbbbbbb” can be compressed as “a8b8”. Now a string with length 16 can be compressed as a string with length 4, which is 25% of its initial length without loosing any information. There will be a problem in case the characters (elements) were dispersed in a different way. What would happen if the characters are the same, but they don’t form long runs? What if the string was “abababababababab”? The same length, the same characters, but we cannot use run-length encoding! Indeed using this algorithm we’ll get at best the same string. In this case, however, we can see another fact. The string consists of too many repeating elements, although not arranged one after another. We can compress this string with a bitmap. This means that we can save the positions of the occurrences of a given element with a sequence of bits, which can be easily converted into a decimal value. In the example above the string “abababababababab” can be compressed as “1010101010101010”, which is 43690 in decimals, and even better AAAA in hexadecimal. Thus the long string can be compressed. When decompressing (decoding) the message we can convert again from decimal/hexadecimal into binary and match the occurrences of the characters. Well, the example above is too simple, but let’s say only one of the characters is repeating and the rest of the string consists of different characters like this: “abacadaeafagahai”. Then we can use bitmap only for the character “a” – “1010101010101010” and compress it as “AAAA bcdefghi”. As you can see all the example strings are exactly 16 characters and that is a limitation. To use bitmaps with variable length of the data is a bit tricky and it is not always easy (if possible) to decompress it. Basically bitmap compression saves the positions of an element that is repeated very often in the message! In the other hand bitmap compression is not only applicable on strings. We can compress also arrays, objects or any kind of data. The example from my previous post is very suitable. Then we had to transfer a large array from a server to the client (browser) using JSON. The data then was very suitable for “run-length encoding”. Now let’s assume we have the same data – a set of different years, which this time are dispersed in a different way. $data = array( 0 => 1991, 1 => 1992, 2 => 1993, 3 => 1994, 4 => 1991, 5 => 1992, 6 => 1993, 7 => 1992, 8 => 1991, 9 => 1991, 10 => 1991, 11 => 1992, 12 => 1992, 13 => 1991, 14 => 1991, 15 => 1992, ... ); The JSON will encoded message will be the following (a simple but yet very large javascript array). [1991,1992,1993,1994,1991,1992,1993,1992,1991,1991,1991,1992,1992,1991,1991,1992, ...] However if we use bitmap compression we’ll get a “shorter” array. $data = array( 0 => array(1991, '1000100011100110'), 1 => array(1992, '0100010100011001'), 2 => array(1993, '0010001000000000'), 3 => array(1994, '0001000000000000'), ); Now the JSON is: [[1991,"1000100011100110"],[1992,"0100010100011001"],[1993,"0010001000000000"],[1994,"0001000000000000"]] It is obvious that the compression ratio is getting better and better as the uncompressed data grows. In fact, most of us know bitmap compression from images, because this algorithm is largely used for image compression. We can imagine how successful it can be when compressing black and white images (as black and white can be represented as 0 and 1s). Actually it is used for more than two colors (256 for instance) and again the level of compression is very high. Implementation The following implementation on PHP aims only to illustrate the bitmap compressing algorithm. As we know this algorithm can be applicable for any kind of data structures. // too many repeating "a" characters $msg = 'aazahalavaatalawacamaahakafaaaqaaaiauaacaaxaauaxaaaaaapaayatagaaoafaawayazavaaaazaaabararaaaaakakaaqaarazacajaazavanazaaaeanaaoajauaaaaaxalaraaapabataaavaaab'; function bitmap($message) { $i = 0; $bits = $rest = ''; while ($v = $message[$i]) { if ($v == 'a') { $bits .= '1'; } else { $bits .= '0'; $rest .= $v; } $i++; } return number_format(bindec($bits), 0, '.', '') . $rest;; } echo bitmap($msg); // uncompressed: acaaaaadaaaabalaaeaaaaganaaxakaavawamaasavajawaaaayaauaaadalanagaeaeamaarafalaazaaaiasaanaahaaazaraxaalaahaaawaaajasamahaajaakarapanaakaoakaanawalaacamauaamaal // compressed: 152299251941730035874325065523548237677352452096zhlvtlwcmhkfqiucxuxpytgofwyzvzbrrkkqrzcjzvnzenojuxlrpbtvb Application This algorithm is very useful when there is an element in our data that repeats very often, so you need to investigate the nature of the data you want to compress. Actually because of this fact this algorithm is used for image compression as PNG8 or GIF. Source: http://www.stoimen.com/blog/2012/01/16/computer-algorithms-data-compression-with-bitmaps/

One of the methods of exchanging data between processes with the multiprocessing module is directly shared memory via multiprocessing.Value. As any method that’s very general, it can sometimes be tricky to use. I’ve seen a variation of this question asked a couple of times on StackOverflow: I have some processes that do work, and I want them to increment some shared counter because [... some irrelevant reason ...] – how can this be done? The wrong way And surprisingly enough, some answers given to this question are wrong, since they use multiprocessing.Value incorrectly, as follows: import time from multiprocessing import Process, Value def func(val): for i in range(50): time.sleep(0.01) val.value += 1 if __name__ == '__main__': v = Value('i', 0) procs = [Process(target=func, args=(v,)) for i in range(10)] for p in procs: p.start() for p in procs: p.join() print v.value This code is a demonstration of the problem, distilling only the usage of the shared counter. A "pool" of 10 processes is created to run the func function. All processes share a Value and increment it 50 times. You would expect this code to eventually print 500, but in all likeness it won’t. Here’s some output taken from 10 runs of that code: > for i in {1..10}; do python sync_nolock_wrong.py; done 435 464 484 448 491 481 490 471 497 494 Why does this happen? I must admit that the documentation of multiprocessing.Value can be a bit confusing here, especially for beginners. It states that by default, a lock is created to synchronize access to the value, so one may be falsely led to believe that it would be OK to modify this value in any way imaginable from multiple processes. But it’s not. Explanation – the default locking done by Value This section is advanced and isn’t strictly required for the overall flow of the post. If you just want to understand how to synchronize the counter correctly, feel free to skip it. The locking done by multiprocessing.Value is very fine-grained. Value is a wrapper around a ctypes object, which has an underlying value attribute representing the actual object in memory. All Value does is ensure that only a single process or thread may read or write this value attribute simultaneously. This is important, since (for some types, on some architectures) writes and reads may not be atomic. I.e. to actually fill up the object’s memory, the CPU may need several instructions, and another process reading the same (shared) memory at the same time could see some intermediate, invalid state. The built-in lock of Value prevents this from happening. However, when we do this: val.value +=1 What Python actually performs is the following (disassembled bytecode with the dis module). I’ve annotated the locking done by Value in #<-- comments: 0 LOAD_FAST 0 (val) 3 DUP_TOP #<--- Value lock acquired 4 LOAD_ATTR 0 (value) #<--- Value lock released 7 LOAD_CONST 1 (1) 10 INPLACE_ADD 11 ROT_TWO #<--- Value lock acquired 12 STORE_ATTR 0 (value) #<--- Value lock released So it’s obvious that while process #1 is now at instruction 7 (LOAD_CONST), nothing prevents process #2 from also loading the (old) value attribute and be on instruction 7 too. Both processes will proceed incrementing their private copy and writing it back. The result: the actual value got incremented only once, not twice. The right way Fortunately, this problem is very easy to fix. A separate Lock is needed to guarantee the atomicity of modifications to the Value: import time from multiprocessing import Process, Value, Lock def func(val, lock): for i in range(50): time.sleep(0.01) with lock: val.value += 1 if __name__ == '__main__': v = Value('i', 0) lock = Lock() procs = [Process(target=func, args=(v, lock)) for i in range(10)] for p in procs: p.start() for p in procs: p.join() print v.value Now we get the expected result: > for i in {1..10}; do python sync_lock_right.py; done 500 500 500 500 500 500 500 500 500 500 A value and a lock may appear like too much baggage to carry around at all times. So, we can create a simple "synchronized shared counter" object to encapsulate this functionality: import time from multiprocessing import Process, Value, Lock class Counter(object): def __init__(self, initval=0): self.val = Value('i', initval) self.lock = Lock() def increment(self): with self.lock: self.val.value += 1 def value(self): with self.lock: return self.val.value def func(counter): for i in range(50): time.sleep(0.01) counter.increment() if __name__ == '__main__': counter = Counter(0) procs = [Process(target=func, args=(counter,)) for i in range(10)] for p in procs: p.start() for p in procs: p.join() print counter.value() Bonus: since we’ve now placed a more coarse-grained lock on the modification of the value, we may throw away Value with its fine-grained lock altogether, and just use multiprocessing.RawValue, that simply wraps a shared object without any locking. Source: http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing/

python -m SimpleHTTPServer is a very handy command that sets up a server listening on a port (by default 8080) having as document root the current directory. However it does not count for the purpose of this article: producing our first dynamic page with Python. We won't care a lot about the performance of various approaches or whether to adopt frameworks like Django: we are just getting started. Even when higher-level abstraction are introduced, we still have to know how the foundation of our house are made. Installation mod_python is an Apache module that bridges the Apache 2 httpd processes with the Python interpreter. You can install it on Ubuntu with a single command: sudo apt-get install libapache2-mod-python See the mod_python website for alternative ways of installation. In the words of its homepage, mod_python is a mature but not dead project: it requires very little maintenance due to its stability. It is of course built only for Apache, and exposes its API; in any case, it promises to be faster than CGI (who doesn't?) Apache configuration This configuration lines should made their way into your httpd.conf or in the other configuration files you have on your system: LoadModule python_module /usr/lib/apache2/modules/mod_python.so AddHandler mod_python .py PythonHandler hello PythonDebug On We loaded the .so file of the module: if you installed mod_python in a Linux distribution, this line should already be present (use the a2enmod command if it's not in the enabled/ directory). In Ubuntu, it's written in the /etc/apache2/mods-enabled/python.load file, where you can add the other lines. The rest of the configuration means that in the directory /var/www/python, we wwant to enable the processing of .py files by mod_python. The hello.py file will be targeted, unless you use a standard handler. PythonDebug On activates a nice stack trace visualization in the browser, in case of runtime errors. Without it, the production setting is to display a blank, generic 500 Server Error. Hello world Our Hello World it's easy to write: from mod_python import apache def handler(req): req.content_type = 'text/plain' req.write("Hello, World!") return apache.OK It's a bit strange that the Content-Type has to be specified on the request object, but there is no response object anyway: all interaction has to be performed on req. A bit less boilerplate? You can use a standard handler to simplify even more your .py files, by changing the configuration to this: PythonHandler mod_python.publisher Now the Hello World example becomes: from mod_python import apache def say(req): return "Hello, World!" and is displayed at http://localhost/python/hello.py/say. While the parameters are actually kept in the request object, you can specify them as required, named parameters: def say(req, name): return "Hello, " + name + "!" http://localhost/python/hello.py/say?name=Giorgio will display: Hello, Giorgio! Including other files import is the standard Python instruction for including other code into the current source file, usually with the form from module import entity. You can use it in your handlers, but you probably need to specify Python paths specific to each of them, to avoid that multiple application clash with each other: PythonPath "sys.path+['/var/python_code']" The string value of this directive should evaluate to a Python list containing all the folders to search for modules, just like sys.path originally does. To show you how it works, I can put this in /var/python_code/hello_lib.py: def say_to(name): return "Hello, " + name + "!" and micro-refactor my handler to: from mod_python import apache from hello_lib import say_to def say(req, name): return say_to(name) Conclusion Of course there's more to mod_python and to build real web applications: they have to manipulate cookies, the session API, and databases. Moreover, their configuration should access Apache internals, with directives for everything from importing automatically modules at the start of a request to setting up HTTP authentication. However, this article is about the HTTP-facing side of a web application. It's interesting to explore how the architecture on the server side works, so probably my next stop will be to try out a framework like Django to see which standard design choices it mades for us (and which may be questionable). In any case, we already have a Turing-complete entry point in our handlers, capable of linking to any Python source code. Resources The official mod_python documentation is long-winded but very good. The Style Guide for Python Code is a more general document that defines the coding standards for the whole Python world, while applying also to our web development scenario.

Domain Specific Languages (DSLs) have become a valuable part of the Groovy idiom. DSLs are used in native Groovy builders, Grails and GORM, and testing frameworks. To a developer, DSLs are consumable and understandable, which makes implementation more fluid compared to traditional programming. But how is a DSL implemented? How does it work behind the scenes? This article will demonstrate a simple DSL that can get a developer kickstarted on the basic concepts. What is a DSL DSL's are meant to target a particular type of problem. They are short expressive means of programming that fit well in a narrow context. For example with GORM, you can express hibernate mapping with a DSL rather than XML. static mapping = { table 'person' columns { name column:'name' } } Much of the theory for DSLs and the benefits they deliver are well documented. Refer to these sources as a starting point: Martin Fowler - DSL Writing Domain Specific Languages - Groovy Programming Groovy - Chapter 18 A Simple DSL Example in Groovy The following example offers a simplified view of implementing an internal DSL. Frameworks have much more advanced methods of creating a DSL. However this example does highlight closure delegation and meta object protocol concepts that are essential to understanding the inner workings of a DSL. Requirements Overview Imagine that a customer needs a memo generator. The memos needs to have a few simple fields, such as "to", "from", and "body". The memo can also have sections such as "Summary" or "Important." The summary fields are dynamic and can be anything on demand. In addition, the memo needs to be outputed in three formats: xml, html, and text. We elect to implement this as a DSL in Groovy. The DSL result looks like this: MemoDsl.make { to "Nirav Assar" from "Barack Obama" body "How are things? We are doing well. Take care" idea "The economy is key" request "Please vote for me" xml } The output from the code yields: Nirav Assar Barack Obama How are things? We are doing well. Take care The economy is key Please vote for me The last line in the DSL can also be changed to 'html' or 'text'. This affects the output format. Implementation A static method that accepts a closure is a hassle free way to implement a DSL. In the memo example, the class MemoDsl has a make method. It creates an instance and delegates all calls in the closure to the instance. This is the mechanism where the "to", and "from" sections end up executing methods inside the MemoDsl class. Once the to() method is called, we store the text in the instance for formatting later on. class MemoDsl { String toText String fromText String body def sections = [] /** * This method accepts a closure which is essentially the DSL. Delegate the * closure methods to * the DSL class so the calls can be processed */ def static make(closure) { MemoDsl memoDsl = new MemoDsl() // any method called in closure will be delegated to the memoDsl class closure.delegate = memoDsl closure() } /** * Store the parameter as a variable and use it later to output a memo */ def to(String toText) { this.toText = toText } def from(String fromText) { this.fromText = fromText } def body(String bodyText) { this.body = bodyText } } Dynamic Sections When the closure includes a method that is not present in the MemoDsl class, groovy identifies it as a missing method. With Groovy's meta object protocol, the methodMissing interface on the class is invoked. This is how we handle sections for the memo. In the client code above we have entries for idea and request. MemoDsl.make { to "Nirav Assar" from "Barack Obama" body "How are things? We are doing well. Take care" idea "The economy is key" request "Please vote for me" xml } The sections are processed with the following code in MemoDsl. It creates a section class and appends it to a list in the instance. /** * When a method is not recognized, assume it is a title for a new section. Create a simple * object that contains the method name and the parameter which is the body. */ def methodMissing(String methodName, args) { def section = new Section(title: methodName, body: args[0]) sections << section } Processing Various Outputs Finally the most interesting part of the DSL is how we process the various outputs. The final line in the closure specifies the output desired. When a closure contains a string such as "xml" with no parameters, groovy assumes this is a 'getter' method. Thus we need to implement 'getXml()' to catch the delegation execution: /** * 'get' methods get called from the dsl by convention. Due to groovy closure delegation, * we had to place MarkUpBuilder and StringWrite code in a static method as the delegate of the closure * did not have access to the system.out */ def getXml() { doXml(this) } /** * Use markupBuilder to create a customer xml output */ private static doXml(MemoDsl memoDsl) { def writer = new StringWriter() def xml = new MarkupBuilder(writer) xml.memo() { to(memoDsl.toText) from(memoDsl.fromText) body(memoDsl.body) // cycle through the stored section objects to create an xml tag for (s in memoDsl.sections) { "$s.title"(s.body) } } println writer } The code for html and text is quite similar. The only variation is how the output is formatted. Entire Code The code in its entirety is displayed next. The best approach I found was to design the DSL client code and the specified formats first, then tackle the implementation. I used TDD and JUnit to drive my implementation. Note that I did not go the extra mile to do asserts on the system output in the tests, although this could be easily enhanced to do so. The code is fully executable inside any IDE. Run the various tests to view the DSL output. package com.solutionsfit.dsl.memotemplate class MemolDslTest extends GroovyTestCase { void testDslUsage_outputXml() { MemoDsl.make { to "Nirav Assar" from "Barack Obama" body "How are things? We are doing well. Take care" idea "The economy is key" request "Please vote for me" xml } } void testDslUsage_outputHtml() { MemoDsl.make { to "Nirav Assar" from "Barack Obama" body "How are things? We are doing well. Take care" idea "The economy is key" request "Please vote for me" html } } void testDslUsage_outputText() { MemoDsl.make { to "Nirav Assar" from "Barack Obama" body "How are things? We are doing well. Take care" idea "The economy is key" request "Please vote for me" text } } } package com.solutionsfit.dsl.memotemplate import groovy.xml.MarkupBuilder /** * Processes a simple DSL to create various formats of a memo: xml, html, and text */ class MemoDsl { String toText String fromText String body def sections = [] /** * This method accepts a closure which is essentially the DSL. Delegate the closure methods to * the DSL class so the calls can be processed */ def static make(closure) { MemoDsl memoDsl = new MemoDsl() // any method called in closure will be delegated to the memoDsl class closure.delegate = memoDsl closure() } /** * Store the parameter as a variable and use it later to output a memo */ def to(String toText) { this.toText = toText } def from(String fromText) { this.fromText = fromText } def body(String bodyText) { this.body = bodyText } /** * When a method is not recognized, assume it is a title for a new section. Create a simple * object that contains the method name and the parameter which is the body. */ def methodMissing(String methodName, args) { def section = new Section(title: methodName, body: args[0]) sections << section } /** * 'get' methods get called from the dsl by convention. Due to groovy closure delegation, * we had to place MarkUpBuilder and StringWrite code in a static method as the delegate of the closure * did not have access to the system.out */ def getXml() { doXml(this) } def getHtml() { doHtml(this) } def getText() { doText(this) } /** * Use markupBuilder to create a customer xml output */ private static doXml(MemoDsl memoDsl) { def writer = new StringWriter() def xml = new MarkupBuilder(writer) xml.memo() { to(memoDsl.toText) from(memoDsl.fromText) body(memoDsl.body) // cycle through the stored section objects to create an xml tag for (s in memoDsl.sections) { "$s.title"(s.body) } } println writer } /** * Use markupBuilder to create an html xml output */ private static doHtml(MemoDsl memoDsl) { def writer = new StringWriter() def xml = new MarkupBuilder(writer) xml.html() { head { title("Memo") } body { h1("Memo") h3("To: ${memoDsl.toText}") h3("From: ${memoDsl.fromText}") p(memoDsl.body) // cycle through the stored section objects and create uppercase/bold section with body for (s in memoDsl.sections) { p { b(s.title.toUpperCase()) } p(s.body) } } } println writer } /** * Use markupBuilder to create an html xml output */ private static doText(MemoDsl memoDsl) { String template = "Memo\nTo: ${memoDsl.toText}\nFrom: ${memoDsl.fromText}\n${memoDsl.body}\n" def sectionStrings ="" for (s in memoDsl.sections) { sectionStrings += s.title.toUpperCase() + "\n" + s.body + "\n" } template += sectionStrings println template } }

In this post we will see how to visualize a function of two variables in two ways. First, we will create an intensity image of the function and, second, we will use the 3D plotting capabilities of matplotlib to create a shaded surface plot. So, let's go with the code: from numpy import exp,arange from pylab import meshgrid,cm,imshow,contour,clabel,colorbar,axis,title,show # the function that I'm going to plot def z_func(x,y): return (1-(x**2+y**3))*exp(-(x**2+y**2)/2) x = arange(-3.0,3.0,0.1) y = arange(-3.0,3.0,0.1) X,Y = meshgrid(x, y) # grid of point Z = z_func(X, Y) # evaluation of the function on the grid im = imshow(Z,cmap=cm.RdBu) # drawing the function # adding the Contour lines with labels cset = contour(Z,arange(-1,1.5,0.2),linewidths=2,cmap=cm.Set2) clabel(cset,inline=True,fmt='%1.1f',fontsize=10) colorbar(im) # adding the colobar on the right # latex fashion title title('$z=(1-x^2+y^3) e^{-(x^2+y^2)/2}$') show() The script would have the following output: And now we are going to use the values stored in X,Y and Z to make a 3D plot using the mplot3d toolkit. Here's the snippet: from mpl_toolkits.mplot3d import Axes3D from matplotlib import cm from matplotlib.ticker import LinearLocator, FormatStrFormatter import matplotlib.pyplot as plt from numpy import sin,sqrt fig = plt.figure() ax = fig.gca(projection='3d') surf = ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.RdBu,linewidth=0, antialiased=False) ax.zaxis.set_major_locator(LinearLocator(10)) ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f')) fig.colorbar(surf, shrink=0.5, aspect=5) plt.show() And this is the result: Source: http://glowingpython.blogspot.com/2012/01/how-to-plot-two-variable-functions-with.html

Using open as a context manager is a great way to ensure your file handles are closed properly and is becoming common: with open('/some/path', 'w') as f: f.write('something') The issue is that even if you mock out the call to open it is the returned object that is used as a context manager (and has __enter__ and __exit__ called). Using MagicMock from the mock library, we can mock out context managers very simply. However, mocking open is fiddly enough that a helper function is useful. Here mock_open creates and configures a MagicMock that behaves as a file context manager. from mock import inPy3k, MagicMock if inPy3k: file_spec = ['_CHUNK_SIZE', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__gt__', '__hash__', '__iter__', '__le__', '__lt__', '__ne__', '__next__', '__repr__', '__str__', '_checkClosed', '_checkReadable', '_checkSeekable', '_checkWritable', 'buffer', 'close', 'closed', 'detach', 'encoding', 'errors', 'fileno', 'flush', 'isatty', 'line_buffering', 'mode', 'name', 'newlines', 'peek', 'raw', 'read', 'read1', 'readable', 'readinto', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 'writable', 'write', 'writelines'] else: file_spec = file def mock_open(mock=None, data=None): if mock is None: mock = MagicMock(spec=file_spec) handle = MagicMock(spec=file_spec) handle.write.return_value = None if data is None: handle.__enter__.return_value = handle else: handle.__enter__.return_value = data mock.return_value = handle return mock >>> m = mock_open() >>> with patch('__main__.open', m, create=True): ... with open('foo', 'w') as h: ... h.write('some stuff') ... >>> m.assert_called_once_with('foo', 'w') >>> m.mock_calls [call('foo', 'w'), call().__enter__(), call().write('some stuff'), call().__exit__(None, None, None)] >>> handle = m() >>> handle.write.assert_called_once_with('some stuff') And for reading files, using a StringIO to represent the file handle: >>> from StringIO import StringIO >>> m = mock_open(data=StringIO('foo bar baz')) >>> with patch('__main__.open', m, create=True): ... with open('foo') as h: ... result = h.read() ... >>> m.assert_called_once_with('foo') >>> assert result == 'foo bar baz' Note that the StringIO will only be used for the data if open is used as a context manager. If you just configure and use mocks they will work whichever way open is used. This helper function will be built into mock 0.9. Source: http://www.voidspace.org.uk/python/weblog/arch_d7_2012_01_07.shtml

today we continue html5 canvas examples. and today is our second tutorial for webgl. we will be creating animated twisting images. also we will add handlers to manipulate the images with mouse and keyboard. here are our demo and downloadable package: live demo download in package ok, download the example files and lets start coding ! step 1. html here are html sources of our demo. as you can see – just an empty page. index.html creating a keyboard sensitive 3d twisted images in webgl | script tutorials you can use your mouse + arrow keys + page up / down step 2. css here are used css styles. css/main.css body { background:#eee; font-family:verdana, helvetica, arial, sans-serif; margin:0; padding:0 } .example { background:#fff; width:500px; font-size:80%; border:1px #000 solid; margin:20px auto; padding:15px; position:relative; -moz-border-radius: 3px; -webkit-border-radius:3px } h3 { text-align:center; } step 3. js js/webgl-utils.js and js/glmatrix-0.9.5.min.js these files we will use in project for working with webgl. both files will in our package. js/script.js var gl; // global webgl object var shaderprogram; var pics_names=['1.png', '2.png', '3.png', '4.png', '5.png', '6.png', '7.png']; var pics_num=pics_names.length; // diffirent initializations function initgl(canvas) { try { gl = canvas.getcontext('experimental-webgl'); gl.viewportwidth = canvas.width; gl.viewportheight = canvas.height; } catch (e) {} if (! gl) { alert('can`t initialise webgl, not supported'); } } function getshader(gl, type) { var str = ''; var shader; if (type == 'x-fragment') { str = "#ifdef gl_es\n"+ "precision highp float;\n"+ "#endif\n"+ "varying vec2 vtexturecoord;\n"+ "uniform sampler2d usampler;\n"+ "void main(void) {\n"+ " gl_fragcolor = texture2d(usampler, vec2(vtexturecoord.s, vtexturecoord.t));\n"+ "}\n"; shader = gl.createshader(gl.fragment_shader); } else if (type == 'x-vertex') { str = "attribute vec3 avertexposition;\n"+ "attribute vec2 atexturecoord;\n"+ "uniform mat4 umvmatrix;\n"+ "uniform mat4 upmatrix;\n"+ "varying vec2 vtexturecoord;\n"+ "void main(void) {\n"+ " gl_position = upmatrix * umvmatrix * vec4(avertexposition, 1.0);\n"+ " vtexturecoord = atexturecoord;\n"+ "}\n"; shader = gl.createshader(gl.vertex_shader); } else { return null; } gl.shadersource(shader, str); gl.compileshader(shader); if (!gl.getshaderparameter(shader, gl.compile_status)) { alert(gl.getshaderinfolog(shader)); return null; } return shader; } function initshaders() { var fragmentshader = getshader(gl, 'x-fragment'); var vertexshader = getshader(gl, 'x-vertex'); shaderprogram = gl.createprogram(); gl.attachshader(shaderprogram, vertexshader); gl.attachshader(shaderprogram, fragmentshader); gl.linkprogram(shaderprogram); if (!gl.getprogramparameter(shaderprogram, gl.link_status)) { alert('can`t initialise shaders'); } gl.useprogram(shaderprogram); shaderprogram.vertexpositionattribute = gl.getattriblocation(shaderprogram, 'avertexposition'); gl.enablevertexattribarray(shaderprogram.vertexpositionattribute); shaderprogram.texturecoordattribute = gl.getattriblocation(shaderprogram, 'atexturecoord'); gl.enablevertexattribarray(shaderprogram.texturecoordattribute); shaderprogram.pmatrixuniform = gl.getuniformlocation(shaderprogram, 'upmatrix'); shaderprogram.mvmatrixuniform = gl.getuniformlocation(shaderprogram, 'umvmatrix'); shaderprogram.sampleruniform = gl.getuniformlocation(shaderprogram, 'usampler'); } var objvertexpositionbuffer=new array(); var objvertextexturecoordbuffer=new array(); var objvertexindexbuffer=new array(); function initobjbuffers() { for (var i=0;i

JSR 303 and Hibernate Validator have been awesome additions to the Java ecosystem, giving you a standard way to validate your domain model across application layers.

Matrices can be really large, sometimes larger than you can hold in one array. You can extend the maximum size by having multiple arrays however this can make your heap size really large and inefficient. An alternative is to use a wrapper over a memory mapped file. The advantage of memory mapped files is that they have very little impact on the heap and can be swapped in and out by the OS fairly transparently. A huge matrix This code supports large matrices of double. It partitions the file into 1 GB mappings. (As Java doesn't support mappings of 2 GB or more at a time, a pet hate of mine ;) import sun.misc.Cleaner; import sun.nio.ch.DirectBuffer; import java.io.Closeable; import java.io.IOException; import java.io.RandomAccessFile; import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel; import java.util.ArrayList; import java.util.List; public class LargeDoubleMatrix implements Closeable { private static final int MAPPING_SIZE = 1 << 30; private final RandomAccessFile raf; private final int width; private final int height; private final List mappings = new ArrayList(); public LargeDoubleMatrix(String filename, int width, int height) throws IOException { this.raf = new RandomAccessFile(filename, "rw"); try { this.width = width; this.height = height; long size = 8L * width * height; for (long offset = 0; offset < size; offset += MAPPING_SIZE) { long size2 = Math.min(size - offset, MAPPING_SIZE); mappings.add(raf.getChannel().map(FileChannel.MapMode.READ_WRITE, offset, size2)); } } catch (IOException e) { raf.close(); throw e; } } protected long position(int x, int y) { return (long) y * width + x; } public int width() { return width; } public int height() { return height; } public double get(int x, int y) { assert x >= 0 && x < width; assert y >= 0 && y < height; long p = position(x, y) * 8; int mapN = (int) (p / MAPPING_SIZE); int offN = (int) (p % MAPPING_SIZE); return mappings.get(mapN).getDouble(offN); } public void set(int x, int y, double d) { assert x >= 0 && x < width; assert y >= 0 && y < height; long p = position(x, y) * 8; int mapN = (int) (p / MAPPING_SIZE); int offN = (int) (p % MAPPING_SIZE); mappings.get(mapN).putDouble(offN, d); } public void close() throws IOException { for (MappedByteBuffer mapping : mappings) clean(mapping); raf.close(); } private void clean(MappedByteBuffer mapping) { if (mapping == null) return; Cleaner cleaner = ((DirectBuffer) mapping).cleaner(); if (cleaner != null) cleaner.clean(); } } public class LargeDoubleMatrixTest { @Test public void getSetMatrix() throws IOException { long start = System.nanoTime(); final long used0 = usedMemory(); LargeDoubleMatrix matrix = new LargeDoubleMatrix("ldm.test", 1000 * 1000, 1000 * 1000); for (int i = 0; i < matrix.width(); i++) matrix.set(i, i, i); for (int i = 0; i < matrix.width(); i++) assertEquals(i, matrix.get(i, i), 0.0); long time = System.nanoTime() - start; final long used = usedMemory() - used0; if (used == 0) System.err.println("You need to use -XX:-UsedTLAB to see small changes in memory usage."); System.out.printf("Setting the diagonal took %,d ms, Heap used is %,d KB%n", time / 1000 / 1000, used / 1024); matrix.close(); } private long usedMemory() { return Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory(); } } With the following test, which writes to each of the diagonal values of a one million * one million matrix. This is too large to hope to create on the heap. Setting the diagonal took 314,819 ms, Heap used is 2,025 KB $ ls -l ldm.test -rw-rw-r-- 1 peter peter 8000000000000 2011-12-30 12:42 ldm.test $ du -s ldm.test 4010600 ldm.test That's 8,000,000,000,000 bytes or ~7.3 TB in virtual memory, in a Java process! This works because it only allocates or pages in the pages which you use. So while the file size is almost 8 TB, the actual disk space and memory used is 4 GB. With a more modest file size of 100K * 100K matrix you see something like the following. Its still an 80 GB matrix, which uses trivial heap space. ;) Setting the diagonal took 110 ms, Heap used is 71 KB $ ls -l ldm.test -rw-rw-r-- 1 peter peter 80000000000 2011-12-30 12:49 ldm.test $ du -s ldm.test 400000 ldm.test From http://vanillajava.blogspot.com/2011/12/using-memory-mapped-file-for-huge.html

This is an excerpt from the upcoming Manning book about Big Data. Big Data Principles and Best Practices of Scalable Realtime Data Systems By Nathan Marz and Samuel E. Ritchie Thrift is a widely used project that originated at Facebook. It can be used for making language-neutral RPC servers, but developers use it for its schema-creation capabilities. In this article based on chapter 2, author Nathan Marz discusses workhorses of Thrift—the struct and union type definitions—and Thrift’s built-in mechanisms for evolving a schema over time. You may also be interested in… Thrift is a widely used project that originated at Facebook. It can be used for making language-neutral RPC servers, but developers use it for its schema-creation capabilities. The workhorses of Thrift are the struct and union type definitions, and Thrift has built-in mechanisms for evolving a schema over time. Orginally Authored by Nathan Marz and Samuel E. Ritchie Structs The following code shows how to define a struct using the Thrift Interface Definition Language (IDL). Defining a struct is like defining a class in an object-oriented language: you specify all the data the object contains. The difference is that a Thrift struct only contains data and doesn't specify any extra behavior for the object. Fields in a struct can be: Primitive types like strings, ints, longs, and doubles. In the Thrift IDL, these are referred to as string, i32, i64, and double, respectively. Collections of other types. Thrift supports list, map, and set. Another Thrift struct or union. struct Person { 1: string twitter_username; 2: string full_name; 3: list interests; } The following code listing shows how to serialize a struct with Java. As you can see, we're using ArrayList, a native Java data structure, as part of the Person object. List interests = new ArrayList() {{ add("hadoop"); add("nosql"); }; Person person = new Person("joesmith", "Joe Smith", interests); TSerializer serializer = new TSerializer(); byte[] serialized = serializer.serialize(person); Here's how to deserialize a Person object in Python. When the object is deserialized, it will be using native Python data structures for any collection types. person = Person() deserialize(person, serialized_bytes) Fields in structs can be defined as being either required or optional. If a field is defined as required, than a value for that field must be provided or else Thrift will give an error upon serialization or deserialization. If a field is optional, the value will be null if not provided. You should always declare fields as being either required or optional. The following code listing shows how to define a struct containing required and optional fields. struct Tweet { 1: required string text; 2: required i64 id; 3: required i64 timestamp; 4: required Person person; 5: optional i64 response_to_tweet_id; Unions You can also define unions in Thrift. A union is a struct that must have exactly one field set. Unions are useful for representing polymorphic data. The following listing shows how to define a "PersonID" using a Thrift union that can be one of many different kinds of identifiers. union PersonID { 1: string email; 2: i64 facebook_id; 3: i64 twitter_id; } Evolving a schema Thrift is designed so that schemas can be evolved over time. The key to evolving Thrift schemas over time is the numeric identifiers used for every field. Those ids are used to identify fields in their serialized form. When you want to change the schema but still be backward compatible with existing data, you must obey the following rules. Fields may be renamed. This is because the serialized form of an object uses the field ids to identify fields, not the names. Fields may be removed, but you must be sure never to reuse that field id. When deserializing, Thrift will skip over any fields that don't match an id it's expecting. So the data for that field will just be ignored in the existing data. If you were to reuse that field id, Thrift will try to deserialize that old data into your new field which will lead to either invalid or incorrect data. Only optional fields can be added to existing structs. You can't add required fields because existing data won't have that field and will not be deserializable. Note that this point does not apply to unions since unions have no notion of required and optional fields. Summary In a relational database, the schema language is part of the database system and is integrated with how the database stores and processes that data. In the Big Data world, you use your own serialization framework that's separate from the storage and processing pieces. You get the flexibility to fine-tune this component to work exactly as needed to fit your data model. There are a few different open source serialization frameworks available, namely Thrift, Protocol Buffers, and Avro. We discussed our favorite, Apache Thrift, because it’s mature and supports most languages, but you could use any of these tools for defining a schema. Here are some other Manning titles you might be interested in: MongoDB in Action Kyle Banker RabbitMQ in Action Alvaro Videla and Jason J.W. Williams Hadoop in Action Chuck Lam Last updated: January 11, 2012

in the graph database space, there are two types of traversal engines: local and distributed. local traversal engines are typically for single-machine graph databases and are used for real-time production applications. distributed traversal engines are typically for multi-machine graph databases and are used for batch processing applications. this divide is quite sharp in the community, but there is nothing that prevents the unification of both models. a discussion of this divide and its unification is presented in this post. local traversal engines in a local traversal engine, there is typically a single processing agent that obeys a program. the agent is called a traverser and the program it follows is called a path description . in gremlin , a friend-of-a-friend path description is defined as such: g.v(1).oute('friend').inv.oute('friend').inv when this path description is interpreted by a traverser over a graph, an instance of the description is realized as actual paths in the graph that match that description. for example, the gremlin traverser starts at vertex 1 and then steps to its outgoing friend -edges. next, it will move to the head/target vertices of those edges (i.e. vertices 3 and 4). after that, it will go to the friend -edges of those vertices and finally, to the head of the previous edges (i.e. vertices 6 and 7). in this way, a single traverser is following all the paths that are exposed with each new atomic graph operation (i.e. each new step after a .). the abstract syntax being: step.step.step . what is returned by this friend-of-a-friend path description, on the example graph diagrammed, is vertices 6 and 7. in many situations, its not the end of the path that is desired, but some side-effect of the traversal. for example, as the traverser walks it can update some global data structure such as a ranking of the vertices. this idea is presented in the path description below, where the oute.inv path is looped over 1000 times. each time a vertex is traversed over, the map m is updated. this global map m maintains keys that are vertices and values that denote the number of times that each vertex has been touched ( groupcount ‘s behavior). m = [:] g.v(1).oute.inv.groupcount(m).loop(3){it.loops < 1000} the local traversal engine pattern is abstractly diagrammed on the right, where a single traverser is obeying some path description ( a.b.c ) and in doing so, moving around on a graph and updating a global data structure (the red boxed map). given the need for traversers to move from element to element, graph databases of this form tend to support strong data locality by means of a direct-reference graph data structure (i.e. vertices have pointers to edges and edges to vertices). a few examples of such graph databases include neo4j , orientdb , and dex . distributed traversal engines in a distributed traversal engine, a traversal is represented as a flow of messages between the elements of the graph. generally, each element (e.g. vertex) is operating independently of the other elements. each element is seen as its own processor with its own (usually homogenous) program to execute. elements communicate with each other via message passing . when no more messages have been passed, the traversal is complete and the results of the traversal are typically represented as a distributed data structure over the elements. graph databases of this nature tend to use the bulk synchronous parallel model of distributed computing. each step is synchronized in a manner analogous to a clock cycle in hardware. instances of this model include agrapa , pregel , trinity , and goldenorb . an example of distributed graph traversing is now presented using a ranking algorithm in java. [ note : in this example, edges are not first class citizens. this is typical of the state of the art in distributed traversal engines. they tend to be for single-relational, unlabeled-edge graphs.] public void evaluatestep(int step) { if(!this.inbox.isempty() && step < 1000) { this.rank = this.rank + this.inbox.size(); for(vertex vertex : this.adjacentvertices()) { for(int i=0; i

I’ve been developing for quite a while and in quite a few languages. Somehow though, I’ve always seemed to fall back to Java when doing my own stuff – maybe partly from habit, partly because it has in my opinion the best open source selection out there, and party because I liked its mix of features and performance. Originally authored by Matan Amir Specifically though, in the web arena things have been moving fast and furious with new languages, approaches, and methods like RoR, Play!, and Lift (and many others!). While I “get it” regarding the benefits of these frameworks, I never felt the need to give them more than an initial deep dive to see how they work. I nod a few times at their similarities and move on back to plain REST-ful services in Java with Spring, maybe an ORM (i try to avoid them these days), and a JS-rich front-end. Recently, two factors made me deep dive into Node.js. First is my growing appreciation with the progress of the JavaScript front-end. What used to be little JavaScript snippets to validate a form “onSubmit” have evolved to a complete ecosystem of technologies, frameworks, and stacks. My personal favorite these days is Backbone.js and try to use it whenever I can. Second was the first hand feedback I got from a friend at Voxer about their success in using and deploying Node.js at real scale (they have a pretty big Node.js deployment). So I jumped in. In the short time I’ve been using it for some (real) projects, I can say that it’s my new favorite language of choice. First of all, the event-driven (and non-blocking) model is a perfect fit for server side development. While this has existed in the other languages and forms (Java Servlet 3.0, Event Machine, Twisted to name a few), I connected with the easy of use and natural maturity of it in JavaScript. We’re all used to using callbacks anyway from your typical run-of-the-mill AJAX work right? Second is the community and how large its gotten in the short time Node has been around. There are lots of useful open source libraries to use to solve your problems and the quality level is high. Third is that it was just so easy to pick up. I have to admit that my core JavaScript was decent before I started using Node, but in terms of the Node core library and feature set itself, it’s pretty lean and mean and provides a good starting point to build what you need in case you can’t find someone who already did (which you most likely can). So having said all that, I wanted to share what resources helped me get up to speed in Node.js coming from a Java background. Getting to know JavaScript There are lots of good resources out there on Node.js and i’ve listed them below. For me, the most critical piece was getting my JavaScript knowledge to the next level. Because you’ll be spending all your time writing your server code in JavaScript, it pays huge dividends to understand how to take full advantage of it. As a Java developer you’re probably trained to think in OO designs. With me, this was the part that I focused the most on. Fortunately (or unforunately), JavaScript is not a classic OO language. It can be if you shoehorn it, but i think that defeats the purpose. Here is my short list of JavaScript resources: JavaScript: The Good Parts - Definitely a requirement. Chapters 4 and 5 (functions, objects, and inheritance) are probably the most important part to understand well for those with an OO background. Learning Javascript with Object Graphs (part 2, and part 3) – howtonode.org has lots of good Node material, but this 3 part post is a good resource to top off your knowledge from the book. Simple “Class” Instantiation – Another good post I read recently. Worth digesting. Learning Node.js With a good JavaScript background, starting to use Node.js is pretty straightforward. The main part to understand is the asynchronous nature of the I/O and the need to produce and consume events to get stuff done. Here is a list of resources I used to get up to speed: DailyJS’s Node Tutorial - A multipart tutorial for Node on DailyJS’s blog. It’s a great resource and worth going through all the posts. Mixu’s Node Book - Not complete, but still worth it. I look forward to reading the future chapters. Node Beginner Book – A good starter read. How To Node – A blog dedicated to Node.js. Bookmark it. I felt that going through these was more than enough to give me the push I needed get started. Hopefully it does for you too (thanks to the authors for taking the time to share their knowledge!). Frameworks? Java is all about open source frameworks. That’s part of why it’s so popular. While Node.js is much newer, lots of people have already done quite a bit of heavy-lifting and have shared their code with the world. Below is what I think is a good mapping of popular Java frameworks to their Node.js equivalents (from what I know so far). Web MVC In Java land, most people are familiar with Web MVC frameworks like Spring MVC, Struts, Wicket, and JSF. More recently though, the trend towards client-side JS MVC frameworks like Ember.js (SproutCore) and Backbone.js reduces the required feature-set of some of these frameworks. Nonetheless, a good comparable Node.js web framework is Express. In a sense, it’s even more than a web framework because it also provides most of the “web server” functionality most Java developers are used to through Tomcat, Jetty, etc (more specifically, Express builds on top of Connect) It’s well thought out and provides the feature set needed to get things done. Application Lifecycle Framework (DI Framework) Spring is a popular framework in Java to provide a great deal of glue and abstraction functionality. It allows easy dependency injection, testing, object lifecycle management, transaction management, etc. It’s usually one of the first things I slap into a new project in Java. In Node.js… I haven’t really missed it. JavaScript allows much more flexible ways of decoupling dependencies and I personally haven’t felt the need to find a replacement for Spring. Maybe that’s a good thing? Object-Relational Mapping (ORMs) I have mixed feelings regarding ORMs in general, but it does make your life easier sometimes. There is no shortage of ORM librares for Node.js. Take your pick. Package Management Tools Maven is probably most popular build management tool for Java. While it is very flexible and powerful with a wide variety of plug-ins, it can get very cumbersome. Npm is the mainstream package manager for Node.js. It’s light, fast, and useful. Testing Framework Java has lots of these for sure, jUnit and company as standard. There are also mock libraries, stub libraries, db test libraries, etc. Node.js has quite a few as well. Pick your poison. I see that nodeunit is popular and is similar to jUnit. I’m personally testing Mocha. Testing tools are a more personal and subjective choice, but the good thing is that there definitely are good choices out there. Logging Java developers have quite a list of choices when choosing a logger library. Commons logging, log4j, logback, and slf4j (wrapper) are some of the more popular ones. Node.js also has a few. I’m currently using winston and have no complaints so far. It has the logging levels, multiple transports (appenders to log4j people), and does it all asynchronously as well. Hopefully this will help someone save some time when peeking into the world of Node. Good luck! Source: http://n0tw0rthy.wordpress.com/2012/01/08/from-java-to-node-js/

Learn the differences between GET and POST requests.

Clojure is a LISP dialect, and as such a functional language based on a large set of functions and a small set of data structures that they operate with. However, it is possible to implement classes and object in Clojure, with constructs well-supported in the language itself. I do not claim you should program in Clojure only with the techniques described in this article: they are just an attempt to bridge Java libraries with Clojure, and to introduce objects and interfaces where needed. Java interoperability Even from the Clojure REPL, you can easily instantiate Java classes as long as they are in the classpath (so use lein if they are in a library) (def ford (Car. arg1 arg2)) Since classes usually reside into packages, in real code you would use their fully qualified name: (def ford (com.dzone.example.Car. arg1 arg2)) You can also call methods: (.brake ford arg1 arg2) The first argument of the evaluation of a .methodName is always the object, while additional arguments are placed after it, in the same s-expression (a fancy name for a list that can be evaluated.) One of the first exercises in the book The Joy of Clojure is an instantiation of a java.awt.Frame object and a rendering done on it, all from the Clojure interactive interpreter. Doing exploratory testing classes and objects for Java development with the REPL is as faster as writing code into JUnit test. Define your own: interfaces defprotocol takes as the first argument an identifier, and a variable number of parameters. Each of this parameters is a method definition like you would do with fn, but without the fn keyword itself and a body to evaluate. The first argument of the methods should always be this, representing the current object (remember Python?). (defprotocol Position (description [this]) (translateX [this dx]) (translateY [this dy]) (doubleCoords [this]) (average [this another])) Define your own: classes Defining classes in Clojure is simple, at least when they implement immutable Value Objects. defrecord takes a class name, a list of parameters for the constructor, an optional protocol name and a variable number of arguments representing the methods. Again, the methods definition are similar to the ones made with fn, but this time with a body, an s-expression to evaluate. There is a catch: you can only implement methods defined in a protocol. (defrecord CartesianPoint [x y] Position (description [this] (str "(x=" x ", y=" y ")")) (translateX [this dx] (CartesianPoint. (+ x dx) y)) (translateY [this dy] (CartesianPoint. x (+ y dy))) (doubleCoords [this] (CartesianPoint. (* 2 x) (* 2 y))) (average [this another] (let [mean (fn [a b] (/ (+ a b) 2))] (CartesianPoint. (mean x (:x another)) (mean y (:y another)))))) The method access the (immutable) fields of the current object with their names, so this is not probably necessary if not for calling other methods on the object. Note that records are not strictly objects in the OO sense, since they do not encapsulate their fields: fields can be accessed anywhere with (:fieldname object). Here's how to work with CartesianPoint objects: (deftest test-point-x-field (is (= 1 (:x (CartesianPoint. 1 2))))) (deftest test-point-y-field (is (= 2 (:y (CartesianPoint. 1 2))))) (deftest test-point-to-string (is (= "(x=1, y=2)" (description (CartesianPoint. 1 2))))) (deftest test-point-translation-x (is (= (CartesianPoint. 11 2) (translateX (CartesianPoint. 1 2) 10)))) (deftest test-point-translation-y (is (= (CartesianPoint. 1 12) (translateY (CartesianPoint. 1 2) 10)))) (deftest test-point-doubling (is (= (CartesianPoint. 2 4) (doubleCoords (CartesianPoint. 1 2))))) (deftest test-points-average (is (= (CartesianPoint. 3 4) (average (CartesianPoint. 1 2) (CartesianPoint. 5 6)))))

Lucene's 3.4.0 release adds a new feature called index-time join (also sometimes called sub-documents, nested documents or parent/child documents), enabling efficient indexing and searching of certain types of relational content. Most search engines can't directly index relational content, as documents in the index logically behave like a single flat database table. Yet, relational content is everywhere! A job listing site has each company joined to the specific listings for that company. Each resume might have separate list of skills, education and past work experience. A music search engine has an artist/band joined to albums and then joined to songs. A source code search engine would have projects joined to modules and then files. Perhaps the PDF documents you need to search are immense, so you break them up and index each section as a separate Lucene document; in this case you'll have common fields (title, abstract, author, date published, etc.) for the overall document, joined to the sub-document (section) with its own fields (text, page number, etc.). XML documents typically contain nested tags, representing joined sub-documents; emails have attachments; office documents can embed other documents. Nearly all search domains have some form of relational content, often requiring more than one join. If such content is so common then how do search applications handle it today? One obvious "solution" is to simply use a relational database instead of a search engine! If relevance scores are less important and you need to do substantial joining, grouping, sorting, etc., then using a database could be best overall. Most databases include some form a text search, some even using Lucene. If you still want to use a search engine, then one common approach is to denormalize the content up front, at index-time, by joining all tables and indexing the resulting rows, duplicating content in the process. For example, you'd index each song as a Lucene document, copying over all fields from the song's joined album and artist/band. This works correctly, but can be horribly wasteful as you are indexing identical fields, possibly including large text fields, over and over. Another approach is to do the join yourself, outside of Lucene, by indexing songs, albums and artist/band as separate Lucene documents, perhaps even in separate indices. At search-time, you first run a query against one collection, for example the songs. Then you iterate through all hits, gathering up (joining) the full set of corresponding albums and then run a second query against the albums, with a large OR'd list of the albums from the first query, repeating this process if you need to join to artist/band as well. This approach will also work, but doesn't scale well as you may have to create possibly immense follow-on queries. Yet another approach is to use a software package that has already implemented one of these approaches for you! elasticsearch, Apache Solr, Apache Jackrabbit, Hibernate Search and many others all handle relational content in some way. With BlockJoinQuery you can now directly search relational content yourself! Let's work through a simple example: imagine you sell shirts online. Each shirt has certain common fields such as name, description, fabric, price, etc. For each shirt you have a number of separate stock keeping units or SKUs, which have their own fields like size, color, inventory count, etc. The SKUs are what you actually sell, and what you must stock, because when someone buys a shirt they buy a specific SKU (size and color). Maybe you are lucky enough to sell the incredible Mountain Three-wolf Moon Short Sleeve Tee, with these SKUs (size, color): small, blue small, black medium, black large, gray Perhaps a user first searches for "wolf shirt", gets a bunch of hits, and then drills down on a particular size and color, resulting in this query: name:wolf AND size=small AND color=blue which should match this shirt. name is a shirt field while the size and color are SKU fields. But if the user drills down instead on a small gray shirt: name:wolf AND size=small AND color=gray then this shirt should not match because the small size only comes in blue and black. How can you run these queries using BlockJoinQuery? Start by indexing each shirt (parent) and all of its SKUs (children) as separate documents, using the new IndexWriter.addDocuments API to add one shirt and all of its SKUs as a single document block. This method atomically adds a block of documents into a single segment as adjacent document IDs, which BlockJoinQuery relies on. You should also add a marker field to each shirt document (e.g. type = shirt), as BlockJoinQuery requires a Filter identifying the parent documents. To run a BlockJoinQuery at search-time, you'll first need to create the parent filter, matching only shirts. Note that the filter must use FixedBitSet under the hood, like CachingWrapperFilter: Filter shirts = new CachingWrapperFilter( new QueryWrapperFilter( new TermQuery( new Term("type", "shirt")))); Create this filter once, up front and re-use it any time you need to perform this join. Then, for each query that requires a join, because it involves both SKU and shirt fields, start with the child query matching only SKU fields: BooleanQuery skuQuery = new BooleanQuery(); skuQuery.add(new TermQuery(new Term("size", "small")), Occur.MUST); skuQuery.add(new TermQuery(new Term("color", "blue")), Occur.MUST); Next, use BlockJoinQuery to translate hits from the SKU document space up to the shirt document space: BlockJoinQuery skuJoinQuery = new BlockJoinQuery( skuQuery, shirts, ScoreMode.None); The ScoreMode enum decides how scores for multiple SKU hits should be aggregated to the score for the corresponding shirt hit. In this query you don't need scores from the SKU matches, but if you did you can aggregate with Avg, Max or Total instead. Finally you are now free to build up an arbitrary shirt query using skuJoinQuery as a clause: BooleanQuery query = new BooleanQuery(); query.add(new TermQuery(new Term("name", "wolf")), Occur.MUST); query.add(skuJoinQuery, Occur.MUST); You could also just run skuJoinQuery as-is if the query doesn't have any shirt fields. Finally, just run this query like normal! The returned hits will be only shirt documents; if you'd also like to see which SKUs matched for each shirt, use BlockJoinCollector: BlockJoinCollector c = new BlockJoinCollector( Sort.RELEVANCE, // sort 10, // numHits true, // trackScores false // trackMaxScore ); searcher.search(query, c); The provided Sort must use only shirt fields (you cannot sort by any SKU fields). When each hit (a shirt) is competitive, this collector will also record all SKUs that matched for that shirt, which you can retrieve like this: TopGroups hits = c.getTopGroups( skuJoinQuery, skuSort, 0, // offset 10, // maxDocsPerGroup 0, // withinGroupOffset true // fillSortFields ); Set skuSort to the sort order for the SKUs within each shirt. The first offset hits are skipped (use this for paging through shirt hits). Under each shirt, at most maxDocsPerGroup SKUs will be returned. Use withinGroupOffset if you want to page within the SKUs. If fillSortFields is true then each SKU hit will have values for the fields from skuSort. The hits returned by BlockJoinCollector.getTopGroups are SKU hits, grouped by shirt. You'd get the exact same results if you had denormalized up-front and then used grouping to group results by shirt. You can also do more than one join in a single query; the joins can be nested (parent to child to grandchild) or parallel (parent to child1 and parent to child2). However, there are some important limitations of index-time joins: The join must be computed at index-time and "compiled" into the index, in that all joined child documents must be indexed along with the parent document, as a single document block. Different document types (for example, shirts and SKUs) must share a single index, which is wasteful as it means non-sparse data structures like FieldCache entries consume more memory than they would if you had separate indices. If you need to re-index a parent document or any of its child documents, or delete or add a child, then the entire block must be re-indexed. This is a big problem in some cases, for example if you index "user reviews" as child documents then whenever a user adds a review you'll have to re-index that shirt as well as all its SKUs and user reviews. There is no QueryParser support, so you need to programmatically create the parent and child queries, separating according to parent and child fields. The join can currently only go in one direction (mapping child docIDs to parent docIDs), but in some cases you need to map parent docIDs to child docIDs. For example, when searching songs, perhaps you want all matching songs sorted by their title. You can't easily do this today because the only way to get song hits is to group by album or band/artist. The join is a one (parent) to many (children), inner join. As usual, patches are welcome! There is work underway to create a more flexible, but likely less performant, query-time join capability, which should address a number of the above limitations. Source: http://blog.mikemccandless.com/2012/01/searching-relational-content-with.html

Many applications record a series of events to file-based storage for later use. This can be anything from logging and auditing, through to keeping a transaction redo log in an event sourced design or its close relative CQRS. Java has a number of means by which a file can be sequentially written to, or read back again. This article explores some of these mechanisms to understand their performance characteristics. For the scope of this article I will be using pre-allocated files because I want to focus on performance. Constantly extending a file imposes a significant performance overhead and adds jitter to an application resulting in highly variable latency. "Why is a pre-allocated file better performance?", I hear you ask. Well, on disk a file is made up from a series of blocks/pages containing the data. Firstly, it is important that these blocks are contiguous to provide fast sequential access. Secondly, meta-data must be allocated to describe this file on disk and saved within the file-system. A typical large file will have a number of "indirect" blocks allocated to describe the chain of data-blocks containing the file contents that make up part of this meta-data. I'll leave it as an exercise for the reader, or maybe a later article, to explore the performance impact of not preallocating the data files. If you have used a database you may have noticed that it preallocates the files it will require. The Test I want to experiment with 2 file sizes. One that is sufficiently large to test sequential access, but can easily fit in the file-system cache, and another that is much larger so that the cache subsystem is forced to retire pages so that new ones can be loaded. For these two cases I'll use 400MB and 8GB respectively. I'll also loop over the files a number of times to show the pre and post warm-up characteristics. I'll test 4 means of writing and reading back files sequentially: RandomAccessFile using a vanilla byte[] of page size. Buffered FileInputStream and FileOutputStream. NIO FileChannel with ByteBuffer of page size. Memory mapping a file using NIO and direct MappedByteBuffer. The tests are run on a 2.0Ghz Sandybridge CPU with 8GB RAM, an Intel 230 SSD on Fedora Core 15 64-bit Linux with an ext4 file system, and Oracle JDK 1.6.0_30. The Code import java.io.*; import java.nio.ByteBuffer; import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel; import static java.lang.Integer.MAX_VALUE; import static java.lang.System.out; import static java.nio.channels.FileChannel.MapMode.READ_ONLY; import static java.nio.channels.FileChannel.MapMode.READ_WRITE; public final class TestSequentialIoPerf { public static final int PAGE_SIZE = 1024 * 4; public static final long FILE_SIZE = PAGE_SIZE * 2000L * 1000L; public static final String FILE_NAME = "test.dat"; public static final byte[] BLANK_PAGE = new byte[PAGE_SIZE]; public static void main(final String[] arg) throws Exception { preallocateTestFile(FILE_NAME); for (final PerfTestCase testCase : testCases) { for (int i = 0; i < 5; i++) { System.gc(); long writeDurationMs = testCase.test(PerfTestCase.Type.WRITE, FILE_NAME); System.gc(); long readDurationMs = testCase.test(PerfTestCase.Type.READ, FILE_NAME); long bytesReadPerSec = (FILE_SIZE * 1000L) / readDurationMs; long bytesWrittenPerSec = (FILE_SIZE * 1000L) / writeDurationMs; out.format("%s\twrite=%,d\tread=%,d bytes/sec\n", testCase.getName(), bytesWrittenPerSec, bytesReadPerSec); } } deleteFile(FILE_NAME); } private static void preallocateTestFile(final String fileName) throws Exception { RandomAccessFile file = new RandomAccessFile(fileName, "rw"); for (long i = 0; i < FILE_SIZE; i += PAGE_SIZE) { file.write(BLANK_PAGE, 0, PAGE_SIZE); } file.close(); } private static void deleteFile(final String testFileName) throws Exception { File file = new File(testFileName); if (!file.delete()) { out.println("Failed to delete test file=" + testFileName); out.println("Windows does not allow mapped files to be deleted."); } } public abstract static class PerfTestCase { public enum Type { READ, WRITE } private final String name; private int checkSum; public PerfTestCase(final String name) { this.name = name; } public String getName() { return name; } public long test(final Type type, final String fileName) { long start = System.currentTimeMillis(); try { switch (type) { case WRITE: { checkSum = testWrite(fileName); break; } case READ: { final int checkSum = testRead(fileName); if (checkSum != this.checkSum) { final String msg = getName() + " expected=" + this.checkSum + " got=" + checkSum; throw new IllegalStateException(msg); } break; } } } catch (Exception ex) { ex.printStackTrace(); } return System.currentTimeMillis() - start; } public abstract int testWrite(final String fileName) throws Exception; public abstract int testRead(final String fileName) throws Exception; } private static PerfTestCase[] testCases = { new PerfTestCase("RandomAccessFile") { public int testWrite(final String fileName) throws Exception { RandomAccessFile file = new RandomAccessFile(fileName, "rw"); final byte[] buffer = new byte[PAGE_SIZE]; int pos = 0; int checkSum = 0; for (long i = 0; i < FILE_SIZE; i++) { byte b = (byte)i; checkSum += b; buffer[pos++] = b; if (PAGE_SIZE == pos) { file.write(buffer, 0, PAGE_SIZE); pos = 0; } } file.close(); return checkSum; } public int testRead(final String fileName) throws Exception { RandomAccessFile file = new RandomAccessFile(fileName, "r"); final byte[] buffer = new byte[PAGE_SIZE]; int checkSum = 0; int bytesRead; while (-1 != (bytesRead = file.read(buffer))) { for (int i = 0; i < bytesRead; i++) { checkSum += buffer[i]; } } file.close(); return checkSum; } }, new PerfTestCase("BufferedStreamFile") { public int testWrite(final String fileName) throws Exception { int checkSum = 0; OutputStream out = new BufferedOutputStream(new FileOutputStream(fileName)); for (long i = 0; i < FILE_SIZE; i++) { byte b = (byte)i; checkSum += b; out.write(b); } out.close(); return checkSum; } public int testRead(final String fileName) throws Exception { int checkSum = 0; InputStream in = new BufferedInputStream(new FileInputStream(fileName)); int b; while (-1 != (b = in.read())) { checkSum += (byte)b; } in.close(); return checkSum; } }, new PerfTestCase("BufferedChannelFile") { public int testWrite(final String fileName) throws Exception { FileChannel channel = new RandomAccessFile(fileName, "rw").getChannel(); ByteBuffer buffer = ByteBuffer.allocate(PAGE_SIZE); int checkSum = 0; for (long i = 0; i < FILE_SIZE; i++) { byte b = (byte)i; checkSum += b; buffer.put(b); if (!buffer.hasRemaining()) { channel.write(buffer); buffer.clear(); } } channel.close(); return checkSum; } public int testRead(final String fileName) throws Exception { FileChannel channel = new RandomAccessFile(fileName, "rw").getChannel(); ByteBuffer buffer = ByteBuffer.allocate(PAGE_SIZE); int checkSum = 0; while (-1 != (channel.read(buffer))) { buffer.flip(); while (buffer.hasRemaining()) { checkSum += buffer.get(); } buffer.clear(); } return checkSum; } }, new PerfTestCase("MemoryMappedFile") { public int testWrite(final String fileName) throws Exception { FileChannel channel = new RandomAccessFile(fileName, "rw").getChannel(); MappedByteBuffer buffer = channel.map(READ_WRITE, 0, Math.min(channel.size(), MAX_VALUE)); int checkSum = 0; for (long i = 0; i < FILE_SIZE; i++) { if (!buffer.hasRemaining()) { buffer = channel.map(READ_WRITE, i, Math.min(channel.size() - i , MAX_VALUE)); } byte b = (byte)i; checkSum += b; buffer.put(b); } channel.close(); return checkSum; } public int testRead(final String fileName) throws Exception { FileChannel channel = new RandomAccessFile(fileName, "rw").getChannel(); MappedByteBuffer buffer = channel.map(READ_ONLY, 0, Math.min(channel.size(), MAX_VALUE)); int checkSum = 0; for (long i = 0; i < FILE_SIZE; i++) { if (!buffer.hasRemaining()) { buffer = channel.map(READ_WRITE, i, Math.min(channel.size() - i , MAX_VALUE)); } checkSum += buffer.get(); } channel.close(); return checkSum; } }, }; } Results 400MB file =========== RandomAccessFile write=379,610,750 read=1,452,482,269 bytes/sec RandomAccessFile write=294,041,636 read=1,494,890,510 bytes/sec RandomAccessFile write=250,980,392 read=1,422,222,222 bytes/sec RandomAccessFile write=250,366,748 read=1,388,474,576 bytes/sec RandomAccessFile write=260,394,151 read=1,422,222,222 bytes/sec BufferedStreamFile write=98,178,331 read=286,433,566 bytes/sec BufferedStreamFile write=100,244,738 read=288,857,545 bytes/sec BufferedStreamFile write=82,948,562 read=154,100,827 bytes/sec BufferedStreamFile write=108,503,311 read=153,869,271 bytes/sec BufferedStreamFile write=113,055,478 read=152,608,047 bytes/sec BufferedChannelFile write=388,246,445 read=358,041,958 bytes/sec BufferedChannelFile write=390,467,111 read=375,091,575 bytes/sec BufferedChannelFile write=321,759,622 read=1,539,849,624 bytes/sec BufferedChannelFile write=318,259,518 read=1,539,849,624 bytes/sec BufferedChannelFile write=322,265,932 read=1,534,082,397 bytes/sec MemoryMappedFile write=300,955,180 read=305,899,925 bytes/sec MemoryMappedFile write=313,149,847 read=310,538,286 bytes/sec MemoryMappedFile write=326,374,501 read=303,857,566 bytes/sec MemoryMappedFile write=327,680,000 read=304,535,315 bytes/sec MemoryMappedFile write=326,895,450 read=303,632,320 bytes/sec 8GB File ============ RandomAccessFile write=167,402,321 read=251,922,012 bytes/sec RandomAccessFile write=193,934,802 read=257,052,307 bytes/sec RandomAccessFile write=192,948,159 read=248,460,768 bytes/sec RandomAccessFile write=191,814,180 read=245,225,408 bytes/sec RandomAccessFile write=190,635,762 read=275,315,073 bytes/sec BufferedStreamFile write=154,823,102 read=248,355,313 bytes/sec BufferedStreamFile write=152,083,913 read=253,418,301 bytes/sec BufferedStreamFile write=133,099,369 read=146,056,197 bytes/sec BufferedStreamFile write=131,065,708 read=146,217,827 bytes/sec BufferedStreamFile write=132,694,052 read=148,116,004 bytes/sec BufferedChannelFile write=406,147,744 read=304,693,892 bytes/sec BufferedChannelFile write=397,457,668 read=298,183,671 bytes/sec BufferedChannelFile write=364,672,364 read=414,281,379 bytes/sec BufferedChannelFile write=371,266,711 read=404,343,534 bytes/sec BufferedChannelFile write=373,705,579 read=406,934,578 bytes/sec MemoryMappedFile write=123,023,322 read=231,530,156 bytes/sec MemoryMappedFile write=121,961,023 read=230,403,600 bytes/sec MemoryMappedFile write=123,317,778 read=229,899,250 bytes/sec MemoryMappedFile write=121,472,738 read=231,739,745 bytes/sec MemoryMappedFile write=120,362,615 read=231,190,382 bytes/sec Analysis For years I was a big fan of using RandomAccessFile directly because of the control it gives and the predictable execution. I never found using buffered streams to be useful from a performance perspective and this still seems to be the case. In more recent testing I've found that using NIO FileChannel and ByteBuffer are the clear winners from a performance perspective. With Java 7 the flexibility of this programming approach has been improved for random access with SeekableByteChannel. I've seen these results vary greatly depending on platform. File system, OS, storage devices, and available memory all have a significant impact. In a few cases I've seen memory-mapped files perform significantly better than the others but this needs to be tested on your platform because your mileage may vary... A special note should be made for the use of memory-mapped large files when pushing for maximum throughput. I've often found the OS can become unresponsive due the the pressure put on the virtual memory sub-system. Conclusion There is a significant difference in performance for the different means of doing sequential file IO from Java. Not all methods are even remotely equal. For most IO I've found the use of ByteBuffers and Channels to be the best optimised parts of the IO libraries. If buffered streams are your IO libraries of choice, then it is worth branching out and and getting familiar with the sub-classes of Channel and Buffer. From http://mechanical-sympathy.blogspot.com/2011/12/java-sequential-io-performance.html

clover is a great tool for generating code coverage reports from your unit tests. it can be executed as a plugin in eclipse, maven or ant. however, not everyone knows that it can also be used to collect coverage data of integration tests. this post explains how to collect coverage data with clover at runtime. this post assumes that you already know what are unit and integration tests. this post assumes that you know what clover is, and already used it either with eclipse, ant or maven. * let me assured you that even though the directions bellow seems complicated and clumsy at first, after doing them once or twice it is really easy to repeat them. motivation the default action of clover is to gather code coverage information during build time or compile time. therefore, this information includes just the coverage data created by unit tests. if you are developing web applications, you probably use more technologies to test your applications beside unit tests. these technologies may include httpunit/ htmlunit or automation technologies (like selenium). these technologies do not work at build time, they can only work during run time, where a web server is up and running and http calls are made. as a result, the code coverage made during build time is not reflecting the actual code coverage. we should be able to test the coverage while a server is running. the idea the idea is that we will first run clover regularly during build time. we will than take the clover artifacts, put them in our server and then run the integration tests. while running the integration tests, the clover database will be updated and we would be able to generate reports from it which will reflect both unit and integration tests. step 1 – preparation make sure that you have a web/application server (tomcat/jboss/weblogic…) with your web application already deployed. execute clover on your application as you would normally do (either by compiling the code on eclipse or by building with maven or ant, it doesn’t matter). the result of this action would be: clover db files. one of the outcome of executing the clover on your code are the db files. the db files hold all the information about your code and the coverage itself. the location of those files may change depending on the way you use clover and according to the way you configured clover in your environment. this is how the files looks like the .db file holds the information regarding your code (classes, methods and so on). all the other files hold all the coverage data. it is important that you will locate those files because we are going to use them in the next step. an instrumented code. another outcome of clover is that it instruments your code. a clover call is injected into each method so it would be reported in the coverage calculation. we will need this code. we will use this instrumented code in runtime to update the coverage data. if you use eclipse than the generated classes would be instrumented. if you use maven or ant than most chances are that a jar with all the instrumented code would be generated separately. search the instrument code jar. again, i can’t tell you exactly where it is located, but usually it generates a jar with a ‘clover’ postfix. example: if your jar name is my_app.jar, than the generated instrumented code jar will probably be something like my_app-clover.jar. so you will need to do some detective work here to find the instrumented classes/jar. if you are not sure whether the classes are instrumented or not, just decompile one of them with jad and search for the word clover inside of it. a code coverage report. this is a report with the unit test coverage. we don’t really need this, but it would be good so that we would be able to compare it with the report we will generate at the end. step 2 – updating the server the next steps are very important, please make sure you do them properly. replacing the existing application jar/classes with the instrumented jar/classes. take the instrumented jar/classes that were created by the clover and add it to your server’s classpath instead of the original jar/classes. the instrumented code will cause the clover db to be updated with the runtime data. adding clover jars and license. since we will use clover on runtime we will need also the clover jars in our server’s classpath. so add the clover own jars to your server’s classpath. if you are using clover with eclipse than these jars are located in the plugin folder of eclipse. if you are using maven than they will be loacted in your repository. * make sure you are using the same version of clover in your server as you used to generate the db files and instrumented code, otherwise it will not work and you will get error messages. also add the clover license file to the same location as the jars. adding the clover java argument. add the following java arg -dclover.initstring.basedir={ location of the db files that were created by the clover } . * notice – the path you have entered above is the path of the folder which contains the db files. example: -dclover.initstring.basedir=c:/workspace/mywebapp/target/clover. this java argument is used by the clover to locate the db files and update them. step 3 – restarting the server and running the tests now that hopefully all is set properly all that you need to do now is to restart your server and than running your integration tests. the tests should trigger the instrumented code which will call the clover api’s and will update the clover db. while running the tests: look at your log/console and search for error messages from clover. look at the folder which holds the clover db files. if everything is going as it should, new files will be created in this folder while running the tests. if not everything is going well the first time, don’t discourage, just go over each of the steps again. step 4 – generating an updated report if everything went well and new files were created in the db folder than that means you just need to generate a new report. if you are using clover with eclipse than you can simply push the reload button to reload the coverage data. if you are using maven or ant you can execute just the task which generates the report. another way is to use the clover htmlreporter to generate a report easily. now compare the new report to the old report. you should see that the new report coverage is much bigger than the old one since it contains also the integration tests coverage. * notice that not all the data is updated, even though the percentages are being updated, for some reason the calls counter does not. to summarize. as mentioned; yes, these instructions seems a bit complicated but after you succeed the first time, it is very easy to repeat it. in the company i work for we even made this whole process automatic and we are able to generate a full coverage report with unit and integrated tests combined. source: http://www.aviyehuda.com/2011/12/use-clover-to-generate-code-coverage-reports-of-your-integrationautomation-tests/

Introduction A common thing I come across is that teams using a mocking framework assume they are mocking. They are not aware that Mocks are just one of a number of 'Test Doubles' which Gerard Meszaros has categorised at xunitpatterns.com. It’s important to realise that each type of test double has a different role to play in testing. In the same way that you need to learn different patterns or refactoring’s, you need to understand the primitive roles of each type of test double. These can then be combined to achieve your testing needs. I'll cover a very brief history of how this classification came about, and how each of the types differs. I'll do this using some short, simple examples in Mockito. A Very Brief History For years people have been writing lightweight versions of system components to help with testing. In general it was called stubbing. In 2000' the article 'Endo-Testing: Unit Testing with Mock Objects' introduced the concept of a Mock Object. Since then Stubs, Mocks and a number of other types of test objects have been classified by Meszaros as Test Doubles. This terminology has been referenced by Martin Fowler in "Mocks Aren't Stubs" and is being adopted within the Microsoft community as shown in "Exploring The Continuum of Test Doubles" A link to each of these important papers are shown in the reference section. Categories of test doubles The diagram above shows the commonly used types of test double. The following URL gives a good cross reference to each of the patterns and their features as well as alternative terminology. http://xunitpatterns.com/Test%20Double.html Mockito Mockito is a test spy framework and it is very simple to learn. Notable with Mockito is that expectations of any mock objects are not defined before the test as they sometimes are in other mocking frameworks. This leads to a more natural style(IMHO) when beginning mocking. The following examples are here purely to give a simple demonstration of using Mockito to implement the different types of test doubles. There are a much larger number of specific examples of how to use Mockito on the website. http://docs.mockito.googlecode.com/hg/latest/org/mockito/Mockito.html Test Doubles with Mockito Below are some basic examples using Mockito to show the role of each test double as defined by Meszaros. I’ve included a link to the main definition for each so you can get more examples and a complete definition. Dummy Object http://xunitpatterns.com/Dummy%20Object.html This is the simplest of all of the test doubles. This is an object that has no implementation which is used purely to populate arguments of method calls which are irrelevant to your test. For example, the code below uses a lot of code to create the customer which is not important to the test. The test couldn't care less which customer is added, as long as the customer count comes back as one. public Customer createDummyCustomer() { County county = new County("Essex"); City city = new City("Romford", county); Address address = new Address("1234 Bank Street", city); Customer customer = new Customer("john", "dobie", address); return customer; } @Test public void addCustomerTest() { Customer dummy = createDummyCustomer(); AddressBook addressBook = new AddressBook(); addressBook.addCustomer(dummy); assertEquals(1, addressBook.getNumberOfCustomers()); } We actually don't care about the contents of customer object - but it is required. We can try a null value, but if the code is correct you would expect some kind of exception to be thrown. @Test(expected=Exception.class) public void addNullCustomerTest() { Customer dummy = null; AddressBook addressBook = new AddressBook(); addressBook.addCustomer(dummy); } To avoid this we can use a simple Mockito dummy to get the desired behaviour. @Test public void addCustomerWithDummyTest() { Customer dummy = mock(Customer.class); AddressBook addressBook = new AddressBook(); addressBook.addCustomer(dummy); Assert.assertEquals(1, addressBook.getNumberOfCustomers()); } It is this simple code which creates a dummy object to be passed into the call. Customer dummy = mock(Customer.class); Don't be fooled by the mock syntax - the role being played here is that of a dummy, not a mock. It's the role of the test double that sets it apart, not the syntax used to create one. This class works as a simple substitute for the customer class and makes the test very easy to read. Test stub http://xunitpatterns.com/Test%20Stub.html The role of the test stub is to return controlled values to the object being tested. These are described as indirect inputs to the test. Hopefully an example will clarify what this means. Take the following code public class SimplePricingService implements PricingService { PricingRepository repository; public SimplePricingService(PricingRepository pricingRepository) { this.repository = pricingRepository; } @Override public Price priceTrade(Trade trade) { return repository.getPriceForTrade(trade); } @Override public Price getTotalPriceForTrades(Collection trades) { Price totalPrice = new Price(); for (Trade trade : trades) { Price tradePrice = repository.getPriceForTrade(trade); totalPrice = totalPrice.add(tradePrice); } return totalPrice; } The SimplePricingService has one collaborating object which is the trade repository. The trade repository provides trade prices to the pricing service through the getPriceForTrade method. For us to test the business logic in the SimplePricingService, we need to control these indirect inputs i.e. inputs we never passed into the test. This is shown below. In the following example we stub the PricingRepository to return known values which can be used to test the business logic of the SimpleTradeService. @Test public void testGetHighestPricedTrade() throws Exception { Price price1 = new Price(10); Price price2 = new Price(15); Price price3 = new Price(25); PricingRepository pricingRepository = mock(PricingRepository.class); when(pricingRepository.getPriceForTrade(any(Trade.class))) .thenReturn(price1, price2, price3); PricingService service = new SimplePricingService(pricingRepository); Price highestPrice = service.getHighestPricedTrade(getTrades()); assertEquals(price3.getAmount(), highestPrice.getAmount()); } Saboteur Example There are 2 common variants of Test Stubs: Responder’s and Saboteur's. Responder's are used to test the happy path as in the previous example. A saboteur is used to test exceptional behaviour as below. @Test(expected=TradeNotFoundException.class) public void testInvalidTrade() throws Exception { Trade trade = new FixtureHelper().getTrade(); TradeRepository tradeRepository = mock(TradeRepository.class); when(tradeRepository.getTradeById(anyLong())) .thenThrow(new TradeNotFoundException()); TradingService tradingService = new SimpleTradingService(tradeRepository); tradingService.getTradeById(trade.getId()); } Mock Object http://xunitpatterns.com/Mock%20Object.html Mock objects are used to verify object behaviour during a test. By object behaviour I mean we check that the correct methods and paths are excercised on the object when the test is run. This is very different to the supporting role of a stub which is used to provide results to whatever you are testing. In a stub we use the pattern of defining a return value for a method. when(customer.getSurname()).thenReturn(surname); In a mock we check the behaviour of the object using the following form. verify(listMock).add(s); Here is a simple example where we want to test that a new trade is audited correctly. Here is the main code. public class SimpleTradingService implements TradingService{ TradeRepository tradeRepository; AuditService auditService; public SimpleTradingService(TradeRepository tradeRepository, AuditService auditService) { this.tradeRepository = tradeRepository; this.auditService = auditService; } public Long createTrade(Trade trade) throws CreateTradeException { Long id = tradeRepository.createTrade(trade); auditService.logNewTrade(trade); return id; } The test below creates a stub for the trade repository and mock for the AuditService We then call verify on the mocked AuditService to make sure that the TradeService calls it's logNewTrade method correctly @Mock TradeRepository tradeRepository; @Mock AuditService auditService; @Test public void testAuditLogEntryMadeForNewTrade() throws Exception { Trade trade = new Trade("Ref 1", "Description 1"); when(tradeRepository.createTrade(trade)).thenReturn(anyLong()); TradingService tradingService = new SimpleTradingService(tradeRepository, auditService); tradingService.createTrade(trade); verify(auditService).logNewTrade(trade); } The following line does the checking on the mocked AuditService. verify(auditService).logNewTrade(trade); This test allows us to show that the audit service behaves correctly when creating a trade. Test Spy http://xunitpatterns.com/Test%20Spy.html It's worth having a look at the above link for the strict definition of a Test Spy. However in Mockito I like to use it to allow you to wrap a real object and then verify or modify it's behaviour to support your testing. Here is an example were we check the standard behaviour of a List. Note that we can both verify that the add method is called and also assert that the item was added to the list. @Spy List listSpy = new ArrayList(); @Test public void testSpyReturnsRealValues() throws Exception { String s = "dobie"; listSpy.add(new String(s)); verify(listSpy).add(s); assertEquals(1, listSpy.size()); } Compare this with using a mock object where only the method call can be validated. Because we only mock the behaviour of the list, it does not record that the item has been added and returns the default value of zero when we call the size() method. @Mock List listMock = new ArrayList(); @Test public void testMockReturnsZero() throws Exception { String s = "dobie"; listMock.add(new String(s)); verify(listMock).add(s); assertEquals(0, listMock.size()); } Another useful feature of the testSpy is the ability to stub return calls. When this is done the object will behave as normal until the stubbed method is called. In this example we stub the get method to always throw a RuntimeException. The rest of the behaviour remains the same. @Test(expected=RuntimeException.class) public void testSpyReturnsStubbedValues() throws Exception { listSpy.add(new String("dobie")); assertEquals(1, listSpy.size()); when(listSpy.get(anyInt())).thenThrow(new RuntimeException()); listSpy.get(0); } In this example we again keep the core behaviour but change the size() method to return 1 initially and 5 for all subsequent calls. public void testSpyReturnsStubbedValues2() throws Exception { int size = 5; when(listSpy.size()).thenReturn(1, size); int mockedListSize = listSpy.size(); assertEquals(1, mockedListSize); mockedListSize = listSpy.size(); assertEquals(5, mockedListSize); mockedListSize = listSpy.size(); assertEquals(5, mockedListSize); } This is pretty Magic! Fake Object http://xunitpatterns.com/Fake%20Object.html Fake objects are usually hand crafted or light weight objects only used for testing and not suitable for production. A good example would be an in-memory database or fake service layer. They tend to provide much more functionality than standard test doubles and as such are probably not usually candidates for implementation using Mockito. That’s not to say that they couldn’t be constructed as such, just that its probably not worth implementing this way. References Test Double Patterns Endo-Testing: Unit Testing with Mock Objects Mock Roles, Not Objects Mocks Aren't Stubs http://msdn.microsoft.com/en-us/magazine/cc163358.aspx Original Article. The original article and others can be found here http://johndobie.blogspot.com/2011/11/test-doubles-with-mockito.html