python -m SimpleHTTPServer is a very handy command that sets up a server listening on a port (by default 8080) having as document root the current directory. However it does not count for the purpose of this article: producing our first dynamic page with Python. We won't care a lot about the performance of various approaches or whether to adopt frameworks like Django: we are just getting started. Even when higher-level abstraction are introduced, we still have to know how the foundation of our house are made. Installation mod_python is an Apache module that bridges the Apache 2 httpd processes with the Python interpreter. You can install it on Ubuntu with a single command: sudo apt-get install libapache2-mod-python See the mod_python website for alternative ways of installation. In the words of its homepage, mod_python is a mature but not dead project: it requires very little maintenance due to its stability. It is of course built only for Apache, and exposes its API; in any case, it promises to be faster than CGI (who doesn't?) Apache configuration This configuration lines should made their way into your httpd.conf or in the other configuration files you have on your system: LoadModule python_module /usr/lib/apache2/modules/mod_python.so AddHandler mod_python .py PythonHandler hello PythonDebug On We loaded the .so file of the module: if you installed mod_python in a Linux distribution, this line should already be present (use the a2enmod command if it's not in the enabled/ directory). In Ubuntu, it's written in the /etc/apache2/mods-enabled/python.load file, where you can add the other lines. The rest of the configuration means that in the directory /var/www/python, we wwant to enable the processing of .py files by mod_python. The hello.py file will be targeted, unless you use a standard handler. PythonDebug On activates a nice stack trace visualization in the browser, in case of runtime errors. Without it, the production setting is to display a blank, generic 500 Server Error. Hello world Our Hello World it's easy to write: from mod_python import apache def handler(req): req.content_type = 'text/plain' req.write("Hello, World!") return apache.OK It's a bit strange that the Content-Type has to be specified on the request object, but there is no response object anyway: all interaction has to be performed on req. A bit less boilerplate? You can use a standard handler to simplify even more your .py files, by changing the configuration to this: PythonHandler mod_python.publisher Now the Hello World example becomes: from mod_python import apache def say(req): return "Hello, World!" and is displayed at http://localhost/python/hello.py/say. While the parameters are actually kept in the request object, you can specify them as required, named parameters: def say(req, name): return "Hello, " + name + "!" http://localhost/python/hello.py/say?name=Giorgio will display: Hello, Giorgio! Including other files import is the standard Python instruction for including other code into the current source file, usually with the form from module import entity. You can use it in your handlers, but you probably need to specify Python paths specific to each of them, to avoid that multiple application clash with each other: PythonPath "sys.path+['/var/python_code']" The string value of this directive should evaluate to a Python list containing all the folders to search for modules, just like sys.path originally does. To show you how it works, I can put this in /var/python_code/hello_lib.py: def say_to(name): return "Hello, " + name + "!" and micro-refactor my handler to: from mod_python import apache from hello_lib import say_to def say(req, name): return say_to(name) Conclusion Of course there's more to mod_python and to build real web applications: they have to manipulate cookies, the session API, and databases. Moreover, their configuration should access Apache internals, with directives for everything from importing automatically modules at the start of a request to setting up HTTP authentication. However, this article is about the HTTP-facing side of a web application. It's interesting to explore how the architecture on the server side works, so probably my next stop will be to try out a framework like Django to see which standard design choices it mades for us (and which may be questionable). In any case, we already have a Turing-complete entry point in our handlers, capable of linking to any Python source code. Resources The official mod_python documentation is long-winded but very good. The Style Guide for Python Code is a more general document that defines the coding standards for the whole Python world, while applying also to our web development scenario.

Domain Specific Languages (DSLs) have become a valuable part of the Groovy idiom. DSLs are used in native Groovy builders, Grails and GORM, and testing frameworks. To a developer, DSLs are consumable and understandable, which makes implementation more fluid compared to traditional programming. But how is a DSL implemented? How does it work behind the scenes? This article will demonstrate a simple DSL that can get a developer kickstarted on the basic concepts. What is a DSL DSL's are meant to target a particular type of problem. They are short expressive means of programming that fit well in a narrow context. For example with GORM, you can express hibernate mapping with a DSL rather than XML. static mapping = { table 'person' columns { name column:'name' } } Much of the theory for DSLs and the benefits they deliver are well documented. Refer to these sources as a starting point: Martin Fowler - DSL Writing Domain Specific Languages - Groovy Programming Groovy - Chapter 18 A Simple DSL Example in Groovy The following example offers a simplified view of implementing an internal DSL. Frameworks have much more advanced methods of creating a DSL. However this example does highlight closure delegation and meta object protocol concepts that are essential to understanding the inner workings of a DSL. Requirements Overview Imagine that a customer needs a memo generator. The memos needs to have a few simple fields, such as "to", "from", and "body". The memo can also have sections such as "Summary" or "Important." The summary fields are dynamic and can be anything on demand. In addition, the memo needs to be outputed in three formats: xml, html, and text. We elect to implement this as a DSL in Groovy. The DSL result looks like this: MemoDsl.make { to "Nirav Assar" from "Barack Obama" body "How are things? We are doing well. Take care" idea "The economy is key" request "Please vote for me" xml } The output from the code yields: Nirav Assar Barack Obama How are things? We are doing well. Take care The economy is key Please vote for me The last line in the DSL can also be changed to 'html' or 'text'. This affects the output format. Implementation A static method that accepts a closure is a hassle free way to implement a DSL. In the memo example, the class MemoDsl has a make method. It creates an instance and delegates all calls in the closure to the instance. This is the mechanism where the "to", and "from" sections end up executing methods inside the MemoDsl class. Once the to() method is called, we store the text in the instance for formatting later on. class MemoDsl { String toText String fromText String body def sections = [] /** * This method accepts a closure which is essentially the DSL. Delegate the * closure methods to * the DSL class so the calls can be processed */ def static make(closure) { MemoDsl memoDsl = new MemoDsl() // any method called in closure will be delegated to the memoDsl class closure.delegate = memoDsl closure() } /** * Store the parameter as a variable and use it later to output a memo */ def to(String toText) { this.toText = toText } def from(String fromText) { this.fromText = fromText } def body(String bodyText) { this.body = bodyText } } Dynamic Sections When the closure includes a method that is not present in the MemoDsl class, groovy identifies it as a missing method. With Groovy's meta object protocol, the methodMissing interface on the class is invoked. This is how we handle sections for the memo. In the client code above we have entries for idea and request. MemoDsl.make { to "Nirav Assar" from "Barack Obama" body "How are things? We are doing well. Take care" idea "The economy is key" request "Please vote for me" xml } The sections are processed with the following code in MemoDsl. It creates a section class and appends it to a list in the instance. /** * When a method is not recognized, assume it is a title for a new section. Create a simple * object that contains the method name and the parameter which is the body. */ def methodMissing(String methodName, args) { def section = new Section(title: methodName, body: args[0]) sections << section } Processing Various Outputs Finally the most interesting part of the DSL is how we process the various outputs. The final line in the closure specifies the output desired. When a closure contains a string such as "xml" with no parameters, groovy assumes this is a 'getter' method. Thus we need to implement 'getXml()' to catch the delegation execution: /** * 'get' methods get called from the dsl by convention. Due to groovy closure delegation, * we had to place MarkUpBuilder and StringWrite code in a static method as the delegate of the closure * did not have access to the system.out */ def getXml() { doXml(this) } /** * Use markupBuilder to create a customer xml output */ private static doXml(MemoDsl memoDsl) { def writer = new StringWriter() def xml = new MarkupBuilder(writer) xml.memo() { to(memoDsl.toText) from(memoDsl.fromText) body(memoDsl.body) // cycle through the stored section objects to create an xml tag for (s in memoDsl.sections) { "$s.title"(s.body) } } println writer } The code for html and text is quite similar. The only variation is how the output is formatted. Entire Code The code in its entirety is displayed next. The best approach I found was to design the DSL client code and the specified formats first, then tackle the implementation. I used TDD and JUnit to drive my implementation. Note that I did not go the extra mile to do asserts on the system output in the tests, although this could be easily enhanced to do so. The code is fully executable inside any IDE. Run the various tests to view the DSL output. package com.solutionsfit.dsl.memotemplate class MemolDslTest extends GroovyTestCase { void testDslUsage_outputXml() { MemoDsl.make { to "Nirav Assar" from "Barack Obama" body "How are things? We are doing well. Take care" idea "The economy is key" request "Please vote for me" xml } } void testDslUsage_outputHtml() { MemoDsl.make { to "Nirav Assar" from "Barack Obama" body "How are things? We are doing well. Take care" idea "The economy is key" request "Please vote for me" html } } void testDslUsage_outputText() { MemoDsl.make { to "Nirav Assar" from "Barack Obama" body "How are things? We are doing well. Take care" idea "The economy is key" request "Please vote for me" text } } } package com.solutionsfit.dsl.memotemplate import groovy.xml.MarkupBuilder /** * Processes a simple DSL to create various formats of a memo: xml, html, and text */ class MemoDsl { String toText String fromText String body def sections = [] /** * This method accepts a closure which is essentially the DSL. Delegate the closure methods to * the DSL class so the calls can be processed */ def static make(closure) { MemoDsl memoDsl = new MemoDsl() // any method called in closure will be delegated to the memoDsl class closure.delegate = memoDsl closure() } /** * Store the parameter as a variable and use it later to output a memo */ def to(String toText) { this.toText = toText } def from(String fromText) { this.fromText = fromText } def body(String bodyText) { this.body = bodyText } /** * When a method is not recognized, assume it is a title for a new section. Create a simple * object that contains the method name and the parameter which is the body. */ def methodMissing(String methodName, args) { def section = new Section(title: methodName, body: args[0]) sections << section } /** * 'get' methods get called from the dsl by convention. Due to groovy closure delegation, * we had to place MarkUpBuilder and StringWrite code in a static method as the delegate of the closure * did not have access to the system.out */ def getXml() { doXml(this) } def getHtml() { doHtml(this) } def getText() { doText(this) } /** * Use markupBuilder to create a customer xml output */ private static doXml(MemoDsl memoDsl) { def writer = new StringWriter() def xml = new MarkupBuilder(writer) xml.memo() { to(memoDsl.toText) from(memoDsl.fromText) body(memoDsl.body) // cycle through the stored section objects to create an xml tag for (s in memoDsl.sections) { "$s.title"(s.body) } } println writer } /** * Use markupBuilder to create an html xml output */ private static doHtml(MemoDsl memoDsl) { def writer = new StringWriter() def xml = new MarkupBuilder(writer) xml.html() { head { title("Memo") } body { h1("Memo") h3("To: ${memoDsl.toText}") h3("From: ${memoDsl.fromText}") p(memoDsl.body) // cycle through the stored section objects and create uppercase/bold section with body for (s in memoDsl.sections) { p { b(s.title.toUpperCase()) } p(s.body) } } } println writer } /** * Use markupBuilder to create an html xml output */ private static doText(MemoDsl memoDsl) { String template = "Memo\nTo: ${memoDsl.toText}\nFrom: ${memoDsl.fromText}\n${memoDsl.body}\n" def sectionStrings ="" for (s in memoDsl.sections) { sectionStrings += s.title.toUpperCase() + "\n" + s.body + "\n" } template += sectionStrings println template } }

In this post we will see how to visualize a function of two variables in two ways. First, we will create an intensity image of the function and, second, we will use the 3D plotting capabilities of matplotlib to create a shaded surface plot. So, let's go with the code: from numpy import exp,arange from pylab import meshgrid,cm,imshow,contour,clabel,colorbar,axis,title,show # the function that I'm going to plot def z_func(x,y): return (1-(x**2+y**3))*exp(-(x**2+y**2)/2) x = arange(-3.0,3.0,0.1) y = arange(-3.0,3.0,0.1) X,Y = meshgrid(x, y) # grid of point Z = z_func(X, Y) # evaluation of the function on the grid im = imshow(Z,cmap=cm.RdBu) # drawing the function # adding the Contour lines with labels cset = contour(Z,arange(-1,1.5,0.2),linewidths=2,cmap=cm.Set2) clabel(cset,inline=True,fmt='%1.1f',fontsize=10) colorbar(im) # adding the colobar on the right # latex fashion title title('$z=(1-x^2+y^3) e^{-(x^2+y^2)/2}$') show() The script would have the following output: And now we are going to use the values stored in X,Y and Z to make a 3D plot using the mplot3d toolkit. Here's the snippet: from mpl_toolkits.mplot3d import Axes3D from matplotlib import cm from matplotlib.ticker import LinearLocator, FormatStrFormatter import matplotlib.pyplot as plt from numpy import sin,sqrt fig = plt.figure() ax = fig.gca(projection='3d') surf = ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.RdBu,linewidth=0, antialiased=False) ax.zaxis.set_major_locator(LinearLocator(10)) ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f')) fig.colorbar(surf, shrink=0.5, aspect=5) plt.show() And this is the result: Source: http://glowingpython.blogspot.com/2012/01/how-to-plot-two-variable-functions-with.html

Using open as a context manager is a great way to ensure your file handles are closed properly and is becoming common: with open('/some/path', 'w') as f: f.write('something') The issue is that even if you mock out the call to open it is the returned object that is used as a context manager (and has __enter__ and __exit__ called). Using MagicMock from the mock library, we can mock out context managers very simply. However, mocking open is fiddly enough that a helper function is useful. Here mock_open creates and configures a MagicMock that behaves as a file context manager. from mock import inPy3k, MagicMock if inPy3k: file_spec = ['_CHUNK_SIZE', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__gt__', '__hash__', '__iter__', '__le__', '__lt__', '__ne__', '__next__', '__repr__', '__str__', '_checkClosed', '_checkReadable', '_checkSeekable', '_checkWritable', 'buffer', 'close', 'closed', 'detach', 'encoding', 'errors', 'fileno', 'flush', 'isatty', 'line_buffering', 'mode', 'name', 'newlines', 'peek', 'raw', 'read', 'read1', 'readable', 'readinto', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 'writable', 'write', 'writelines'] else: file_spec = file def mock_open(mock=None, data=None): if mock is None: mock = MagicMock(spec=file_spec) handle = MagicMock(spec=file_spec) handle.write.return_value = None if data is None: handle.__enter__.return_value = handle else: handle.__enter__.return_value = data mock.return_value = handle return mock >>> m = mock_open() >>> with patch('__main__.open', m, create=True): ... with open('foo', 'w') as h: ... h.write('some stuff') ... >>> m.assert_called_once_with('foo', 'w') >>> m.mock_calls [call('foo', 'w'), call().__enter__(), call().write('some stuff'), call().__exit__(None, None, None)] >>> handle = m() >>> handle.write.assert_called_once_with('some stuff') And for reading files, using a StringIO to represent the file handle: >>> from StringIO import StringIO >>> m = mock_open(data=StringIO('foo bar baz')) >>> with patch('__main__.open', m, create=True): ... with open('foo') as h: ... result = h.read() ... >>> m.assert_called_once_with('foo') >>> assert result == 'foo bar baz' Note that the StringIO will only be used for the data if open is used as a context manager. If you just configure and use mocks they will work whichever way open is used. This helper function will be built into mock 0.9. Source: http://www.voidspace.org.uk/python/weblog/arch_d7_2012_01_07.shtml

today we continue html5 canvas examples. and today is our second tutorial for webgl. we will be creating animated twisting images. also we will add handlers to manipulate the images with mouse and keyboard. here are our demo and downloadable package: live demo download in package ok, download the example files and lets start coding ! step 1. html here are html sources of our demo. as you can see – just an empty page. index.html creating a keyboard sensitive 3d twisted images in webgl | script tutorials you can use your mouse + arrow keys + page up / down step 2. css here are used css styles. css/main.css body { background:#eee; font-family:verdana, helvetica, arial, sans-serif; margin:0; padding:0 } .example { background:#fff; width:500px; font-size:80%; border:1px #000 solid; margin:20px auto; padding:15px; position:relative; -moz-border-radius: 3px; -webkit-border-radius:3px } h3 { text-align:center; } step 3. js js/webgl-utils.js and js/glmatrix-0.9.5.min.js these files we will use in project for working with webgl. both files will in our package. js/script.js var gl; // global webgl object var shaderprogram; var pics_names=['1.png', '2.png', '3.png', '4.png', '5.png', '6.png', '7.png']; var pics_num=pics_names.length; // diffirent initializations function initgl(canvas) { try { gl = canvas.getcontext('experimental-webgl'); gl.viewportwidth = canvas.width; gl.viewportheight = canvas.height; } catch (e) {} if (! gl) { alert('can`t initialise webgl, not supported'); } } function getshader(gl, type) { var str = ''; var shader; if (type == 'x-fragment') { str = "#ifdef gl_es\n"+ "precision highp float;\n"+ "#endif\n"+ "varying vec2 vtexturecoord;\n"+ "uniform sampler2d usampler;\n"+ "void main(void) {\n"+ " gl_fragcolor = texture2d(usampler, vec2(vtexturecoord.s, vtexturecoord.t));\n"+ "}\n"; shader = gl.createshader(gl.fragment_shader); } else if (type == 'x-vertex') { str = "attribute vec3 avertexposition;\n"+ "attribute vec2 atexturecoord;\n"+ "uniform mat4 umvmatrix;\n"+ "uniform mat4 upmatrix;\n"+ "varying vec2 vtexturecoord;\n"+ "void main(void) {\n"+ " gl_position = upmatrix * umvmatrix * vec4(avertexposition, 1.0);\n"+ " vtexturecoord = atexturecoord;\n"+ "}\n"; shader = gl.createshader(gl.vertex_shader); } else { return null; } gl.shadersource(shader, str); gl.compileshader(shader); if (!gl.getshaderparameter(shader, gl.compile_status)) { alert(gl.getshaderinfolog(shader)); return null; } return shader; } function initshaders() { var fragmentshader = getshader(gl, 'x-fragment'); var vertexshader = getshader(gl, 'x-vertex'); shaderprogram = gl.createprogram(); gl.attachshader(shaderprogram, vertexshader); gl.attachshader(shaderprogram, fragmentshader); gl.linkprogram(shaderprogram); if (!gl.getprogramparameter(shaderprogram, gl.link_status)) { alert('can`t initialise shaders'); } gl.useprogram(shaderprogram); shaderprogram.vertexpositionattribute = gl.getattriblocation(shaderprogram, 'avertexposition'); gl.enablevertexattribarray(shaderprogram.vertexpositionattribute); shaderprogram.texturecoordattribute = gl.getattriblocation(shaderprogram, 'atexturecoord'); gl.enablevertexattribarray(shaderprogram.texturecoordattribute); shaderprogram.pmatrixuniform = gl.getuniformlocation(shaderprogram, 'upmatrix'); shaderprogram.mvmatrixuniform = gl.getuniformlocation(shaderprogram, 'umvmatrix'); shaderprogram.sampleruniform = gl.getuniformlocation(shaderprogram, 'usampler'); } var objvertexpositionbuffer=new array(); var objvertextexturecoordbuffer=new array(); var objvertexindexbuffer=new array(); function initobjbuffers() { for (var i=0;i

JSR 303 and Hibernate Validator have been awesome additions to the Java ecosystem, giving you a standard way to validate your domain model across application layers.

Matrices can be really large, sometimes larger than you can hold in one array. You can extend the maximum size by having multiple arrays however this can make your heap size really large and inefficient. An alternative is to use a wrapper over a memory mapped file. The advantage of memory mapped files is that they have very little impact on the heap and can be swapped in and out by the OS fairly transparently. A huge matrix This code supports large matrices of double. It partitions the file into 1 GB mappings. (As Java doesn't support mappings of 2 GB or more at a time, a pet hate of mine ;) import sun.misc.Cleaner; import sun.nio.ch.DirectBuffer; import java.io.Closeable; import java.io.IOException; import java.io.RandomAccessFile; import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel; import java.util.ArrayList; import java.util.List; public class LargeDoubleMatrix implements Closeable { private static final int MAPPING_SIZE = 1 << 30; private final RandomAccessFile raf; private final int width; private final int height; private final List mappings = new ArrayList(); public LargeDoubleMatrix(String filename, int width, int height) throws IOException { this.raf = new RandomAccessFile(filename, "rw"); try { this.width = width; this.height = height; long size = 8L * width * height; for (long offset = 0; offset < size; offset += MAPPING_SIZE) { long size2 = Math.min(size - offset, MAPPING_SIZE); mappings.add(raf.getChannel().map(FileChannel.MapMode.READ_WRITE, offset, size2)); } } catch (IOException e) { raf.close(); throw e; } } protected long position(int x, int y) { return (long) y * width + x; } public int width() { return width; } public int height() { return height; } public double get(int x, int y) { assert x >= 0 && x < width; assert y >= 0 && y < height; long p = position(x, y) * 8; int mapN = (int) (p / MAPPING_SIZE); int offN = (int) (p % MAPPING_SIZE); return mappings.get(mapN).getDouble(offN); } public void set(int x, int y, double d) { assert x >= 0 && x < width; assert y >= 0 && y < height; long p = position(x, y) * 8; int mapN = (int) (p / MAPPING_SIZE); int offN = (int) (p % MAPPING_SIZE); mappings.get(mapN).putDouble(offN, d); } public void close() throws IOException { for (MappedByteBuffer mapping : mappings) clean(mapping); raf.close(); } private void clean(MappedByteBuffer mapping) { if (mapping == null) return; Cleaner cleaner = ((DirectBuffer) mapping).cleaner(); if (cleaner != null) cleaner.clean(); } } public class LargeDoubleMatrixTest { @Test public void getSetMatrix() throws IOException { long start = System.nanoTime(); final long used0 = usedMemory(); LargeDoubleMatrix matrix = new LargeDoubleMatrix("ldm.test", 1000 * 1000, 1000 * 1000); for (int i = 0; i < matrix.width(); i++) matrix.set(i, i, i); for (int i = 0; i < matrix.width(); i++) assertEquals(i, matrix.get(i, i), 0.0); long time = System.nanoTime() - start; final long used = usedMemory() - used0; if (used == 0) System.err.println("You need to use -XX:-UsedTLAB to see small changes in memory usage."); System.out.printf("Setting the diagonal took %,d ms, Heap used is %,d KB%n", time / 1000 / 1000, used / 1024); matrix.close(); } private long usedMemory() { return Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory(); } } With the following test, which writes to each of the diagonal values of a one million * one million matrix. This is too large to hope to create on the heap. Setting the diagonal took 314,819 ms, Heap used is 2,025 KB $ ls -l ldm.test -rw-rw-r-- 1 peter peter 8000000000000 2011-12-30 12:42 ldm.test $ du -s ldm.test 4010600 ldm.test That's 8,000,000,000,000 bytes or ~7.3 TB in virtual memory, in a Java process! This works because it only allocates or pages in the pages which you use. So while the file size is almost 8 TB, the actual disk space and memory used is 4 GB. With a more modest file size of 100K * 100K matrix you see something like the following. Its still an 80 GB matrix, which uses trivial heap space. ;) Setting the diagonal took 110 ms, Heap used is 71 KB $ ls -l ldm.test -rw-rw-r-- 1 peter peter 80000000000 2011-12-30 12:49 ldm.test $ du -s ldm.test 400000 ldm.test From http://vanillajava.blogspot.com/2011/12/using-memory-mapped-file-for-huge.html

This is an excerpt from the upcoming Manning book about Big Data. Big Data Principles and Best Practices of Scalable Realtime Data Systems By Nathan Marz and Samuel E. Ritchie Thrift is a widely used project that originated at Facebook. It can be used for making language-neutral RPC servers, but developers use it for its schema-creation capabilities. In this article based on chapter 2, author Nathan Marz discusses workhorses of Thrift—the struct and union type definitions—and Thrift’s built-in mechanisms for evolving a schema over time. You may also be interested in… Thrift is a widely used project that originated at Facebook. It can be used for making language-neutral RPC servers, but developers use it for its schema-creation capabilities. The workhorses of Thrift are the struct and union type definitions, and Thrift has built-in mechanisms for evolving a schema over time. Orginally Authored by Nathan Marz and Samuel E. Ritchie Structs The following code shows how to define a struct using the Thrift Interface Definition Language (IDL). Defining a struct is like defining a class in an object-oriented language: you specify all the data the object contains. The difference is that a Thrift struct only contains data and doesn't specify any extra behavior for the object. Fields in a struct can be: Primitive types like strings, ints, longs, and doubles. In the Thrift IDL, these are referred to as string, i32, i64, and double, respectively. Collections of other types. Thrift supports list, map, and set. Another Thrift struct or union. struct Person { 1: string twitter_username; 2: string full_name; 3: list interests; } The following code listing shows how to serialize a struct with Java. As you can see, we're using ArrayList, a native Java data structure, as part of the Person object. List interests = new ArrayList() {{ add("hadoop"); add("nosql"); }; Person person = new Person("joesmith", "Joe Smith", interests); TSerializer serializer = new TSerializer(); byte[] serialized = serializer.serialize(person); Here's how to deserialize a Person object in Python. When the object is deserialized, it will be using native Python data structures for any collection types. person = Person() deserialize(person, serialized_bytes) Fields in structs can be defined as being either required or optional. If a field is defined as required, than a value for that field must be provided or else Thrift will give an error upon serialization or deserialization. If a field is optional, the value will be null if not provided. You should always declare fields as being either required or optional. The following code listing shows how to define a struct containing required and optional fields. struct Tweet { 1: required string text; 2: required i64 id; 3: required i64 timestamp; 4: required Person person; 5: optional i64 response_to_tweet_id; Unions You can also define unions in Thrift. A union is a struct that must have exactly one field set. Unions are useful for representing polymorphic data. The following listing shows how to define a "PersonID" using a Thrift union that can be one of many different kinds of identifiers. union PersonID { 1: string email; 2: i64 facebook_id; 3: i64 twitter_id; } Evolving a schema Thrift is designed so that schemas can be evolved over time. The key to evolving Thrift schemas over time is the numeric identifiers used for every field. Those ids are used to identify fields in their serialized form. When you want to change the schema but still be backward compatible with existing data, you must obey the following rules. Fields may be renamed. This is because the serialized form of an object uses the field ids to identify fields, not the names. Fields may be removed, but you must be sure never to reuse that field id. When deserializing, Thrift will skip over any fields that don't match an id it's expecting. So the data for that field will just be ignored in the existing data. If you were to reuse that field id, Thrift will try to deserialize that old data into your new field which will lead to either invalid or incorrect data. Only optional fields can be added to existing structs. You can't add required fields because existing data won't have that field and will not be deserializable. Note that this point does not apply to unions since unions have no notion of required and optional fields. Summary In a relational database, the schema language is part of the database system and is integrated with how the database stores and processes that data. In the Big Data world, you use your own serialization framework that's separate from the storage and processing pieces. You get the flexibility to fine-tune this component to work exactly as needed to fit your data model. There are a few different open source serialization frameworks available, namely Thrift, Protocol Buffers, and Avro. We discussed our favorite, Apache Thrift, because it’s mature and supports most languages, but you could use any of these tools for defining a schema. Here are some other Manning titles you might be interested in: MongoDB in Action Kyle Banker RabbitMQ in Action Alvaro Videla and Jason J.W. Williams Hadoop in Action Chuck Lam Last updated: January 11, 2012

in the graph database space, there are two types of traversal engines: local and distributed. local traversal engines are typically for single-machine graph databases and are used for real-time production applications. distributed traversal engines are typically for multi-machine graph databases and are used for batch processing applications. this divide is quite sharp in the community, but there is nothing that prevents the unification of both models. a discussion of this divide and its unification is presented in this post. local traversal engines in a local traversal engine, there is typically a single processing agent that obeys a program. the agent is called a traverser and the program it follows is called a path description . in gremlin , a friend-of-a-friend path description is defined as such: g.v(1).oute('friend').inv.oute('friend').inv when this path description is interpreted by a traverser over a graph, an instance of the description is realized as actual paths in the graph that match that description. for example, the gremlin traverser starts at vertex 1 and then steps to its outgoing friend -edges. next, it will move to the head/target vertices of those edges (i.e. vertices 3 and 4). after that, it will go to the friend -edges of those vertices and finally, to the head of the previous edges (i.e. vertices 6 and 7). in this way, a single traverser is following all the paths that are exposed with each new atomic graph operation (i.e. each new step after a .). the abstract syntax being: step.step.step . what is returned by this friend-of-a-friend path description, on the example graph diagrammed, is vertices 6 and 7. in many situations, its not the end of the path that is desired, but some side-effect of the traversal. for example, as the traverser walks it can update some global data structure such as a ranking of the vertices. this idea is presented in the path description below, where the oute.inv path is looped over 1000 times. each time a vertex is traversed over, the map m is updated. this global map m maintains keys that are vertices and values that denote the number of times that each vertex has been touched ( groupcount ‘s behavior). m = [:] g.v(1).oute.inv.groupcount(m).loop(3){it.loops < 1000} the local traversal engine pattern is abstractly diagrammed on the right, where a single traverser is obeying some path description ( a.b.c ) and in doing so, moving around on a graph and updating a global data structure (the red boxed map). given the need for traversers to move from element to element, graph databases of this form tend to support strong data locality by means of a direct-reference graph data structure (i.e. vertices have pointers to edges and edges to vertices). a few examples of such graph databases include neo4j , orientdb , and dex . distributed traversal engines in a distributed traversal engine, a traversal is represented as a flow of messages between the elements of the graph. generally, each element (e.g. vertex) is operating independently of the other elements. each element is seen as its own processor with its own (usually homogenous) program to execute. elements communicate with each other via message passing . when no more messages have been passed, the traversal is complete and the results of the traversal are typically represented as a distributed data structure over the elements. graph databases of this nature tend to use the bulk synchronous parallel model of distributed computing. each step is synchronized in a manner analogous to a clock cycle in hardware. instances of this model include agrapa , pregel , trinity , and goldenorb . an example of distributed graph traversing is now presented using a ranking algorithm in java. [ note : in this example, edges are not first class citizens. this is typical of the state of the art in distributed traversal engines. they tend to be for single-relational, unlabeled-edge graphs.] public void evaluatestep(int step) { if(!this.inbox.isempty() && step < 1000) { this.rank = this.rank + this.inbox.size(); for(vertex vertex : this.adjacentvertices()) { for(int i=0; i

I’ve been developing for quite a while and in quite a few languages. Somehow though, I’ve always seemed to fall back to Java when doing my own stuff – maybe partly from habit, partly because it has in my opinion the best open source selection out there, and party because I liked its mix of features and performance. Originally authored by Matan Amir Specifically though, in the web arena things have been moving fast and furious with new languages, approaches, and methods like RoR, Play!, and Lift (and many others!). While I “get it” regarding the benefits of these frameworks, I never felt the need to give them more than an initial deep dive to see how they work. I nod a few times at their similarities and move on back to plain REST-ful services in Java with Spring, maybe an ORM (i try to avoid them these days), and a JS-rich front-end. Recently, two factors made me deep dive into Node.js. First is my growing appreciation with the progress of the JavaScript front-end. What used to be little JavaScript snippets to validate a form “onSubmit” have evolved to a complete ecosystem of technologies, frameworks, and stacks. My personal favorite these days is Backbone.js and try to use it whenever I can. Second was the first hand feedback I got from a friend at Voxer about their success in using and deploying Node.js at real scale (they have a pretty big Node.js deployment). So I jumped in. In the short time I’ve been using it for some (real) projects, I can say that it’s my new favorite language of choice. First of all, the event-driven (and non-blocking) model is a perfect fit for server side development. While this has existed in the other languages and forms (Java Servlet 3.0, Event Machine, Twisted to name a few), I connected with the easy of use and natural maturity of it in JavaScript. We’re all used to using callbacks anyway from your typical run-of-the-mill AJAX work right? Second is the community and how large its gotten in the short time Node has been around. There are lots of useful open source libraries to use to solve your problems and the quality level is high. Third is that it was just so easy to pick up. I have to admit that my core JavaScript was decent before I started using Node, but in terms of the Node core library and feature set itself, it’s pretty lean and mean and provides a good starting point to build what you need in case you can’t find someone who already did (which you most likely can). So having said all that, I wanted to share what resources helped me get up to speed in Node.js coming from a Java background. Getting to know JavaScript There are lots of good resources out there on Node.js and i’ve listed them below. For me, the most critical piece was getting my JavaScript knowledge to the next level. Because you’ll be spending all your time writing your server code in JavaScript, it pays huge dividends to understand how to take full advantage of it. As a Java developer you’re probably trained to think in OO designs. With me, this was the part that I focused the most on. Fortunately (or unforunately), JavaScript is not a classic OO language. It can be if you shoehorn it, but i think that defeats the purpose. Here is my short list of JavaScript resources: JavaScript: The Good Parts - Definitely a requirement. Chapters 4 and 5 (functions, objects, and inheritance) are probably the most important part to understand well for those with an OO background. Learning Javascript with Object Graphs (part 2, and part 3) – howtonode.org has lots of good Node material, but this 3 part post is a good resource to top off your knowledge from the book. Simple “Class” Instantiation – Another good post I read recently. Worth digesting. Learning Node.js With a good JavaScript background, starting to use Node.js is pretty straightforward. The main part to understand is the asynchronous nature of the I/O and the need to produce and consume events to get stuff done. Here is a list of resources I used to get up to speed: DailyJS’s Node Tutorial - A multipart tutorial for Node on DailyJS’s blog. It’s a great resource and worth going through all the posts. Mixu’s Node Book - Not complete, but still worth it. I look forward to reading the future chapters. Node Beginner Book – A good starter read. How To Node – A blog dedicated to Node.js. Bookmark it. I felt that going through these was more than enough to give me the push I needed get started. Hopefully it does for you too (thanks to the authors for taking the time to share their knowledge!). Frameworks? Java is all about open source frameworks. That’s part of why it’s so popular. While Node.js is much newer, lots of people have already done quite a bit of heavy-lifting and have shared their code with the world. Below is what I think is a good mapping of popular Java frameworks to their Node.js equivalents (from what I know so far). Web MVC In Java land, most people are familiar with Web MVC frameworks like Spring MVC, Struts, Wicket, and JSF. More recently though, the trend towards client-side JS MVC frameworks like Ember.js (SproutCore) and Backbone.js reduces the required feature-set of some of these frameworks. Nonetheless, a good comparable Node.js web framework is Express. In a sense, it’s even more than a web framework because it also provides most of the “web server” functionality most Java developers are used to through Tomcat, Jetty, etc (more specifically, Express builds on top of Connect) It’s well thought out and provides the feature set needed to get things done. Application Lifecycle Framework (DI Framework) Spring is a popular framework in Java to provide a great deal of glue and abstraction functionality. It allows easy dependency injection, testing, object lifecycle management, transaction management, etc. It’s usually one of the first things I slap into a new project in Java. In Node.js… I haven’t really missed it. JavaScript allows much more flexible ways of decoupling dependencies and I personally haven’t felt the need to find a replacement for Spring. Maybe that’s a good thing? Object-Relational Mapping (ORMs) I have mixed feelings regarding ORMs in general, but it does make your life easier sometimes. There is no shortage of ORM librares for Node.js. Take your pick. Package Management Tools Maven is probably most popular build management tool for Java. While it is very flexible and powerful with a wide variety of plug-ins, it can get very cumbersome. Npm is the mainstream package manager for Node.js. It’s light, fast, and useful. Testing Framework Java has lots of these for sure, jUnit and company as standard. There are also mock libraries, stub libraries, db test libraries, etc. Node.js has quite a few as well. Pick your poison. I see that nodeunit is popular and is similar to jUnit. I’m personally testing Mocha. Testing tools are a more personal and subjective choice, but the good thing is that there definitely are good choices out there. Logging Java developers have quite a list of choices when choosing a logger library. Commons logging, log4j, logback, and slf4j (wrapper) are some of the more popular ones. Node.js also has a few. I’m currently using winston and have no complaints so far. It has the logging levels, multiple transports (appenders to log4j people), and does it all asynchronously as well. Hopefully this will help someone save some time when peeking into the world of Node. Good luck! Source: http://n0tw0rthy.wordpress.com/2012/01/08/from-java-to-node-js/

Learn the differences between GET and POST requests.

Clojure is a LISP dialect, and as such a functional language based on a large set of functions and a small set of data structures that they operate with. However, it is possible to implement classes and object in Clojure, with constructs well-supported in the language itself. I do not claim you should program in Clojure only with the techniques described in this article: they are just an attempt to bridge Java libraries with Clojure, and to introduce objects and interfaces where needed. Java interoperability Even from the Clojure REPL, you can easily instantiate Java classes as long as they are in the classpath (so use lein if they are in a library) (def ford (Car. arg1 arg2)) Since classes usually reside into packages, in real code you would use their fully qualified name: (def ford (com.dzone.example.Car. arg1 arg2)) You can also call methods: (.brake ford arg1 arg2) The first argument of the evaluation of a .methodName is always the object, while additional arguments are placed after it, in the same s-expression (a fancy name for a list that can be evaluated.) One of the first exercises in the book The Joy of Clojure is an instantiation of a java.awt.Frame object and a rendering done on it, all from the Clojure interactive interpreter. Doing exploratory testing classes and objects for Java development with the REPL is as faster as writing code into JUnit test. Define your own: interfaces defprotocol takes as the first argument an identifier, and a variable number of parameters. Each of this parameters is a method definition like you would do with fn, but without the fn keyword itself and a body to evaluate. The first argument of the methods should always be this, representing the current object (remember Python?). (defprotocol Position (description [this]) (translateX [this dx]) (translateY [this dy]) (doubleCoords [this]) (average [this another])) Define your own: classes Defining classes in Clojure is simple, at least when they implement immutable Value Objects. defrecord takes a class name, a list of parameters for the constructor, an optional protocol name and a variable number of arguments representing the methods. Again, the methods definition are similar to the ones made with fn, but this time with a body, an s-expression to evaluate. There is a catch: you can only implement methods defined in a protocol. (defrecord CartesianPoint [x y] Position (description [this] (str "(x=" x ", y=" y ")")) (translateX [this dx] (CartesianPoint. (+ x dx) y)) (translateY [this dy] (CartesianPoint. x (+ y dy))) (doubleCoords [this] (CartesianPoint. (* 2 x) (* 2 y))) (average [this another] (let [mean (fn [a b] (/ (+ a b) 2))] (CartesianPoint. (mean x (:x another)) (mean y (:y another)))))) The method access the (immutable) fields of the current object with their names, so this is not probably necessary if not for calling other methods on the object. Note that records are not strictly objects in the OO sense, since they do not encapsulate their fields: fields can be accessed anywhere with (:fieldname object). Here's how to work with CartesianPoint objects: (deftest test-point-x-field (is (= 1 (:x (CartesianPoint. 1 2))))) (deftest test-point-y-field (is (= 2 (:y (CartesianPoint. 1 2))))) (deftest test-point-to-string (is (= "(x=1, y=2)" (description (CartesianPoint. 1 2))))) (deftest test-point-translation-x (is (= (CartesianPoint. 11 2) (translateX (CartesianPoint. 1 2) 10)))) (deftest test-point-translation-y (is (= (CartesianPoint. 1 12) (translateY (CartesianPoint. 1 2) 10)))) (deftest test-point-doubling (is (= (CartesianPoint. 2 4) (doubleCoords (CartesianPoint. 1 2))))) (deftest test-points-average (is (= (CartesianPoint. 3 4) (average (CartesianPoint. 1 2) (CartesianPoint. 5 6)))))

Lucene's 3.4.0 release adds a new feature called index-time join (also sometimes called sub-documents, nested documents or parent/child documents), enabling efficient indexing and searching of certain types of relational content. Most search engines can't directly index relational content, as documents in the index logically behave like a single flat database table. Yet, relational content is everywhere! A job listing site has each company joined to the specific listings for that company. Each resume might have separate list of skills, education and past work experience. A music search engine has an artist/band joined to albums and then joined to songs. A source code search engine would have projects joined to modules and then files. Perhaps the PDF documents you need to search are immense, so you break them up and index each section as a separate Lucene document; in this case you'll have common fields (title, abstract, author, date published, etc.) for the overall document, joined to the sub-document (section) with its own fields (text, page number, etc.). XML documents typically contain nested tags, representing joined sub-documents; emails have attachments; office documents can embed other documents. Nearly all search domains have some form of relational content, often requiring more than one join. If such content is so common then how do search applications handle it today? One obvious "solution" is to simply use a relational database instead of a search engine! If relevance scores are less important and you need to do substantial joining, grouping, sorting, etc., then using a database could be best overall. Most databases include some form a text search, some even using Lucene. If you still want to use a search engine, then one common approach is to denormalize the content up front, at index-time, by joining all tables and indexing the resulting rows, duplicating content in the process. For example, you'd index each song as a Lucene document, copying over all fields from the song's joined album and artist/band. This works correctly, but can be horribly wasteful as you are indexing identical fields, possibly including large text fields, over and over. Another approach is to do the join yourself, outside of Lucene, by indexing songs, albums and artist/band as separate Lucene documents, perhaps even in separate indices. At search-time, you first run a query against one collection, for example the songs. Then you iterate through all hits, gathering up (joining) the full set of corresponding albums and then run a second query against the albums, with a large OR'd list of the albums from the first query, repeating this process if you need to join to artist/band as well. This approach will also work, but doesn't scale well as you may have to create possibly immense follow-on queries. Yet another approach is to use a software package that has already implemented one of these approaches for you! elasticsearch, Apache Solr, Apache Jackrabbit, Hibernate Search and many others all handle relational content in some way. With BlockJoinQuery you can now directly search relational content yourself! Let's work through a simple example: imagine you sell shirts online. Each shirt has certain common fields such as name, description, fabric, price, etc. For each shirt you have a number of separate stock keeping units or SKUs, which have their own fields like size, color, inventory count, etc. The SKUs are what you actually sell, and what you must stock, because when someone buys a shirt they buy a specific SKU (size and color). Maybe you are lucky enough to sell the incredible Mountain Three-wolf Moon Short Sleeve Tee, with these SKUs (size, color): small, blue small, black medium, black large, gray Perhaps a user first searches for "wolf shirt", gets a bunch of hits, and then drills down on a particular size and color, resulting in this query: name:wolf AND size=small AND color=blue which should match this shirt. name is a shirt field while the size and color are SKU fields. But if the user drills down instead on a small gray shirt: name:wolf AND size=small AND color=gray then this shirt should not match because the small size only comes in blue and black. How can you run these queries using BlockJoinQuery? Start by indexing each shirt (parent) and all of its SKUs (children) as separate documents, using the new IndexWriter.addDocuments API to add one shirt and all of its SKUs as a single document block. This method atomically adds a block of documents into a single segment as adjacent document IDs, which BlockJoinQuery relies on. You should also add a marker field to each shirt document (e.g. type = shirt), as BlockJoinQuery requires a Filter identifying the parent documents. To run a BlockJoinQuery at search-time, you'll first need to create the parent filter, matching only shirts. Note that the filter must use FixedBitSet under the hood, like CachingWrapperFilter: Filter shirts = new CachingWrapperFilter( new QueryWrapperFilter( new TermQuery( new Term("type", "shirt")))); Create this filter once, up front and re-use it any time you need to perform this join. Then, for each query that requires a join, because it involves both SKU and shirt fields, start with the child query matching only SKU fields: BooleanQuery skuQuery = new BooleanQuery(); skuQuery.add(new TermQuery(new Term("size", "small")), Occur.MUST); skuQuery.add(new TermQuery(new Term("color", "blue")), Occur.MUST); Next, use BlockJoinQuery to translate hits from the SKU document space up to the shirt document space: BlockJoinQuery skuJoinQuery = new BlockJoinQuery( skuQuery, shirts, ScoreMode.None); The ScoreMode enum decides how scores for multiple SKU hits should be aggregated to the score for the corresponding shirt hit. In this query you don't need scores from the SKU matches, but if you did you can aggregate with Avg, Max or Total instead. Finally you are now free to build up an arbitrary shirt query using skuJoinQuery as a clause: BooleanQuery query = new BooleanQuery(); query.add(new TermQuery(new Term("name", "wolf")), Occur.MUST); query.add(skuJoinQuery, Occur.MUST); You could also just run skuJoinQuery as-is if the query doesn't have any shirt fields. Finally, just run this query like normal! The returned hits will be only shirt documents; if you'd also like to see which SKUs matched for each shirt, use BlockJoinCollector: BlockJoinCollector c = new BlockJoinCollector( Sort.RELEVANCE, // sort 10, // numHits true, // trackScores false // trackMaxScore ); searcher.search(query, c); The provided Sort must use only shirt fields (you cannot sort by any SKU fields). When each hit (a shirt) is competitive, this collector will also record all SKUs that matched for that shirt, which you can retrieve like this: TopGroups hits = c.getTopGroups( skuJoinQuery, skuSort, 0, // offset 10, // maxDocsPerGroup 0, // withinGroupOffset true // fillSortFields ); Set skuSort to the sort order for the SKUs within each shirt. The first offset hits are skipped (use this for paging through shirt hits). Under each shirt, at most maxDocsPerGroup SKUs will be returned. Use withinGroupOffset if you want to page within the SKUs. If fillSortFields is true then each SKU hit will have values for the fields from skuSort. The hits returned by BlockJoinCollector.getTopGroups are SKU hits, grouped by shirt. You'd get the exact same results if you had denormalized up-front and then used grouping to group results by shirt. You can also do more than one join in a single query; the joins can be nested (parent to child to grandchild) or parallel (parent to child1 and parent to child2). However, there are some important limitations of index-time joins: The join must be computed at index-time and "compiled" into the index, in that all joined child documents must be indexed along with the parent document, as a single document block. Different document types (for example, shirts and SKUs) must share a single index, which is wasteful as it means non-sparse data structures like FieldCache entries consume more memory than they would if you had separate indices. If you need to re-index a parent document or any of its child documents, or delete or add a child, then the entire block must be re-indexed. This is a big problem in some cases, for example if you index "user reviews" as child documents then whenever a user adds a review you'll have to re-index that shirt as well as all its SKUs and user reviews. There is no QueryParser support, so you need to programmatically create the parent and child queries, separating according to parent and child fields. The join can currently only go in one direction (mapping child docIDs to parent docIDs), but in some cases you need to map parent docIDs to child docIDs. For example, when searching songs, perhaps you want all matching songs sorted by their title. You can't easily do this today because the only way to get song hits is to group by album or band/artist. The join is a one (parent) to many (children), inner join. As usual, patches are welcome! There is work underway to create a more flexible, but likely less performant, query-time join capability, which should address a number of the above limitations. Source: http://blog.mikemccandless.com/2012/01/searching-relational-content-with.html

Many applications record a series of events to file-based storage for later use. This can be anything from logging and auditing, through to keeping a transaction redo log in an event sourced design or its close relative CQRS. Java has a number of means by which a file can be sequentially written to, or read back again. This article explores some of these mechanisms to understand their performance characteristics. For the scope of this article I will be using pre-allocated files because I want to focus on performance. Constantly extending a file imposes a significant performance overhead and adds jitter to an application resulting in highly variable latency. "Why is a pre-allocated file better performance?", I hear you ask. Well, on disk a file is made up from a series of blocks/pages containing the data. Firstly, it is important that these blocks are contiguous to provide fast sequential access. Secondly, meta-data must be allocated to describe this file on disk and saved within the file-system. A typical large file will have a number of "indirect" blocks allocated to describe the chain of data-blocks containing the file contents that make up part of this meta-data. I'll leave it as an exercise for the reader, or maybe a later article, to explore the performance impact of not preallocating the data files. If you have used a database you may have noticed that it preallocates the files it will require. The Test I want to experiment with 2 file sizes. One that is sufficiently large to test sequential access, but can easily fit in the file-system cache, and another that is much larger so that the cache subsystem is forced to retire pages so that new ones can be loaded. For these two cases I'll use 400MB and 8GB respectively. I'll also loop over the files a number of times to show the pre and post warm-up characteristics. I'll test 4 means of writing and reading back files sequentially: RandomAccessFile using a vanilla byte[] of page size. Buffered FileInputStream and FileOutputStream. NIO FileChannel with ByteBuffer of page size. Memory mapping a file using NIO and direct MappedByteBuffer. The tests are run on a 2.0Ghz Sandybridge CPU with 8GB RAM, an Intel 230 SSD on Fedora Core 15 64-bit Linux with an ext4 file system, and Oracle JDK 1.6.0_30. The Code import java.io.*; import java.nio.ByteBuffer; import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel; import static java.lang.Integer.MAX_VALUE; import static java.lang.System.out; import static java.nio.channels.FileChannel.MapMode.READ_ONLY; import static java.nio.channels.FileChannel.MapMode.READ_WRITE; public final class TestSequentialIoPerf { public static final int PAGE_SIZE = 1024 * 4; public static final long FILE_SIZE = PAGE_SIZE * 2000L * 1000L; public static final String FILE_NAME = "test.dat"; public static final byte[] BLANK_PAGE = new byte[PAGE_SIZE]; public static void main(final String[] arg) throws Exception { preallocateTestFile(FILE_NAME); for (final PerfTestCase testCase : testCases) { for (int i = 0; i < 5; i++) { System.gc(); long writeDurationMs = testCase.test(PerfTestCase.Type.WRITE, FILE_NAME); System.gc(); long readDurationMs = testCase.test(PerfTestCase.Type.READ, FILE_NAME); long bytesReadPerSec = (FILE_SIZE * 1000L) / readDurationMs; long bytesWrittenPerSec = (FILE_SIZE * 1000L) / writeDurationMs; out.format("%s\twrite=%,d\tread=%,d bytes/sec\n", testCase.getName(), bytesWrittenPerSec, bytesReadPerSec); } } deleteFile(FILE_NAME); } private static void preallocateTestFile(final String fileName) throws Exception { RandomAccessFile file = new RandomAccessFile(fileName, "rw"); for (long i = 0; i < FILE_SIZE; i += PAGE_SIZE) { file.write(BLANK_PAGE, 0, PAGE_SIZE); } file.close(); } private static void deleteFile(final String testFileName) throws Exception { File file = new File(testFileName); if (!file.delete()) { out.println("Failed to delete test file=" + testFileName); out.println("Windows does not allow mapped files to be deleted."); } } public abstract static class PerfTestCase { public enum Type { READ, WRITE } private final String name; private int checkSum; public PerfTestCase(final String name) { this.name = name; } public String getName() { return name; } public long test(final Type type, final String fileName) { long start = System.currentTimeMillis(); try { switch (type) { case WRITE: { checkSum = testWrite(fileName); break; } case READ: { final int checkSum = testRead(fileName); if (checkSum != this.checkSum) { final String msg = getName() + " expected=" + this.checkSum + " got=" + checkSum; throw new IllegalStateException(msg); } break; } } } catch (Exception ex) { ex.printStackTrace(); } return System.currentTimeMillis() - start; } public abstract int testWrite(final String fileName) throws Exception; public abstract int testRead(final String fileName) throws Exception; } private static PerfTestCase[] testCases = { new PerfTestCase("RandomAccessFile") { public int testWrite(final String fileName) throws Exception { RandomAccessFile file = new RandomAccessFile(fileName, "rw"); final byte[] buffer = new byte[PAGE_SIZE]; int pos = 0; int checkSum = 0; for (long i = 0; i < FILE_SIZE; i++) { byte b = (byte)i; checkSum += b; buffer[pos++] = b; if (PAGE_SIZE == pos) { file.write(buffer, 0, PAGE_SIZE); pos = 0; } } file.close(); return checkSum; } public int testRead(final String fileName) throws Exception { RandomAccessFile file = new RandomAccessFile(fileName, "r"); final byte[] buffer = new byte[PAGE_SIZE]; int checkSum = 0; int bytesRead; while (-1 != (bytesRead = file.read(buffer))) { for (int i = 0; i < bytesRead; i++) { checkSum += buffer[i]; } } file.close(); return checkSum; } }, new PerfTestCase("BufferedStreamFile") { public int testWrite(final String fileName) throws Exception { int checkSum = 0; OutputStream out = new BufferedOutputStream(new FileOutputStream(fileName)); for (long i = 0; i < FILE_SIZE; i++) { byte b = (byte)i; checkSum += b; out.write(b); } out.close(); return checkSum; } public int testRead(final String fileName) throws Exception { int checkSum = 0; InputStream in = new BufferedInputStream(new FileInputStream(fileName)); int b; while (-1 != (b = in.read())) { checkSum += (byte)b; } in.close(); return checkSum; } }, new PerfTestCase("BufferedChannelFile") { public int testWrite(final String fileName) throws Exception { FileChannel channel = new RandomAccessFile(fileName, "rw").getChannel(); ByteBuffer buffer = ByteBuffer.allocate(PAGE_SIZE); int checkSum = 0; for (long i = 0; i < FILE_SIZE; i++) { byte b = (byte)i; checkSum += b; buffer.put(b); if (!buffer.hasRemaining()) { channel.write(buffer); buffer.clear(); } } channel.close(); return checkSum; } public int testRead(final String fileName) throws Exception { FileChannel channel = new RandomAccessFile(fileName, "rw").getChannel(); ByteBuffer buffer = ByteBuffer.allocate(PAGE_SIZE); int checkSum = 0; while (-1 != (channel.read(buffer))) { buffer.flip(); while (buffer.hasRemaining()) { checkSum += buffer.get(); } buffer.clear(); } return checkSum; } }, new PerfTestCase("MemoryMappedFile") { public int testWrite(final String fileName) throws Exception { FileChannel channel = new RandomAccessFile(fileName, "rw").getChannel(); MappedByteBuffer buffer = channel.map(READ_WRITE, 0, Math.min(channel.size(), MAX_VALUE)); int checkSum = 0; for (long i = 0; i < FILE_SIZE; i++) { if (!buffer.hasRemaining()) { buffer = channel.map(READ_WRITE, i, Math.min(channel.size() - i , MAX_VALUE)); } byte b = (byte)i; checkSum += b; buffer.put(b); } channel.close(); return checkSum; } public int testRead(final String fileName) throws Exception { FileChannel channel = new RandomAccessFile(fileName, "rw").getChannel(); MappedByteBuffer buffer = channel.map(READ_ONLY, 0, Math.min(channel.size(), MAX_VALUE)); int checkSum = 0; for (long i = 0; i < FILE_SIZE; i++) { if (!buffer.hasRemaining()) { buffer = channel.map(READ_WRITE, i, Math.min(channel.size() - i , MAX_VALUE)); } checkSum += buffer.get(); } channel.close(); return checkSum; } }, }; } Results 400MB file =========== RandomAccessFile write=379,610,750 read=1,452,482,269 bytes/sec RandomAccessFile write=294,041,636 read=1,494,890,510 bytes/sec RandomAccessFile write=250,980,392 read=1,422,222,222 bytes/sec RandomAccessFile write=250,366,748 read=1,388,474,576 bytes/sec RandomAccessFile write=260,394,151 read=1,422,222,222 bytes/sec BufferedStreamFile write=98,178,331 read=286,433,566 bytes/sec BufferedStreamFile write=100,244,738 read=288,857,545 bytes/sec BufferedStreamFile write=82,948,562 read=154,100,827 bytes/sec BufferedStreamFile write=108,503,311 read=153,869,271 bytes/sec BufferedStreamFile write=113,055,478 read=152,608,047 bytes/sec BufferedChannelFile write=388,246,445 read=358,041,958 bytes/sec BufferedChannelFile write=390,467,111 read=375,091,575 bytes/sec BufferedChannelFile write=321,759,622 read=1,539,849,624 bytes/sec BufferedChannelFile write=318,259,518 read=1,539,849,624 bytes/sec BufferedChannelFile write=322,265,932 read=1,534,082,397 bytes/sec MemoryMappedFile write=300,955,180 read=305,899,925 bytes/sec MemoryMappedFile write=313,149,847 read=310,538,286 bytes/sec MemoryMappedFile write=326,374,501 read=303,857,566 bytes/sec MemoryMappedFile write=327,680,000 read=304,535,315 bytes/sec MemoryMappedFile write=326,895,450 read=303,632,320 bytes/sec 8GB File ============ RandomAccessFile write=167,402,321 read=251,922,012 bytes/sec RandomAccessFile write=193,934,802 read=257,052,307 bytes/sec RandomAccessFile write=192,948,159 read=248,460,768 bytes/sec RandomAccessFile write=191,814,180 read=245,225,408 bytes/sec RandomAccessFile write=190,635,762 read=275,315,073 bytes/sec BufferedStreamFile write=154,823,102 read=248,355,313 bytes/sec BufferedStreamFile write=152,083,913 read=253,418,301 bytes/sec BufferedStreamFile write=133,099,369 read=146,056,197 bytes/sec BufferedStreamFile write=131,065,708 read=146,217,827 bytes/sec BufferedStreamFile write=132,694,052 read=148,116,004 bytes/sec BufferedChannelFile write=406,147,744 read=304,693,892 bytes/sec BufferedChannelFile write=397,457,668 read=298,183,671 bytes/sec BufferedChannelFile write=364,672,364 read=414,281,379 bytes/sec BufferedChannelFile write=371,266,711 read=404,343,534 bytes/sec BufferedChannelFile write=373,705,579 read=406,934,578 bytes/sec MemoryMappedFile write=123,023,322 read=231,530,156 bytes/sec MemoryMappedFile write=121,961,023 read=230,403,600 bytes/sec MemoryMappedFile write=123,317,778 read=229,899,250 bytes/sec MemoryMappedFile write=121,472,738 read=231,739,745 bytes/sec MemoryMappedFile write=120,362,615 read=231,190,382 bytes/sec Analysis For years I was a big fan of using RandomAccessFile directly because of the control it gives and the predictable execution. I never found using buffered streams to be useful from a performance perspective and this still seems to be the case. In more recent testing I've found that using NIO FileChannel and ByteBuffer are the clear winners from a performance perspective. With Java 7 the flexibility of this programming approach has been improved for random access with SeekableByteChannel. I've seen these results vary greatly depending on platform. File system, OS, storage devices, and available memory all have a significant impact. In a few cases I've seen memory-mapped files perform significantly better than the others but this needs to be tested on your platform because your mileage may vary... A special note should be made for the use of memory-mapped large files when pushing for maximum throughput. I've often found the OS can become unresponsive due the the pressure put on the virtual memory sub-system. Conclusion There is a significant difference in performance for the different means of doing sequential file IO from Java. Not all methods are even remotely equal. For most IO I've found the use of ByteBuffers and Channels to be the best optimised parts of the IO libraries. If buffered streams are your IO libraries of choice, then it is worth branching out and and getting familiar with the sub-classes of Channel and Buffer. From http://mechanical-sympathy.blogspot.com/2011/12/java-sequential-io-performance.html

clover is a great tool for generating code coverage reports from your unit tests. it can be executed as a plugin in eclipse, maven or ant. however, not everyone knows that it can also be used to collect coverage data of integration tests. this post explains how to collect coverage data with clover at runtime. this post assumes that you already know what are unit and integration tests. this post assumes that you know what clover is, and already used it either with eclipse, ant or maven. * let me assured you that even though the directions bellow seems complicated and clumsy at first, after doing them once or twice it is really easy to repeat them. motivation the default action of clover is to gather code coverage information during build time or compile time. therefore, this information includes just the coverage data created by unit tests. if you are developing web applications, you probably use more technologies to test your applications beside unit tests. these technologies may include httpunit/ htmlunit or automation technologies (like selenium). these technologies do not work at build time, they can only work during run time, where a web server is up and running and http calls are made. as a result, the code coverage made during build time is not reflecting the actual code coverage. we should be able to test the coverage while a server is running. the idea the idea is that we will first run clover regularly during build time. we will than take the clover artifacts, put them in our server and then run the integration tests. while running the integration tests, the clover database will be updated and we would be able to generate reports from it which will reflect both unit and integration tests. step 1 – preparation make sure that you have a web/application server (tomcat/jboss/weblogic…) with your web application already deployed. execute clover on your application as you would normally do (either by compiling the code on eclipse or by building with maven or ant, it doesn’t matter). the result of this action would be: clover db files. one of the outcome of executing the clover on your code are the db files. the db files hold all the information about your code and the coverage itself. the location of those files may change depending on the way you use clover and according to the way you configured clover in your environment. this is how the files looks like the .db file holds the information regarding your code (classes, methods and so on). all the other files hold all the coverage data. it is important that you will locate those files because we are going to use them in the next step. an instrumented code. another outcome of clover is that it instruments your code. a clover call is injected into each method so it would be reported in the coverage calculation. we will need this code. we will use this instrumented code in runtime to update the coverage data. if you use eclipse than the generated classes would be instrumented. if you use maven or ant than most chances are that a jar with all the instrumented code would be generated separately. search the instrument code jar. again, i can’t tell you exactly where it is located, but usually it generates a jar with a ‘clover’ postfix. example: if your jar name is my_app.jar, than the generated instrumented code jar will probably be something like my_app-clover.jar. so you will need to do some detective work here to find the instrumented classes/jar. if you are not sure whether the classes are instrumented or not, just decompile one of them with jad and search for the word clover inside of it. a code coverage report. this is a report with the unit test coverage. we don’t really need this, but it would be good so that we would be able to compare it with the report we will generate at the end. step 2 – updating the server the next steps are very important, please make sure you do them properly. replacing the existing application jar/classes with the instrumented jar/classes. take the instrumented jar/classes that were created by the clover and add it to your server’s classpath instead of the original jar/classes. the instrumented code will cause the clover db to be updated with the runtime data. adding clover jars and license. since we will use clover on runtime we will need also the clover jars in our server’s classpath. so add the clover own jars to your server’s classpath. if you are using clover with eclipse than these jars are located in the plugin folder of eclipse. if you are using maven than they will be loacted in your repository. * make sure you are using the same version of clover in your server as you used to generate the db files and instrumented code, otherwise it will not work and you will get error messages. also add the clover license file to the same location as the jars. adding the clover java argument. add the following java arg -dclover.initstring.basedir={ location of the db files that were created by the clover } . * notice – the path you have entered above is the path of the folder which contains the db files. example: -dclover.initstring.basedir=c:/workspace/mywebapp/target/clover. this java argument is used by the clover to locate the db files and update them. step 3 – restarting the server and running the tests now that hopefully all is set properly all that you need to do now is to restart your server and than running your integration tests. the tests should trigger the instrumented code which will call the clover api’s and will update the clover db. while running the tests: look at your log/console and search for error messages from clover. look at the folder which holds the clover db files. if everything is going as it should, new files will be created in this folder while running the tests. if not everything is going well the first time, don’t discourage, just go over each of the steps again. step 4 – generating an updated report if everything went well and new files were created in the db folder than that means you just need to generate a new report. if you are using clover with eclipse than you can simply push the reload button to reload the coverage data. if you are using maven or ant you can execute just the task which generates the report. another way is to use the clover htmlreporter to generate a report easily. now compare the new report to the old report. you should see that the new report coverage is much bigger than the old one since it contains also the integration tests coverage. * notice that not all the data is updated, even though the percentages are being updated, for some reason the calls counter does not. to summarize. as mentioned; yes, these instructions seems a bit complicated but after you succeed the first time, it is very easy to repeat it. in the company i work for we even made this whole process automatic and we are able to generate a full coverage report with unit and integrated tests combined. source: http://www.aviyehuda.com/2011/12/use-clover-to-generate-code-coverage-reports-of-your-integrationautomation-tests/

Introduction A common thing I come across is that teams using a mocking framework assume they are mocking. They are not aware that Mocks are just one of a number of 'Test Doubles' which Gerard Meszaros has categorised at xunitpatterns.com. It’s important to realise that each type of test double has a different role to play in testing. In the same way that you need to learn different patterns or refactoring’s, you need to understand the primitive roles of each type of test double. These can then be combined to achieve your testing needs. I'll cover a very brief history of how this classification came about, and how each of the types differs. I'll do this using some short, simple examples in Mockito. A Very Brief History For years people have been writing lightweight versions of system components to help with testing. In general it was called stubbing. In 2000' the article 'Endo-Testing: Unit Testing with Mock Objects' introduced the concept of a Mock Object. Since then Stubs, Mocks and a number of other types of test objects have been classified by Meszaros as Test Doubles. This terminology has been referenced by Martin Fowler in "Mocks Aren't Stubs" and is being adopted within the Microsoft community as shown in "Exploring The Continuum of Test Doubles" A link to each of these important papers are shown in the reference section. Categories of test doubles The diagram above shows the commonly used types of test double. The following URL gives a good cross reference to each of the patterns and their features as well as alternative terminology. http://xunitpatterns.com/Test%20Double.html Mockito Mockito is a test spy framework and it is very simple to learn. Notable with Mockito is that expectations of any mock objects are not defined before the test as they sometimes are in other mocking frameworks. This leads to a more natural style(IMHO) when beginning mocking. The following examples are here purely to give a simple demonstration of using Mockito to implement the different types of test doubles. There are a much larger number of specific examples of how to use Mockito on the website. http://docs.mockito.googlecode.com/hg/latest/org/mockito/Mockito.html Test Doubles with Mockito Below are some basic examples using Mockito to show the role of each test double as defined by Meszaros. I’ve included a link to the main definition for each so you can get more examples and a complete definition. Dummy Object http://xunitpatterns.com/Dummy%20Object.html This is the simplest of all of the test doubles. This is an object that has no implementation which is used purely to populate arguments of method calls which are irrelevant to your test. For example, the code below uses a lot of code to create the customer which is not important to the test. The test couldn't care less which customer is added, as long as the customer count comes back as one. public Customer createDummyCustomer() { County county = new County("Essex"); City city = new City("Romford", county); Address address = new Address("1234 Bank Street", city); Customer customer = new Customer("john", "dobie", address); return customer; } @Test public void addCustomerTest() { Customer dummy = createDummyCustomer(); AddressBook addressBook = new AddressBook(); addressBook.addCustomer(dummy); assertEquals(1, addressBook.getNumberOfCustomers()); } We actually don't care about the contents of customer object - but it is required. We can try a null value, but if the code is correct you would expect some kind of exception to be thrown. @Test(expected=Exception.class) public void addNullCustomerTest() { Customer dummy = null; AddressBook addressBook = new AddressBook(); addressBook.addCustomer(dummy); } To avoid this we can use a simple Mockito dummy to get the desired behaviour. @Test public void addCustomerWithDummyTest() { Customer dummy = mock(Customer.class); AddressBook addressBook = new AddressBook(); addressBook.addCustomer(dummy); Assert.assertEquals(1, addressBook.getNumberOfCustomers()); } It is this simple code which creates a dummy object to be passed into the call. Customer dummy = mock(Customer.class); Don't be fooled by the mock syntax - the role being played here is that of a dummy, not a mock. It's the role of the test double that sets it apart, not the syntax used to create one. This class works as a simple substitute for the customer class and makes the test very easy to read. Test stub http://xunitpatterns.com/Test%20Stub.html The role of the test stub is to return controlled values to the object being tested. These are described as indirect inputs to the test. Hopefully an example will clarify what this means. Take the following code public class SimplePricingService implements PricingService { PricingRepository repository; public SimplePricingService(PricingRepository pricingRepository) { this.repository = pricingRepository; } @Override public Price priceTrade(Trade trade) { return repository.getPriceForTrade(trade); } @Override public Price getTotalPriceForTrades(Collection trades) { Price totalPrice = new Price(); for (Trade trade : trades) { Price tradePrice = repository.getPriceForTrade(trade); totalPrice = totalPrice.add(tradePrice); } return totalPrice; } The SimplePricingService has one collaborating object which is the trade repository. The trade repository provides trade prices to the pricing service through the getPriceForTrade method. For us to test the business logic in the SimplePricingService, we need to control these indirect inputs i.e. inputs we never passed into the test. This is shown below. In the following example we stub the PricingRepository to return known values which can be used to test the business logic of the SimpleTradeService. @Test public void testGetHighestPricedTrade() throws Exception { Price price1 = new Price(10); Price price2 = new Price(15); Price price3 = new Price(25); PricingRepository pricingRepository = mock(PricingRepository.class); when(pricingRepository.getPriceForTrade(any(Trade.class))) .thenReturn(price1, price2, price3); PricingService service = new SimplePricingService(pricingRepository); Price highestPrice = service.getHighestPricedTrade(getTrades()); assertEquals(price3.getAmount(), highestPrice.getAmount()); } Saboteur Example There are 2 common variants of Test Stubs: Responder’s and Saboteur's. Responder's are used to test the happy path as in the previous example. A saboteur is used to test exceptional behaviour as below. @Test(expected=TradeNotFoundException.class) public void testInvalidTrade() throws Exception { Trade trade = new FixtureHelper().getTrade(); TradeRepository tradeRepository = mock(TradeRepository.class); when(tradeRepository.getTradeById(anyLong())) .thenThrow(new TradeNotFoundException()); TradingService tradingService = new SimpleTradingService(tradeRepository); tradingService.getTradeById(trade.getId()); } Mock Object http://xunitpatterns.com/Mock%20Object.html Mock objects are used to verify object behaviour during a test. By object behaviour I mean we check that the correct methods and paths are excercised on the object when the test is run. This is very different to the supporting role of a stub which is used to provide results to whatever you are testing. In a stub we use the pattern of defining a return value for a method. when(customer.getSurname()).thenReturn(surname); In a mock we check the behaviour of the object using the following form. verify(listMock).add(s); Here is a simple example where we want to test that a new trade is audited correctly. Here is the main code. public class SimpleTradingService implements TradingService{ TradeRepository tradeRepository; AuditService auditService; public SimpleTradingService(TradeRepository tradeRepository, AuditService auditService) { this.tradeRepository = tradeRepository; this.auditService = auditService; } public Long createTrade(Trade trade) throws CreateTradeException { Long id = tradeRepository.createTrade(trade); auditService.logNewTrade(trade); return id; } The test below creates a stub for the trade repository and mock for the AuditService We then call verify on the mocked AuditService to make sure that the TradeService calls it's logNewTrade method correctly @Mock TradeRepository tradeRepository; @Mock AuditService auditService; @Test public void testAuditLogEntryMadeForNewTrade() throws Exception { Trade trade = new Trade("Ref 1", "Description 1"); when(tradeRepository.createTrade(trade)).thenReturn(anyLong()); TradingService tradingService = new SimpleTradingService(tradeRepository, auditService); tradingService.createTrade(trade); verify(auditService).logNewTrade(trade); } The following line does the checking on the mocked AuditService. verify(auditService).logNewTrade(trade); This test allows us to show that the audit service behaves correctly when creating a trade. Test Spy http://xunitpatterns.com/Test%20Spy.html It's worth having a look at the above link for the strict definition of a Test Spy. However in Mockito I like to use it to allow you to wrap a real object and then verify or modify it's behaviour to support your testing. Here is an example were we check the standard behaviour of a List. Note that we can both verify that the add method is called and also assert that the item was added to the list. @Spy List listSpy = new ArrayList(); @Test public void testSpyReturnsRealValues() throws Exception { String s = "dobie"; listSpy.add(new String(s)); verify(listSpy).add(s); assertEquals(1, listSpy.size()); } Compare this with using a mock object where only the method call can be validated. Because we only mock the behaviour of the list, it does not record that the item has been added and returns the default value of zero when we call the size() method. @Mock List listMock = new ArrayList(); @Test public void testMockReturnsZero() throws Exception { String s = "dobie"; listMock.add(new String(s)); verify(listMock).add(s); assertEquals(0, listMock.size()); } Another useful feature of the testSpy is the ability to stub return calls. When this is done the object will behave as normal until the stubbed method is called. In this example we stub the get method to always throw a RuntimeException. The rest of the behaviour remains the same. @Test(expected=RuntimeException.class) public void testSpyReturnsStubbedValues() throws Exception { listSpy.add(new String("dobie")); assertEquals(1, listSpy.size()); when(listSpy.get(anyInt())).thenThrow(new RuntimeException()); listSpy.get(0); } In this example we again keep the core behaviour but change the size() method to return 1 initially and 5 for all subsequent calls. public void testSpyReturnsStubbedValues2() throws Exception { int size = 5; when(listSpy.size()).thenReturn(1, size); int mockedListSize = listSpy.size(); assertEquals(1, mockedListSize); mockedListSize = listSpy.size(); assertEquals(5, mockedListSize); mockedListSize = listSpy.size(); assertEquals(5, mockedListSize); } This is pretty Magic! Fake Object http://xunitpatterns.com/Fake%20Object.html Fake objects are usually hand crafted or light weight objects only used for testing and not suitable for production. A good example would be an in-memory database or fake service layer. They tend to provide much more functionality than standard test doubles and as such are probably not usually candidates for implementation using Mockito. That’s not to say that they couldn’t be constructed as such, just that its probably not worth implementing this way. References Test Double Patterns Endo-Testing: Unit Testing with Mock Objects Mock Roles, Not Objects Mocks Aren't Stubs http://msdn.microsoft.com/en-us/magazine/cc163358.aspx Original Article. The original article and others can be found here http://johndobie.blogspot.com/2011/11/test-doubles-with-mockito.html

1. Overview This is the third of a series of articles about Persistence with Spring. This article will focus on the configuration and implementation of Spring with JPA. For a step by step introduction about setting up the Spring context using Java based configuration and the basic Maven pom for the project, see this article. The Persistence with Spring series: Part 1 – The Persistence Layer with Spring 3.1 and Hibernate Part 2 – Simplifying the Data Access Layer with Spring and Java Generics Part 4 – The Persistence Layer with Spring Data JPA Part 5 – Transaction configuration with JPA and Spring 3.1 2. No More Spring Templates As of Spring 3.1, the JpaTemplate and the corresponding JpaDaoSupport have been deprecated in favor of using the native Java Persistence API. Also, both of these classes are only relevant for JPA 1 (from the JpaTemplate javadoc): Note that this class did not get upgraded to JPA 2.0 and never will. As a consequence, it is now best practice to use the Java Persistence API directly instead of the JpaTemplate, which will effectively decouple the DAO layer implementation from Spring entirely. Exception Translation without the template One of the responsibilities of JpaTemplate is exception translation – translating the low level exceptions – which tie the API to JPA – into higher level, generic Spring exceptions. Without the template to do that, exception translation can still be enabled by annotating the DAOs with the @Repository annotation. That, coupled with a Spring bean postprocessor will advice all @Repository beans with all the implementations of PersistenceExceptionTranslator found in the Container – to provide exception translation without using the template. Exception translation is done through proxies; in order for Spring to be able to create proxies around the DAO classes, these must not be declared final. Injecting the JPA EntityManager with Spring without the template The EntityManager is the API of the persistence context; this can be injected directly in the DAO. The Spring Container is capable of acting as a JPA container and of injecting the EntityManager by honoring the @PersistenceContext (both as field-level and a method-level annotation). For this to work, the PersistenceAnnotationBeanPostProcessor bean must exist in the Spring Container. The bean can be either created explicitly by defining it in the configuration, or automatically, by defining context:annotation-config or context:component-scan in the configuration. 3. The Spring Java configuration The EntityManager is set up in the configuration by creating a Spring factory bean to manage it; this will allow the PersistenceAnnotationBeanPostProcessor to retrieve it from the Container. There are two options to set this up – either the simpler LocalEntityManagerFactoryBean or the more flexible LocalContainerEntityManagerFactoryBean. The latter option is used here, so that additional properties can be configured on it: @Configuration @EnableTransactionManagement public class PersistenceJPAConfig{ @Bean public LocalContainerEntityManagerFactoryBean entityManagerFactoryBean(){ LocalContainerEntityManagerFactoryBean factoryBean = new LocalContainerEntityManagerFactoryBean(); factoryBean.setDataSource( this.restDataSource() ); factoryBean.setPackagesToScan( new String[ ] { "org.rest" } ); JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter(){ { // JPA properties ... } }; factoryBean.setJpaVendorAdapter( vendorAdapter ); factoryBean.setJpaProperties( this.additionlProperties() ); return factoryBean; } @Bean public DataSource restDataSource(){ DriverManagerDataSource dataSource = new DriverManagerDataSource(); dataSource.setDriverClassName( this.driverClassName ); dataSource.setUrl( this.url ); dataSource.setUsername( "restUser" ); dataSource.setPassword( "restmy5ql" ); return dataSource; } @Bean public PlatformTransactionManager transactionManager(){ JpaTransactionManager transactionManager = new JpaTransactionManager(); transactionManager.setEntityManagerFactory( this.entityManagerFactoryBean().getObject() ); return transactionManager; } @Bean public PersistenceExceptionTranslationPostProcessor exceptionTranslation(){ return new PersistenceExceptionTranslationPostProcessor(); } } Also, note that cglib must be on the classpath for Java @Configuration classes to work; to better understand the need for cglib as a dependency, see this article. 4. The Spring XML configuration The same Spring configuration with XML: There is a relatively small difference between the way Spring is configured in XML and the new Java based configuration – in XML, a reference to another bean can point to either the bean or a bean factory for that bean. In Java however, since the types are different, the compiler doesn’t allow it, and so the EntityManagerFactory is first retrieved from it’s bean factory and then passed to the transaction manager: txManager.setEntityManagerFactory( this.entityManagerFactoryBean().getObject() ); 5. Going full XML-less Usually, JPA defines a persistence unit through the META-INF/persistence.xml file. Starting with Spring 3.1, this XML file is no longer necessary – the LocalContainerEntityManagerFactoryBean now supports a ‘packagesToScan’ property where the packages to scan for @Entity classes can be specified. The persistence.xml file was the last piece of XML to be removed – now, JPA can be fully set up with no XML. 5.1. The JPA Properties JPA properties would usually be specified in the persistence.xml file; alternatively, the properties can be specified directly to the entity manager factory bean: factoryBean.setJpaProperties( this.additionlProperties() ); As a side-note, if Hibernate would be the persistence provider, then this would be the way to specify Hibernate specific properties. 5.2. The DAO Each DAO will be based on an parametrized, abstract DAO class class with support for the common generic operations: public abstract class AbstractJpaDAO< T extends Serializable > { private Class< T > clazz; @PersistenceContext EntityManager entityManager; public void setClazz( Class< T > clazzToSet ){ this.clazz = clazzToSet; } public T findOne( Long id ){ return this.entityManager.find( this.clazz, id ); } public List< T > findAll(){ return this.entityManager.createQuery( "from " + this.clazz.getName() ) .getResultList(); } public void save( T entity ){ this.entityManager.persist( entity ); } public void update( T entity ){ this.entityManager.merge( entity ); } public void delete( T entity ){ this.entityManager.remove( entity ); } public void deleteById( Long entityId ){ T entity = this.getById( entityId ); this.delete( entity ); } } A few aspects are interesting here – as discussed, the abstract DAO does not extend any Spring template (such as JpaTemplate). Instead, the JPA EntityManager is injected directly in the DAO, and will have the role of the main Persistence API Also, note that the entity Class is passed in the constructor to be used in the generic operations: @Repository public class FooDAO extends AbstractHibernateDAO< Foo > implements IFooDAO{ public FooDAO(){ setClazz(Foo.class ); } } 6. The Maven configuration In addition to the Maven configuration defined in a previous article, the following dependencies are addeed: spring-orm (which also has spring-tx as its dependency) and hibernate-entitymanager: org.springframework spring-orm 3.2.2.RELEASE org.hibernate hibernate-entitymanager 4.2.0.Final runtime mysql mysql-connector-java 5.1.24 runtime Note that the MySQL dependency is included as a reference – a driver is needed to configure the datasource, but any Hibernate supported database will do. 7. Conclusion This article covered the configuration and implementation of the persistence layer with JPA 2 and Spring 3.1, using both XML and Java based configuration. The reasons to stop relying on templates for the DAO layer was discussed, as well as getting rid of the last piece of XML usually associated with JPA – the persistence.xml. The final result is a lightweight, clean DAO implementation, with almost no compile-time reliance on Spring. You can check out the full implementation in the github project. OriginalThe Persistence Layer with Spring 3.1 and JPA from the Persistence with Spring series

Thursday's CSS3 bitmaps were clever and fun, but a little counter-HTML5-cultural: the whole point of SVG, Canvas, and so forth, is that vectors are better, because simpler, than bitmaps. Today's interactive geometric CSS3 shape is just the opposite: far more pixels than pre-rendering could possibly justify, emphatically composed of 2D surfaces, and fully animated in 3D. It's a folding/unfolding dodecahedron (not in FF/IE): On to the code: Because it's a simple shape, the div-organization is too plain to be interesting. The CSS is more satisfying -- with each side-pentagon transformed around x, y, and z axes, as dodecahedronity requires: #dodecahedron.closed #group5 { -webkit-transform: rotateZ(-324deg) rotateX(63deg); -moz-transform: rotateZ(-324deg) rotateX(63deg); transform: rotateZ(-324deg) rotateX(63deg); } and each pentagon defined with gratuitously pleasing transparency: .p2 { position: absolute; left: 0px; top: 0px; width: 0; height: 0; border-bottom: 59px solid #ff0000; border-left: 81px solid transparent; border-right: 81px solid transparent; } The JavaScript is what you might guess after a few seconds' interaction -- but is written efficiently and clearly enough to merit a look. Worth checking out, as an excellent, direct instantiation of several cool CSS3 elements.

In my last blog I looked a Process Related Classic Mistakes from Rapid Development: Taming Wild Software Schedules by Steve McConnell, which although it’s now been around for at least 10 years, and times have changed, is still as relevant today as when it was written. As Steve’s book states, classic mistakes are classic mistakes because they’re mistakes that are made so often and by so many people. They have predictably bad results and, when you know them, they stick out like a sore thumb and the idea behind listing them here is that, once you know them, you can spot them and hopefully do something to remedy their effect. Classic mistakes can be divided in to four types: People Related Mistakes Process Related Mistakes Product Related Mistakes Technology Related Mistakes Today’s blog takes a quick look at the third of Steve’s categories of mistakes: Product Related Mistakes, which include: Requirements Gold-Plating Feature Creep Developer Gold Plating Push-me, Pull-me Negotiation Research Orientated Development Requirements Gold-Plating Defining a luxury Rolls Royce of a product when an economical hatchback is all that the user requires. This is adding unnecessary features or function points. Users tend to be less interested in complex features than either Marketing or Development and complex features add a disproportionate amount of time to the development schedule. Feature Creep The average project experiences about a 25% change in the requirements over its lifetime. This will add at least 25% to the schedule. Developer Gold Plating Developers like to play with new technology and are anxious to try new things out - even if its only as a means to bolstering their CV’s. They also like adding features because they’re ‘neat’ or ‘cool’ and will take no time at all. There’s no such thing as a free lunch, all new features need to be tested, documented, supported, reviewed etc. etc. Add these features in the next release after evaluating whether or not they’re really required. Push-me, Pull-me Negotiation This is bizarre! Some managers approve a schedule slip on a project that is progressing more slowly than expected and then add in additional new tasks after the schedule change to make the schedule wrong again. Research Orientated Development If your project pushes the state of the art limits, requires new algorithms, hardware designs and higher speed then you’re doing research not development. Research schedules are very problematic and usually wrong. Beware, don’t expect to advance the state of the art to rapidly. From http://www.captaindebug.com/2011/12/product-related-classic-mistakes.html

Have you ever found all sorts of weird errors when running your webapp because several jar files included have the same classes in different versions and the wrong one is being picked up by the application server? Using JBoss Tattletale tool and its Tattletale Maven plugin you can easily find out if you have duplicated classes in your WAR WEB-INF/lib folder and most importantly fail the build automatically if that’s the case before it’s too late and you get bitten in production. Just add the following plugin configuration to your WAR pom build/plugins section. It can also be used for EAR, assemblies and other types of projects. org.jboss.tattletale tattletale-maven 1.1.0.Final verify report ${project.build.directory}/${project.build.finalName}/WEB-INF/lib ${project.reporting.outputDirectory}/tattletale jar multiplejars java6 true persistence-api- xmldsig- You will need to add the JBoss Maven repository to your POM repositories section, or to your repository manager. Make sure you use the repository that only contains JBoss artifacts or you may experience conflicts between artifacts in that repo and the Maven Central repo. Adding extra repositories is a common source of problems and makes builds longer (all repos are queried for artifacts). What I do is add an Apache Archiva proxy connector with a whitelist entry for org/jboss/** so the repo is only queried for org.jboss.* groupIds. jboss https://repository.jboss.org/nexus/content/repositories/releases true false Source: http://blog.carlossanchez.eu/2011/03/23/finding-duplicate-classes-in-your-war-files-with-tattletale/